search

karthink / gptel

A simple LLM client for Emacs
GNU General Public License v3.0
1.56k stars 151 forks source link

Allow customizing the formatting of context insertions to suit model expectations #367

Open algal opened 2 months ago

algal commented 2 months ago

Right now, gptel lets you add files and buffers to the context. But if I understand right, from from the code of functions like gptel-context--insert-buffer-string and gptel-context--file-string, it always adds this material to the context using header line like In file ...: or In buffer ...:.

However, Anthropic is very clear that their model performs better if quoted files and other such material are contained within XML-like tags. This has been my experience as well.

It would be valuable to be able to customize how gptel builds the context string, so that it could be optimized to work with a particular model. I think it'd be especially valuable to do this for Claude right now, since the sonnet-3.5 model is very strong at coding assistance in a conversational style.

I'd be happy to work on this change, if it seems like it would be worth pursuing.

karthink commented 2 months ago

If I understand correctly, you're looking for the option gptel-context-wrap-function. See its documentation for details, and let me know what you think.

I'll leave this issue open for now since I think there's room for improvement. Perhaps gptel can provide a few wrap functions out of the box tailored to the expectations of the big LLM providers.

tschwinge commented 1 month ago

Has anyone already implemented anything here, re Anthropic: "Structure document content and metadata with XML tags", https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/long-context-tips#example-multi-document-structure, for example? If not, I'll have a look.

tschwinge commented 4 weeks ago

I found @daedsidog's #349 "Add better default context string", which also relates to this topic here.

daedsidog commented 2 days ago

I found @daedsidog's #349 "Add better default context string", which also relates to this topic here.

As mentioned before, you can control the context formatting via customizing the gptel-context-wrap-function. The PR you linked only slightly improves the default context string. To do what you want, you would have to define your wrap function depending on the model you are using. I believe it's not within the scope of this project to have a tailored wrap function for every specific backend.