LLM context aggregator

[daedsidog]

This adds the contexter I've been working on for my own personal use to GPTel. As of now, this pull request is not complete because I'm unsure what's the best path to integrate it, and also how it will fit with the new UI refactor. I synced up with the master branch of GPTel, and just added that in, but my transient menu seems to be vastly different from the previews I've seen.

Because this started as part of my config & grew organically, there are still some features that are part of my config and not in GPTel. Those parts are added to the end of this pull request in the form of a snippet from my config.

In a nutshell, I personally find this to be an incredibly useful programming tool for productivity, more so than any other AI programming tool I've tried. It pampered me to the point where I don't think I can go back to programming without it.

Here is a short guide how to use it with my config (this will change the more things I'll offload to GPTel):

1. Select various regions in buffers and add them to the context buffer with C-c g p. You can press c to remove selected context snippets from the context buffer directly, or C-c g p again to remove context snippets from the code buffers.

2. Write an instruction comment in code, then select it and use C-c g r to send it to GPTel.

With Lisp, the contexter automatically collapses s-expressions in areas that they are not required in, such as functions with docstrings, thus saving tokens:

Here is the part of my config file which sets all this up:

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; -------------------------------- GPTEL --------------------------------- ;;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

(use-package gptel-contexter
  :commands (gptel-context-string
             gptel-context-substring
             gptel-context-in-region
             gptel-pop-or-push-context
             gptel-remove-context))

(defun my/code-refactoring-prompt-string (code-regions)
  "Create a prompt for an LLM to refactor CODE-REGIONS in the current buffer."
  (let* ((context (gptel-context-string))
         (have-context (not (zerop (length context)))))
    (concat "I need you to refactor and/or write code. Write carefully with\
 great respect to existing syntax, styling, & conventions, this also includes\
 column width."
            (when have-context
              (concat "Listed below is context that may help you. Use it as a\
 read-only reference. You are strictly prohibited from changing it in any\
 way, even if asked directly."
                      "\n\n" context))
            (if have-context
                "\n\nNow, given the above context, do as instructed by the\
 INSTRUCTION comments in the following source:"
              "\n\nDo as instructed by the INSTRUCTION comments in the following\
 source:")
            "\n\n"
            (gptel-context-substring (current-buffer) code-regions)
            "\n\n"
            "When writing code, do not delete existing comments. Do not fix\
 what you may think are bugs, unless explicitly instructed to. Do not refactor\
 things you were not instructed to. You are to remove instruction comments\
 after they are completed.

You have more creative freedom when you are generating code, but you must still\
 follow your instructions closely.

As opposed to the context, the source code you have been given has been erased\
 from its original location. Therefore, you are expected to return it, fully.\
 Of course, you are not to return the ellipses or line numbers. Remember, you\
 may have been given partial code, so be mindful and return it in the same way\
 you found it. Return **ALL** of the modified target code, leave nothing behind.

Return only code, nothing else. When documenting your work, make no reference\
 to the instructions, so that an outside reader will think the code was written\
 in a natural fashion.")))

(defun my/predicated-prog-lang (prog-lang-major-mode)
  "Derive the programming language name from the PROG-LANG-MAJOR-MODE.
Adds a predicate before the name."
  (pcase prog-lang-major-mode
    ('emacs-lisp-mode "an Emacs Lisp")
    ('js-mode "a JavaScript")
    ('c-mode "a C")
    ('c++-mode "a C++")
    ('lisp-mode "a Common Lisp")
    ('twee-mode "a Twee")
    ('web-mode "a Web")
    (_ (concat "a " (substring (symbol-name prog-lang-major-mode) nil -5)))))

(defun my/refactoring-message ()
    "Set a generic refactor/rewrite message for the buffer."
    (if (derived-mode-p 'prog-mode)
        (concat (format "You are %s programmer. Write legible, elegant, and\
 very terse code based on instructions. Do not stray from instructions."
                        (my/predicated-prog-lang major-mode)))
      (format "You are a prose editor. Rewrite the following text to be more\
 professional.")))

(defun my/clean-up-llm-chat-response (beg end)
  (save-excursion
    (goto-char beg)
    (org-latex-preview)
    (my/resize-org-latex-overlays)
    (my/adjust-org-latex-overlay-padding)
    (pulse-momentary-highlight-region beg end)))

(use-package gptel
  :commands (gptel-clean-up-llm-code))

(defun my/clean-up-llm-response (beg end)
  (if (bound-and-true-p gptel-mode)
      (my/clean-up-llm-chat-response beg end)
    (gptel-clean-up-llm-code (current-buffer) beg end)))

;; OpenAI ChatGPT API interface
(use-package gptel-curl
  :commands (gptel gptel-menu gptel-abort gptel-clean-up-llm-code)
  :config
  (require 'gptel-transient)
  (customize-set-variable 'gptel-directives
                          (let ((new-directives gptel-directives))
                            (setf (cdr (assoc 'default new-directives))
                                  "You are a large language model living in\
 Emacs and a helpful assistant. Respond as concisely as possible.")
                            new-directives))
  (customize-set-variable 'gptel-default-mode 'org-mode)
  (customize-set-variable
   'gptel-prompt-prefix-alist
   (cons (cons 'org-mode "* Query\n\n")
         (assq-delete-all 'org-mode gptel-prompt-prefix-alist)))
  (customize-set-variable
   'gptel-response-prefix-alist
   (cons (cons 'org-mode "** Response\n\n")
         (assq-delete-all 'org-mode gptel-response-prefix-alist)))
  (customize-set-variable 'gptel-model "gpt-4-0125-preview")
  (add-hook 'gptel-mode-hook
            (lambda () (interactive)
              (display-time-mode)
              (setq gptel--system-message
                    (alist-get 'default gptel-directives))))
  (add-hook 'gptel-post-response-functions #'my/clean-up-llm-response)
  :bind (("C-c g c" . #'gptel)
         ("C-c g m" . #'gptel-menu)
         ("C-c g p" . #'gptel-pop-or-push-context)
         ("C-c g r" . #'my/refactor-using-llm-with-context)
         ("C-c g a" . #'gptel-abort)
         (:map gptel-mode-map
               ;; Insert a prefix when deleting the entire buffer in
               ;; `gptel-mode'.
               (("C-<tab> DEL" . (lambda ()
                            (interactive)
                            (my/delete-entire-buffer)
                            (catch 'finish
                              (dolist (it gptel-prompt-prefix-alist)
                                (let ((mode (car it))
                                      (prefix (cdr it)))
                                  (when (eq major-mode mode)
                                    (insert prefix)
                                    (goto-char (point-max))
                                    (throw 'finish nil)))))))))))

(cl-defun my/refactor-using-llm-with-context ()
  (interactive)
  (unless (use-region-p)
    (cl-return-from my/refactor-using-llm-with-context))
  (let* ((start (region-beginning))
         (end (region-end))
         (context (gptel-context-in-region (current-buffer) start end))
         (prompt)
         (region-substring (buffer-substring-no-properties start end)))
    (when context
      (gptel-remove-context context t))
    ;; Check for the presence of instruction in region.
    (when (not (string-match-p "INSTRUCTION:" region-substring))
      (error "No instruction found within region"))
    (setq prompt (my/code-refactoring-prompt-string (list (list start end))))
    (deactivate-mark)
    (delete-region start end)
    (set-mark (point))
    (insert prompt)
    (activate-mark)
    (gptel-menu)
    ;; Set the system message through the transient menu and keybord
    ;; macros, which is just a temporary solution.
    (execute-kbd-macro (kbd "ss"))
    (delete-region (point) (point-max))
    (insert (my/refactoring-message))
    (execute-kbd-macro (kbd "C-c C-c"))
    (execute-kbd-macro (kbd "rR"))))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; ----------------------------- END OF GPTEL ----------------------------- ;;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

As can be plainly seen from the config file, the mechanism to actually send this to GPTel is rather primitive. It uses keyboard macros to navigate the transient menu, and then just dumps the prompt into the buffer, and uses the transient refactor menu to send it to GPTel.

[karthink]

I share this sentiment now. It never occurred to me to customize a buffer like this, but now I will definitely do this over haphazardly overwriting the dynamic variables when doing in-buffer rewrites.

What we'd like is a very simple default for injecting context with gptel, but support for scriptability so the user can plug in their own function to inject the context instead. This function can accept either the context string (from all the context chunks) or perhaps gptel--context-overlay-alist.

Something like this:

(defvar gptel-context-injection-function #'gptel-context--wrap-system)

where gptel provides the functions gptel-context--wrap-system and gptel-context--wrap-user, but you can plug in your own. Not suggesting we do this, there may be better designs. Just thinking out loud.

The advantage will be that gptel's default behavior will remain (sort of) simple and predictable, but any kind of workflow is now possible. Depending on your task, you can set up the exact interleaving of text and context chunks that you want, and I don't have to try to support every possible arrangement. With recursive-edit, the context-injection process can also be made interactive.

The disadvantage is that you can no longer pick a behavior from the transient menu, at least not without a little extra work. This work may be worth doing, we'll see.

The majority of the time that I used the context was not in in-buffer rewrites (the whole reason for the preamble/postamble mess) but to actually just build a list of things which I insert in the chat buffer, which come formatted in nice Markdown.

I'm trying to understand this. If you're using gptel in a chat buffer, the context chunks can be inserted into the chat buffer itself, right? (Unless you want them to be part of the system message.) In this case it's easy to write a function that collects the context and inserts it wherever you want in the chat buffer. It doesn't need to be invisibly appended to the user prompt by gptel just before sending the request.

I thought this PR's features were mainly for use in code buffers, where you don't want to modify the buffer itself.

However, a major convenience was also having GPTel automatically embed the context in, say, the last prompt, without having to do it manually. What has been done so far was messy, but it was nice nonetheless. It would still be possible to replicate strictly with Emacs commands, but that would mean both levers (destination & usage in chat buffer) would become obsolete.

Yeah, gptel should still provide the choice between adding the context to the system message vs the user prompt. I'm not sure of how to structure this code yet, just that the available choices (before/after system message, before/after user prompt) are at once too numerous and yet not flexible enough.

[daedsidog]

I'm trying to understand this. If you're using gptel in a chat buffer, the context chunks can be inserted into the chat buffer itself, right? (Unless you want them to be part of the system message.) In this case it's easy to write a function that collects the context and inserts it wherever you want in the chat buffer. It doesn't need to be invisibly appended to the user prompt by gptel just before sending the request.

If simply inserted into the chat, then they are inserted into the history, and thus accrue a token cost & "mental overhead" over time, unless manually deleted & re-entered. With automatic insertion into the last prompt, this process is automated. Example where this came very handy: I had a TikZ graphic which I wanted to refine, so I selected my current TikZ code as context, and asked the model to give me instructions on how to refine it. I then modified the actual code according to its instructions, tested it, and kept prompting for more refinements. In this situation, the chat history, with appending to the final prompt, looked something like:

• Me: I need my graphic to have X.
• Model: Do Y and then you get X.
• Me: This is nice, now how do I get Z?
• Model: Do W to get Z.
• Me (final prompt): I have the following TikZ code: <code>. Change X to be more like Z.

I.e., the I have the following TikZ code: <code>, which is the context, where I have the following Tikz code: is its preamble, does not persist in the history, and only gets applied to the final prompt, saving tokens and model efficiency (maybe?).

I thought this PR's features were mainly for use in code buffers, where you don't want to modify the buffer itself.

The original inception was a more powerful rewrite feature for code refactoring via instructions s.t. the model will have access to the general structure of an existing code base, but use cases like the one described above popped up with time. If by "where you don't want to modify the buffer" you mean the context buffer, then yes, I don't modify it other than marking contexts for deletion.

[daedsidog]

where gptel provides the functions gptel-context--wrap-system and gptel-context--wrap-user, but you can plug in your own. Not suggesting we do this, there may be better designs. Just thinking out loud.

I'm in favor of having a simple default behavior and a very versatile programmable advanced behavior. An idea I had was provide a variable which holds a format string. Something like Here is the context: %1. %2.. where %1 would be where the context would go and %2 is where the prompt/system message would be, for example.

However, I assume down the line the decorations would be customizable as well, so the final solution would have to, like you suggested, use the gptel--context-overlay-alist, which would give full user control. I suppose the default decorating behavior could be provided as the default function. Should decorating and ordering be done in the same function?

[karthink]

I suppose the default decorating behavior could be provided as the default function. Should decorating and ordering be done in the same function?

After thinking over it for a bit I have another idea, I'll prototype it soon.

How important is it to you to have the context chunks sorted by buffer? Is there a problem with just including each chunk (along with the line number and buffer name) in the order in which they were added using gptel-add-context? I ask because this will simplify the implementation of this and future ideas a fair bit. It also makes the ordering predictable and in control of the user.

[daedsidog]

I suppose the default decorating behavior could be provided as the default function. Should decorating and ordering be done in the same function?

After thinking over it for a bit I have another idea, I'll prototype it soon.

How important is it to you to have the context chunks sorted by buffer? Is there a problem with just including each chunk (along with the line number and buffer name) in the order in which they were added using gptel-add-context? I ask because this will simplify the implementation of this and future ideas a fair bit. It also makes the ordering predictable and in control of the user.

When you say sorted by buffer, are you referring to the fact that chunks belonging to the same buffer appear together under the same decoration? If so, that depends: I would really want the user to be able to generate whatever context string he wants (even being able to collapse functions if he chooses to, as I had done before).

Or are you instead talking about the order of contexts as they appear in the buffer? Having them appear in the other they were added would be confusing for the model in some cases, I think. At the very least it should be user-configurable by via a custom function.

[karthink]

Or are you instead talking about the order of contexts as they appear in the buffer? Having them appear in the other they were added would be confusing for the model in some cases, I think. At the very least it should be user-configurable by via a custom function.

I mean this -- the contexts are sent in the order in which they were added. I'm not sure what "configurable via a custom function" means. Is the function supplied by the user?

When you say sorted by buffer, are you referring to the fact that chunks belonging to the same buffer appear together under the same decoration? If so, that depends: I would really want the user to be able to generate whatever context string he wants (even being able to collapse functions if he chooses to, as I had done before).

The previous point will mean that they won't appear inside the same decoration.

I'll see what I can do to make this stuff configurable.

[daedsidog]

I mean this -- the contexts are sent in the order in which they were added.

I believe context insertion order should not matter for both the contexts buffer and the context string.

For the contexts buffer, having the chunks scattered around without regard to their "nearness" might make marking for deletion harder, and would force users to "hunt down" contexts. I think having them grouped together by buffer is much tidier and friendlier.

For the context string, I fear it would introduce "cognitive overhead" for the model if contexts from different buffers were "interlaced" together. More on this:

I'm not sure what "configurable via a custom function" means. Is the function supplied by the user?

I assumed that you wanted users to provide some sort of insertion/decoration function to dictate how the contexts are decorated and where they are inserted in the system message or prompt.

The context string's final form should be 100% configurable by the user (with provided default?). The user may choose the order of appearance, the decorations, how its inserted into the system/prompt, etc. If the user chooses so, he can make the context appear grouped by the buffers--or not. This should be completely up to him.

For example, right now, I am wrapping contexts from the same buffer with the "In buffer X:" text, but another user may choose to do something different.

[karthink]

daedsidog @.***> writes:

I believe context insertion order should not matter for both the contexts buffer and the context string.

For the contexts buffer, having the chunks scattered around without regard to their "nearness" might make marking for deletion harder, and would force users to "hunt down" contexts. I think having them grouped together by buffer is much tidier and friendlier.

For the context string, I fear it would introduce "cognitive overhead" for the model if contexts from different buffers were "interlaced" together.

Okay. I take this to mean you want contexts grouped by buffer to be the default behavior.

I'm not sure what "configurable via a custom function" means. Is the function supplied by the user?

I assumed that you wanted users to provide some sort of insertion/decoration function to dictate how the contexts are decorated and where they are inserted in the system message or prompt.

The context string's final form should be 100% configurable by the user (with provided default?). The user may choose the order of appearance, the decorations, how its inserted into the system/prompt, etc. If the user chooses so, he can make the context appear grouped by the buffers--or not. This should be completely up to him.

For example, right now, I am wrapping contexts from the same buffer with the "In buffer X:" text, but another user may choose to do something different.

I think this is out of scope for gptel. As discussed above, we can add an option for the user to provide their own function that accepts gptel--context-overlay-alist and returns the final context string to be sent with the request. The default function that ships with gptel will do what we're currently doing: add a "Request context:" string as the preamble, add "In buffer X" where appropriate, and group contexts by buffer.

For reasons mentioned above, I also want to avoid adding the preamble and postamble options. This behavior can be implemented by the function provided by the user.

[daedsidog]

I think this is out of scope for gptel. As discussed above, we can add an option for the user to provide their own function that accepts gptel--context-overlay-alist and returns the final context string to be sent with the request. The default function that ships with gptel will do what we're currently doing: add a "Request context:" string as the preamble, add "In buffer X" where appropriate, and group contexts by buffer.

What is out of scope exactly? Seems like what you suggest (user-provided function that accepts gptel--context-overlay-alist) satisfies the criteria?

For reasons mentioned above, I also want to avoid adding the preamble and postamble options. This behavior can be implemented by the function provided by the user.

Indeed they are obsolete if a user can provide their own function.

[karthink]

What is out of scope exactly? Seems like what you suggest (user-provided function that accepts gptel--context-overlay-alist) satisfies the criteria?

I think I misunderstood, I was talking about providing features with gptel to replace the interstitial text (between context chunks), preamble etc as opposed to just the plug-in function.

I've added the gptel-context-string-function option now to provide that functionality.

[daedsidog]

I've added the gptel-context-string-function option now to provide that functionality.

I assume gptel-context-injection-destination will be rewritten to only include 3 destinations (none, system, prompt) later on? Right now it's still before/after prompt/system.

[karthink]

I assume gptel-context-injection-destination will be rewritten to only include 3 destinations (none, system, prompt) later on? Right now it's still before/after prompt/system.

Yeah, I'll do that next -- none, (append to system message), (prepend to user prompt). If this turns out to be inadequate we can add back in the other options later.

I haven't worked on it yet since modifying the user prompt requires changing every backend type's prompt creation function (i.e. individually for OpenAI, Anthropic, Ollama etc). I need to figure out how to change the request construction pipeline to do this at a higher level, like we do for the system message right now.

[karthink]

I haven't worked on it yet since modifying the user prompt requires changing every backend type's prompt creation function (i.e. individually for OpenAI, Anthropic, Ollama etc). I need to figure out how to change the request construction pipeline to do this at a higher level, like we do for the system message right now.

After looking at this carefully, it looks like adding this logic to each backend separately is the least worst solution. Every other method I can think of involves injecting it when the buffer is being parsed, which will make debugging/maintaining this feature harder.

[karthink]

• Renamed gptel-context-injection-destination to gptel-use-context.
• Simplified allowed values to nil, system and user.
• Added support for injecting the context into the user prompt for all supported backends.

[karthink]

@daedsidog Do we need gptel-use-context-in-chat? There's already a gptel-use-context (formerly gptel-context-injection-destination) that can be set to nil to disable the context. If required, the user can do

(add-hook 'gptel-mode-hook (lambda () (setq-local gptel-use-context nil)))`

to achieve the same effect.

[daedsidog]

@daedsidog Do we need gptel-use-context-in-chat? There's already a gptel-use-context (formerly gptel-context-injection-destination) that can be set to nil to disable the context. If required, the user can do

(add-hook 'gptel-mode-hook (lambda () (setq-local gptel-use-context nil)))`

to achieve the same effect.

I don't see a reason to keep gptel-use-context-in-chat since the settings can be buffer local. I also don't see a reason to use a hook since one could just locally disable/enable the context much more conveniently through the transient menu. Nevermind, it's just a way to set the chat buffer local setting to default to not using context.

Some testing I've done with the more recent commits don't have the key bindings in the context buffer working (i.e. the next/previous & deletion keys). Removed by design? Might want to change the help message at the top of the buffer to reflect the active keys if you want those to be customizable.

[karthink]

I don't see a reason to keep gptel-use-context-in-chat since the settings can be buffer local. ~I also don't see a reason to use a hook since one could just locally disable/enable the context much more conveniently through the transient menu.~ Nevermind, it's just a way to set the chat buffer local setting to default to not using context.

Okay, I'll remove the -in-chat option.

Some testing I've done with the more recent commits don't have the key bindings in the context buffer working (i.e. the next/previous & deletion keys). Removed by design? Might want to change the help message at the top of the buffer to reflect the active keys if you want those to be customizable.

Not by design. This is a bug, I'll fix it.

[karthink]

Some testing I've done with the more recent commits don't have the key bindings in the context buffer working (i.e. the next/previous & deletion keys).

Should be fixed.

[karthink]

• gptel-add-context is now available as gptel-add.
• Consistent naming across the board.

All that's left is to fix up the transient menu, lint the new code and maybe add a couple of tests.

EDIT: Oh, need to update the README as well.

[daedsidog]

• gptel-add-context is now available as gptel-add.

  • Consistent naming across the board.

All that's left is to fix up the transient menu, lint the new code and maybe add a couple of tests.

EDIT: Oh, need to update the README as well.

Okay. I'll be using for a while and see if there was anything else that needs fixing/adding.

[karthink]

Added support for files as context.

Still tweaking the transient menu, I can't seem to find a good, non-busy arrangement. One thing I'm reasonably sure of is that once it's in good shape, we shouldn't hide the context features behind gptel-expert-commands, as I want it to be easily discoverable.

[karthink]

• Added DWIM dired support to gptel-add. It is still possible to send the text of the dired buffer instead of adding files, you just have to select a region first (or C-x h).
• Grouped together all the context options in the menu again. This is an experiment.

[daedsidog]

Apologies for the poor quality of the commit messages, Magit has an issue committing on my setup right now. I limit my changes to files only this time around to remedy this for now.

Still tweaking the transient menu, I can't seem to find a good, non-busy arrangement. One thing I'm reasonably sure of is that once it's in good shape, we shouldn't hide the context features behind gptel-expert-commands, as I want it to be easily discoverable.

I think it's a major improvement over how it looked previously when I added it in. The current arrangement is nice, but maybe the keys need some work. Maybe.

The discoverability seems nicer now, and being able to tell the user that you can add contexts in files or regions from the transient menu is a nice way to let them know these things exist.

Unrelated: Is there any reason you prefer "There are no active gptel contexts." over "There are no active GPTel contexts."?

[karthink]

Apologies for the poor quality of the commit messages, Magit has an issue committing on my setup right now. I limit my changes to files only this time around to remedy this for now.

No worries, I will fix the commit messages when I squash things down to 5-6 commits at the end.

Still tweaking the transient menu, I can't seem to find a good, non-busy arrangement. One thing I'm reasonably sure of is that once it's in good shape, we shouldn't hide the context features behind gptel-expert-commands, as I want it to be easily discoverable.

I think it's a major improvement over how it looked previously when I added it in. The current arrangement is nice, but maybe the keys need some work. Maybe.

The discoverability seems nicer now, and being able to tell the user that you can add contexts in files or regions from the transient menu is a nice way to let them know these things exist.

The main thing I can't decide on is where the -i Include (with system-message etc) option should go -- under the Context heading where it right now, or under Request Parameters to the left.

• It makes sense in both places.
• Additionally it can be set buffer-locally, so it makes sense to put it under the Set (globally|for this buffer) option.
• On the other hand, there are too many things in the Request Parameters column already, and this is easy to miss.

Unrelated: Is there any reason you prefer "There are no active gptel contexts." over "There are no active GPTel contexts."?

Yeah, I just stick with gptel everywhere in the docs and Readme, even when starting a sentence.

[daedsidog]

Additionally it can be set buffer-locally, so it makes sense to put it under the Set (globally|for this buffer) option.

All the parameters are affected by the locality setting, e.g. the system message, which is under its own heading. If I would've been pedantic, I would move the locality setting to its own heading, and keep the context inclusion option where it us (under context).

[karthink]

Additionally it can be set buffer-locally, so it makes sense to put it under the Set (globally|for this buffer) option.

All the parameters are affected by the locality setting, e.g. the system message, which is under its own heading. If I would've been pedantic, I would move the locality setting to its own heading, and keep the context inclusion option where it us (under context).

Not all the parameters are. None of the input and output redirection options are, nor is the additional directive, since these are per-request. If you set them with transient-set (C-x s) they are then global.

The internal logic I've used is that if the option begins with a -, as in -m, it is persistent. This is why all the context related infixes begin with -. Buffer-local setting is limited right now to the Request Parameters column.

[daedsidog]

Additionally it can be set buffer-locally, so it makes sense to put it under the Set (globally|for this buffer) option.

All the parameters are affected by the locality setting, e.g. the system message, which is under its own heading. If I would've been pedantic, I would move the locality setting to its own heading, and keep the context inclusion option where it us (under context).

Not all the parameters are. None of the input and output redirection options are, nor is the additional directive, since these are per-request. If you set them with transient-set (C-x s) they are then global.

The internal logic I've used is that if the option begins with a -, as in -m, it is persistent. This is why all the context related infixes begin with -. Buffer-local setting is limited right now to the Request Parameters column.

I see. But why then is the system message not prefixed?

[karthink]

I see. But why then is the system message not prefixed?

Because it gets its own menu, where you can control its buffer-local value:

I could still prefix it, I guess, so there are also historical reasons.

[karthink]

Anything else we need to do before merging?

1. I'll update the Readme today.
2. Right now the only ways to remove a context interactively is from the context buffer, or by using gptel-add with the cursor inside a context. Do we need a separate gptel-remove command to select a context (maybe with completing read) and remove it? I'm not sure offers any benefits over using the context buffer.
3. Do we need a separate gptel-add-buffer command?
4. Do we need a gptel-context command that brings up the context buffer?
5. Should the context options get their own transient menu? This would be similar to how gptel-system-prompt is its own command and transient menu for setting the system message.
6. I'll add some indication about the context to the header line in chat buffers.

[daedsidog]

Ah, I never noticed that. They share the same value, then. I figured the only locality switch was under the model.

Actually, I just tried that. They don't share the same value. You can have separate system message & model param locality settings. Oddly enough though, when you set the model parameter switch, it will also affect the system message when you don't use the system message switch. Intentional?

[karthink]

Actually, I just tried that. They don't share the same value. You can have separate system message & model param locality settings. Oddly enough though, when you set the model parameter switch, it will also affect the system message when you don't use the system message switch. Intentional?

Looks like a bug with how transient updates its display -- the setting is indeed shared. The locality setting in both menus is the same transient infix and sets a single variable, gptel--set-buffer-locally. I'll fix it after merging this PR.

[daedsidog]

Anything else we need to do before merging?

I'm at the process of testing out all the features I needed personality from this feature. My main requirement right now is code refactoring. Having the crucial bits supported out of the box simplified the code. But, as I was trying to understand how gptel-context-string-function was working, I noticed a bug where if the inclusion target is user, then an args out of range error is signaled, so that needs fixing.

There also seems to be a bug where overlays may be deleted in a manner which the evaporation doesn't clean them., but I haven't confirmed this.

I'm working on these as I go.

Just in case you are interested, here is the portion of my user code on how I am approaching the refactoring function:

(defun my/predicated-prog-lang (prog-lang-major-mode)
  "Derive the programming language name from the PROG-LANG-MAJOR-MODE.
Adds a predicate before the name."
  (pcase prog-lang-major-mode
    ('emacs-lisp-mode "an Emacs Lisp")
    ('js-mode "a JavaScript")
    ('c-mode "a C")
    ('c++-mode "a C++")
    ('lisp-mode "a Common Lisp")
    ('twee-mode "a Twee")
    ('web-mode "a Web")
    (_ (concat "a " (substring (symbol-name prog-lang-major-mode) nil -5)))))

(defun my/refactoring-system-message ()
  "Set a generic refactor/rewrite system message for the buffer."
  (if (derived-mode-p 'prog-mode)
      (concat (format "You are %s programmer. Write legible, elegant, and\
 very terse code based on instructions. Do not stray from instructions."
                      (my/predicated-prog-lang major-mode)))
    (format "You are a prose editor. Rewrite the following text to be more\
 professional.")))

(cl-defun my/refactor-using-gptel (&optional arg)
  "Refactor via LLM.  Non-nil ARG means dry run."
  (interactive "P")
  (unless (use-region-p)
    (cl-return-from my/refactor-using-gptel))
  (let* ((start (region-beginning))
         (end (region-end))
         (contexts (gptel-context--in-region (current-buffer) start end))
         (region-substring (buffer-substring-no-properties start end)))
    (when contexts
      (gptel-context-remove (car contexts)))
    ;; Check for the presence of instruction in region.
    (when (not (string-match-p "INSTRUCTION:" region-substring))
      (error "No instruction found within region"))
    (let ((gptel--system-message (my/refactoring-system-message))
          (gptel-use-context 'user))
      ;; TODO: Define a gptel-context-string-function which will instruct the model on how to refactor the code.
      ;; The idea right now is to have the currently-selected region be manipulated to be wrapped in the context
      ;; string, and be sent to the model.
      )))

[karthink]

But, as I was trying to understand how gptel-context-string-function was working, I noticed a bug where if the inclusion target is user, then an args out of range error is signaled, so that needs fixing.

I'm assuming you're working on fixing this? You can give me specific bug reproduction instructions if you want me to fix it instead.

[daedsidog]

But, as I was trying to understand how gptel-context-string-function was working, I noticed a bug where if the inclusion target is user, then an args out of range error is signaled, so that needs fixing.

I'm assuming you're working on fixing this? You can give me specific bug reproduction instructions if you want me to fix it instead.

1. Mark a context chunk
2. Use -i to set destination to user prompt
3. In a buffer, select some text in a region and send it with a dry run
4. You should get an error

[karthink]

It's much worse, Emacs locks up when I try this.

If I don't mark a region (step 3) it works fine, which is strange.

[daedsidog]

Another issue is that if you create a context in some buffer, delete the buffer, then try to bring up the transient menu, you get:

transient-setup: Wrong type argument: stringp, #<killed buffer>

[karthink]

Fixed both bugs.

[daedsidog]

I don't see a mechanism allowing the manipulation of the context in relation to the destination string.

Yes, gptel-context-string-function allows you to decorate the context, but after that your only use of it is to prepend it to either the system message or user prompt. It would be very useful to be able to wrap the context around text which is meant for refactoring.

For example, let's say I have the context

(defun my/fun (a b) (+ a b))

and a line in a buffer:

;; INSTRUCTION: Implement function f via the context

I would then like to configure it so that when I select the above line, and send it for rewriting, the dry run would look something like this:

(:model "gpt-4o" :messages
        [(:role "system" :content "You are a large language model living in Emacs and a helpful assistant. Respond concisely.")
         (:role "user" :content "In buffer `eld.eld`:

```lisp-data
... (Line 39)
(defun my/fun (a b) (+ a b))
```

Use the above context to implement the following instruction, which is located on line 23 in buffer main.c:

```
;; INSTRUCTION: Implement function f via the context
```

Make sure to use the context when doing so.")]
        :stream t :temperature 1.0)

Right now, without having the ability to do so, my alternative would be to disable the destination entirely, format that string in-place, and send that to the model.

This is not the end of the world, and may even be outside of the scope, but having this done automatically to the last message would save a lot of work in other areas, like the dedicated chat buffer. In the chat buffer, I don't quite have the luxury of engineering the prompt in-place the same way I do in programming buffers (for clarity, by in-place I mean deleting the string as it appears and sending something else in its place).

My suggestion is to modify the context function to something like a prompt/system formatting function. It accepts the alist and the original prompt/system. This would allow for the most flexibility.

[karthink]

IIUC, you would like to be able to wrap the user prompt with the context, i.e. add part of the context before and the rest after the prompt. In this case neither of the options you originally provded (:before-user-prompt, :after-user-prompt) would suffice either.

My suggestion is to modify the context function to something like a prompt/system formatting function. It accepts the alist and the original prompt/system. This would allow for the most flexibility.

This is a little tricky to do neatly, but anyway: are you suggesting this function should receive the original prompt/system as a string as the second argument? How will you splice a string that doesn't have any imposed structure?

[daedsidog]

IIUC, you would like to be able to wrap the user prompt with the context, i.e. add part of the context before and the rest after the prompt.

That's correct in this case, but there may be other variations.

In this case neither of the options you originally provded (:before-user-prompt, :after-user-prompt) would suffice either.

They did suffice, although the usage was extremely filthy: I used the preamble and postamble to sandwich the prompt between them, and the resulting dry run would be like the one in my example. If you want to get an idea of how ugly that was: the opening markdown code delimiter had to appear in the preamble, and the closing one in the postamble.

This is a little tricky to do neatly, but anyway: are you suggesting this function should receive the original prompt/system as a string as the second argument? How will you splice a string that doesn't have any imposed structure?

I don't understand the last question. What does it mean for a string to not have an imposed structure?

[karthink]

My suggestion is to modify the context function to something like a prompt/system formatting function. It accepts the alist and the original prompt/system. This would allow for the most flexibility.

Note: I'm adding a prototype for this now, in case you were planning to push more commits in the next half hour.

[karthink]

I don't understand the last question. What does it mean for a string to not have an imposed structure?

I mean that your user prompt has to have a very specific structure if you want to do anything more than adding a preamble + postamble to it via the context.

[daedsidog]

I don't understand the last question. What does it mean for a string to not have an imposed structure?

I mean that your user prompt has to have a very specific structure if you want to do anything more than adding a preamble + postamble to it via the context.

So, for example, doing things like cutting the prompt in half, plugging the context in the middle, that sort of thing?

If so, so long as the user has both the prompt and context as parameters, that's on him to deal with.

[karthink]

Okay, I think I did this semi-cleanly. There is no gptel-context-string-function any more. Instead, there's a gptel-context-wrap-function user option. This function receives two arguments: the message (system or user prompt) and the context alist, and should return a string with everything combined as necessary.

Now you can add a pre/postamble, as well as any kind of interlacing if you can figure out how to split the message. You can even use recursive-edit to pop up a buffer where you can do this interlacing interactively and end with C-M-c (exit-recursive-edit).

EDIT: The default behavior -- prepend/append to user prompt/system message -- is unchanged, and done in gptel-context--wrap-default, the default value of the new user option.

[karthink]

Let me know if this is flexible enough, and if it works without bugs.

[karthink]

So, for example, doing things like cutting the prompt in half, plugging the context in the middle, that sort of thing?

Yes.

If so, so long as the user has both the prompt and context as parameters, that's on him to deal with.

I understand, I'm just wondering if the tradeoffs of imposing this much structure on your user prompts is worth it. At that point you may as well do the whole thing programmatically, using gptel-request or equivalent.

[daedsidog]

Let me know if this is flexible enough, and if it works without bugs.

It's perfect. Thank you.

Do you think perhaps gptel-context--string should also be customizable? Right now it is only used in the default wrapping function. Truth be told, I only ever used the raw string to insert context to the chat buffer conveniently, but now what it can be done automatically, I see no reason to ever modify it (whatever I need I would get by modifying the wrapping function). It could be merged into the default formatting function. EDIT: Same thing applies to gptel-context--insert-file-string.

I understand, I'm just wondering if the tradeoffs of imposing this much structure on your user prompts is worth it. At that point you may as well do the whole thing programmatically, using gptel-request or equivalent.

You're not wrong, but this simplifies it (at least for me) when I want to use the dedicated chat buffer.

[karthink]

Do you think perhaps gptel-context--string should also be customizable? Right now it is only used in the default wrapping function. Truth be told, I only ever used the raw string to insert context to the chat buffer conveniently, but now what it can be done automatically, I see no reason to ever modify it (whatever I need I would get by modifying the wrapping function). EDIT: Same thing applies to gptel-context--insert-file-string.

gptel-context--string, gptel-context--insert-buffer-string and gptel-context--insert-file-string are not customizable, so I don't follow.

It could be merged into the default formatting function.

I guess you meant that they don't need to be separate functions? That's fine, I think they can be separate just for modularity and readability. They can also be advised individually in case someone wants to modify only one part of context-string creation process.

EDIT: Similarly, the code for inserting overlays and inserting files when preparing the context buffer can also be two separate functions. I might refactor it that way in the future, but I think it's fine for now.

[karthink]

You're not wrong, but this simplifies it (at least for me) when I want to use the dedicated chat buffer.

I haven't yet used gptel the way you do -- with context chunks added from various buffers + interaction in the chat buffer that uses these chunks. I'll try it after merging.

[daedsidog]

gptel-context--string, gptel-context--insert-buffer-string and gptel-context--insert-file-string are not customizable, so I don't follow.

They are currently not customizable, but are only used in gptel-context-wrap-function through gptel-context--string. If the user has their own custom wrapping function, they are never used again. If the user wishes only to have access to the context string and nothing more, he is limited to the existing gptel-context--string.

I guess what I am trying to say is that the current way these 3 functions are written do little do suggest that they are just "default suggestions", especially since they use Markdown in the generated strings.

I guess you meant that they don't need to be separate functions? That's fine, I think they can be separate just for modularity and readability. They can also be advised individually in case someone wants to modify only one part of context-string creation process.

Not exactly... but now that you mention that they can be advised as a sort of customization, I guess that's okay. I thought more in the direction if having all 3 of them customizable, and the current functions be their defaults, just like gptel-context-wrap-function currently is.

Anywho, everything else is on point. Cheers!