Interacting with privateGPT (specifically for RAG)

[Aquan1412]

Hello, I'm looking for a way to use privateGPT as a backend for gptel, in order to use its simple RAG pipeline. Specifically, I'm looking for a way to provide additional keywords to the backend (such as "use_context" and "include_sources").

I can generally use privateGPT as an "openai"-like backend with the following configuration:

(gptel-make-openai "privateGPT"
  :protocol "http"
  :host "localhost:8001"
  :models '("private-gpt"))

I tried simply adding the keyword to the configuration:

(gptel-make-openai "privateGPT"
  :protocol "http"
  :host "localhost:8001"
  :use_context t
  :models '("private-gpt"))

However, if I do that, I get the following error message:

Debugger entered--Lisp error: (error "Keyword argument :use_context not one of (:curl-ar...") signal(error ("Keyword argument :use_context not one of (:curl-ar...")) error("Keyword argument %s not one of (:curl-args :models..." :use_context) gptel-make-openai("privateGPT-Context" :protocol "http" :host "localhost:8001" :use_context t :models ("private-gpt")) (progn (gptel-make-openai "privateGPT-Context" :protocol "http" :host "localhost:8001" :use_context t :models '("private-gpt"))) eval((progn (gptel-make-openai "privateGPT-Context" :protocol "http" :host "localhost:8001" :use_context t :models '("private-gpt"))) t) elisp--eval-last-sexp(nil) eval-last-sexp(nil) funcall-interactively(eval-last-sexp nil) call-interactively(eval-last-sexp nil nil) command-execute(eval-last-sexp)

So, is there a way to get this to work? Or am I approaching it completely wrong?

[karthink]

You're going to have to provide more context for me to understand what you're looking for. What do you expect use_context and include_sources to do?

[Aquan1412]

I want to use privateGPT to generate responses based on embeddings I previously generated by "ingesting" several PDFs (in privateGPT lingo). According to the API reference, if I set use_context = true, it should use the embeddings to generate responses based on the ingested papers. Additionally, if I set include_sources = true, it should include the source chunks, on which the generated answers are based.

[kenbolton]

I am an elisp novice reading open issues to learn.

https://github.com/karthink/gptel/blob/8ccdc31b12a1f5b050c6b70393014710f8dbc5c8/gptel-openai.el#L102-L107

I believe inserting:

  :use_context t
  :include_sources t

after line 103 will do what you want but is likely to break other things. This change will add those key/value pairs to the data sent to the OpenAI API through curl.

[karthink]

I believe inserting:

  :use_context t
  :include_sources t

after line 103 will do what you want but is likely to break other things. This change will add those key/value pairs to the data sent to the OpenAI API through curl.

This is correct. I'll need to find some way of adding this via the configuration.

[Aquan1412]

I am an elisp novice reading open issues to learn.

https://github.com/karthink/gptel/blob/8ccdc31b12a1f5b050c6b70393014710f8dbc5c8/gptel-openai.el#L102-L107

I believe inserting:

  :use_context t
  :include_sources t

after line 103 will do what you want but is likely to break other things. This change will add those key/value pairs to the data sent to the OpenAI API through curl.

I just tried your proposed change, and it worked, at least for respecting the given context! Thanks! However, now I also noticed that in order to include the used sources, I also need to adjust the parsing of the response.

Maybe I'll try and create a separate gptel-make-privateGPT based on gptel-make-openai. Now that I roughly know where to look, i shouldn't be too hard.

[karthink]

Maybe I'll try and create a separate gptel-make-privateGPT based on gptel-make-openai. Now that I roughly know where to look, i shouldn't be too hard.

You'll need to write a new struct type gptel-privategpt that inherits from gptel-openai, and three cl-defmethods that specialize on the backend-type gptel-privategpt: gptel--request-data, gptel-curl--parse-stream and gptel--parse-response. We can add it to gptel afterwards. I can help if you have questions.

[Aquan1412]

Maybe I'll try and create a separate gptel-make-privateGPT based on gptel-make-openai. Now that I roughly know where to look, i shouldn't be too hard.

You'll need to write a new struct type gptel-privategpt that inherits from gptel-openai, and three cl-defmethods that specialize on the backend-type gptel-privategpt: gptel--request-data, gptel-curl--parse-stream and gptel--parse-response. We can add it to gptel afterwards. I can help if you have questions.

Great, thanks for the information! I'll see how far I get, and if I encounter any big problems, I'll come back to you.

[Aquan1412]

So, after a bit of trial and error, I managed to create a first working version of gptel-privategpt.

Here are the different definitions: 1) The gptel-privategpt struct:

(cl-defstruct (gptel-privategpt (:constructor gptel--make-privategpt)
                               (:copier nil)
                               (:include gptel-openai))
  use_context include_sources
  ) 

2) The three methods for requesting and parsing of the response:

(cl-defmethod gptel-curl--parse-stream ((_backend gptel-privategpt) _info)
  (let* ((content-strs))
    (condition-case nil
        (while (re-search-forward "^data:" nil t)
          (save-match-data
            (unless (looking-at " *\\[DONE\\]")
              (let* ((response (gptel--json-read))
             (finish-reason (map-nested-elt
                       response '(:choices 0 :finish_reason))))
        (if finish-reason
            ;; finish_reason "stop": stream has ended, therefore put sources at the bottom of the printed text
            (progn
              (setq-local counter 0)
              (setq-local source-string-list (list))
              (while-let ((names (map-nested-elt persistent-sources (list :sources counter :document :doc_metadata :file_name)))
                  (pages (map-nested-elt persistent-sources (list :sources counter :document :doc_metadata :page_label)))
                  )
            (cl-pushnew (format "- %s (page %s)" names pages) source-string-list :test #'string=)
            (setq counter (+ 1 counter)))
              (push (format "\n\nSources:\n%s" (mapconcat (lambda (s) s) (nreverse source-string-list) "\n")) content-strs))
          ;; finish_reason "nil": stream is still ongoing, therefore extract current content
          (let* ((delta (map-nested-elt
                 response '(:choices 0 :delta)))
             (content (plist-get delta :content))
             (sources (map-nested-elt response '(:choices 0)))
             )
            (progn
              ;; sources are only returned as long as finish_reason is "nil", therefore they have to be buffered so that they can be printed once the stream has ended
              (setq-local persistent-sources sources)
              (push content content-strs)))))
          ))
      )
    (error
     (goto-char (match-beginning 0))))
  (apply #'concat (nreverse content-strs))
  ))

(cl-defmethod gptel--parse-response ((_backend gptel-privategpt) response _info)
  (let ((response-string (map-nested-elt response '(:choices 0 :message :content)))
    (sources (map-nested-elt response '(:choices 0)))
    (counter 0)
    (source-string-list (list))
    )
    (while-let ((names (map-nested-elt sources (list :sources counter :document :doc_metadata :file_name)))
        (pages (map-nested-elt sources (list :sources counter :document :doc_metadata :page_label)))
        )
      (cl-pushnew (format "- %s (page %s)" names pages) source-string-list :test #'string=)
      (setq counter (+ 1 counter)))
    (format "%s\n\nSources:\n%s" response-string (mapconcat (lambda (s) s) (nreverse source-string-list) "\n")))
)

(cl-defmethod gptel--request-data ((_backend gptel-privategpt) prompts)
  "JSON encode PROMPTS for sending to ChatGPT."
  (let ((prompts-plist
         `(:model ,gptel-model
       :messages [,@prompts]
       :use_context t
       :include_sources t
           :stream ,(or (and gptel-stream gptel-use-curl
                         (gptel-backend-stream gptel-backend))
                     :json-false))))
    (when gptel-temperature
      (plist-put prompts-plist :temperature gptel-temperature))
    (when gptel-max-tokens
      (plist-put prompts-plist :max_tokens gptel-max-tokens))
    prompts-plist))

Generally the code is working, however I still have an open question: I'd like to make the two new keywords use_context and include_sources configurable when the backend is registered. Currently they are hardcoded to t. How can I access the values I set when I register the backend gptel-make-privategpt?

[karthink]

Thanks! Would you like to raise a PR? I can review the code and we can add it to gptel.

[Aquan1412]

Sure, I just raised the PR. Let me know if there any further changes necessary.

[karthink]

PrivateGPT support has been added.