gptel--make-openai with custom endpoint (anyscale) not recognizing model

LazerJesus commented 5 months ago

Hey 👋 ive been playing around with different endpoints and one ive had good results with elsewhere is mixtral. Sadly I dont have the 48gb required to run it locally 🤷 anyscale offers it as an api and its a drop in replacement for openai's api. in python you can just use the official Python OpenAI library, only replacing the URL and KEY. ive tried that, any new session now starts with the correct backend but i cant select the model. what am i doing wrong?

(setq-default
   gptel-backend (gptel--make-openai
                  :name "Anyscale"
                  :host "api.endpoints.anyscale.com"     
                  :endpoint "v1"
                  :key  "xxx"
                  :models '("mistralai/Mixtral-8x7B-Instruct-v0.1")
                  )
   gptel-model "mistralai/Mixtral-8x7B-Instruct-v0.1"
   )

karthink commented 5 months ago

It looks like there's a typo in your configuration, it should be gptel-make-openai, not gptel--make-openai.

LazerJesus commented 5 months ago

ok yes that allowed my to chose the provider. funny, i copied that snippet out of some issue. sadly it didnt work. response cant be parsed.

karthink commented 5 months ago

Hang on, I'm pretty sure we can get it to work. If you are interested: set the variable gptel--debug to t, then try sending a query. A buffer containing the response should pop up. You can paste the contents of that buffer here after removing your API key from it.

On Fri, Jan 12, 2024, 4:26 AM Finn @.***> wrote:

ok yes that allowed my to chose the provider. funny, i copied that snippet out of some issue. sadly it didnt work. response cant be parsed.

— Reply to this email directly, view it on GitHub https://github.com/karthink/gptel/issues/177#issuecomment-1889050737, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACBVOLGN2TUQ24Y4YRDR3GTYOETYNAVCNFSM6AAAAABBWMUMBOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBZGA2TANZTG4 . You are receiving this because you commented.Message ID: @.***>

LazerJesus commented 5 months ago

gptel--debug is t *Messages

Querying Anyscale...
("--location" "--silent" "--compressed" "--disable" "-XPOST" "-w(4267eddb21697a77a895875e606cbe34 . %{size_header})" "-m300" "-D-" "-d{\"model\":\"mistralai/Mixtral-8x7B-Instruct-v0.1\",\"messages\":[{\"role\":\"system\",\"content\":\"You are a large language model living in Emacs and a helpful assistant. Respond concisely.\"},{\"role\":\"user\",\"content\":\"Hello World\"}],\"stream\":false,\"temperature\":1.0}" "-HContent-Type: application/json" "https://api.endpoints.anyscale.comv1")

ChatGPT response error: (((4267eddb21697a77a895875e606cbe34 . 0)) Could not parse HTTP response.) Could not parse HTTP response.

*buffer

HTTP/2 401 
content-type: application/json
content-length: 37
date: Fri, 12 Jan 2024 19:52:29 GMT
server: uvicorn
x-request-id: 722bc1e9-f158-4905-81dc-6a95ee2cd606
x-cache: Error from cloudfront
via: 1.1 cba85c100a294e3d2360b9852eb535fc.cloudfront.net (CloudFront)
x-amz-cf-pop: WAW51-P4
x-amz-cf-id: _mmN_2idH6t19tE2VGCNJvNsxi7vf0fKm8SLJSujE7hr53T_chI5tQ==

{"err":"Auth token must be provided"}(a7ebb812780a8e7c6c1f78ad4c857c26 . 371)

so authtoken is missing. i am setting :key tho.

karthink commented 5 months ago

EDIT: Please ignore this in favor of my next message below.

(setq-default
   gptel-backend (gptel-make-openai
                  :name "Anyscale"
                  :host "api.endpoints.anyscale.com"
                  :endpoint "v1"
                  :header (lambda () `(("Authorization" . ,(concat "Bearer " (gptel--get-api-key)))))
                  :key  "xxx"
                  :models '("mistralai/Mixtral-8x7B-Instruct-v0.1"))
   gptel-model "mistralai/Mixtral-8x7B-Instruct-v0.1")

LazerJesus commented 5 months ago

(setq-default
   gptel-backend (gptel-make-openai
    "anyscale"
    :host "api.endpoints.anyscale.com/"
    :endpoint "v1"
    :header (lambda () `(("Authorization" . ,(concat "Bearer " (gptel--get-api-key)))))
    :key "x"
    :models '("mistralai/Mixtral-8x7B-Instruct-v0.1"))
   gptel-model "mistralai/Mixtral-8x7B-Instruct-v0.1")

host must end in "/"
i have to omit :word, otherwise i get this error when evaluating the expression: gptel-make-openai: Keyword argument Anyscale not one of (:header :models :stream :key :host :protocol :endpoint)

i get a new error.

ChatGPT response error: (((31fc0bff67b7109e9f2b541e2c33c2fe . 0)) Could not parse HTTP response.) Could not parse HTTP response.

messages

("--location" "--silent" "--compressed" "--disable" "-XPOST" "-w(f95cd7bd409426d3343340f166313f34 . %{size_header})" "-m300" "-D-" "-d{\"model\":\"mistralai/Mixtral-8x7B-Instruct-v0.1\",\"messages\":[{\"role\":\"system\",\"content\":\"You are a large language model living in Emacs and a helpful assistant. Respond concisely.\"},{\"role\":\"user\",\"content\":\"\"}],\"stream\":false,\"temperature\":1.0}" "-HContent-Type: application/json" "-HAuthorization: Bearer esecret_XXX" "https://api.endpoints.anyscale.com/v1")

buffer

HTTP/2 404 
content-type: application/json
content-length: 22
date: Thu, 18 Jan 2024 15:02:37 GMT
server: uvicorn
x-request-id: c58e6ef6-2933-430f-9808-efa9433e33af
x-cache: Error from cloudfront
via: 1.1 747bd05dc55fe028e8f6b18dbc8ee720.cloudfront.net (CloudFront)
x-amz-cf-pop: WAW51-P4
x-amz-cf-id: xiMftwGDpFRq6iH14V1G3irVd3lh1yyRZDYdhLdjUyMHIOQJwlYbuw==

{"detail":"Not Found"}(f95cd7bd409426d3343340f166313f34 . 371)

why i am getting cloudfront blocked i sure dont know.

karthink commented 5 months ago

i have to omit :word, otherwise i get this error

My bad, sorry.

I looked up the Anyscale API docs. The following should work:

(setq-default
   gptel-backend (gptel-make-openai
    "anyscale"
    :host "api.endpoints.anyscale.com"
    :header (lambda () `(("Authorization" . ,(concat "Bearer " (gptel--get-api-key)))))
    :key "x"
    :models '("mistralai/Mixtral-8x7B-Instruct-v0.1"))
   gptel-model "mistralai/Mixtral-8x7B-Instruct-v0.1")

Since there are many services offering OpenAI-compatible APIs, I think I need to make it easier to define an OpenAI-compatible backend with gptel.

LazerJesus commented 5 months ago

🥳

LazerJesus commented 5 months ago

with this gptel becomes even more important to my daily work. i love it! i got a couple of questions, not sure if here is the best spot, so let me know if i should start a new issue about any of these.

managing multiple backends

at the moment i toggle models by keybinding custom expressions. like this

  (defun set-gpt4-universal ()
    (interactive)
    (setq gptel-model "gpt-4-1106-preview")
    (setq chatgpt-shell-chatgpt-model-version "gpt-4-1106-preview"))

  (defun gptel-with-argument ()
    (interactive)
    (setq current-prefix-arg '(4))
    (call-interactively #'gptel))

  (general-menu "ai" "a"
    "r" '(gptel-send :wk "gptel region")
    "R" '(gptel-send-with-argument :wk "gptel region prefixed")
    "a" '(gptel-abort :wk "abort")
    "C-<return>" '(gptel-send-with-argument  :wk "gptel send prefixed")
    "3" '(set-gpt3-universal  :wk "gpt 3.5 turbo")
    "4" '(set-gpt4-universal  :wk "gpt 4"))

but with multiple backends that is becoming tedious. is there a better way?

whats the fastest way to provide one-off directives

i often highlight text regions and want to write just a few words as directive. using the gpt-system buffer (reached via h h from the gptel minibuffer) is about 8 key hits for me. open minibuffer, h h, closing minibuffer, clearing text, c-c c-c. is there a faster way?

managing directives / directives per model?

different models have different strengths and weaknesses, therefore i use them for different reasons. for example, i use gpt4 for coding and mixtral for summarizing large bodies of text. that requires different directives. is it possible to have directives per model? how would i do that?

karthink / gptel