together.ai "ChatGPT response error: ((HTTP/2 401) Malformed JSON in response.) Malformed JSON in response"

[trekta]

hey I am trying to use gptel with together.ai thru the gptel-make-openai function. When calling gptel-send using this backend I get this error:

ChatGPT response error: ((HTTP/2 401) Malformed JSON in response.) Malformed JSON in response

this is my use-package declaration; that so far worked well for the other two defined backends:

(use-package gptel
  :straight t
  :hook ((gptel-post-response-functions . gptel-end-of-response))
  :config
  (setq-default gptel-model "llama2-uncensored" ; Pick your default model
        gptel-backend (gptel-make-ollama
                   "Ollama" ; Any name of your choosing
                   :host "localhost:11434" ; Where it's running
                   :stream t               ; Stream responses
                   :models '("mistral:latest"
                     "llama2-uncensored"
                     "llama2-uncensored:7b-chat"))
        gptel-max-tokens 500
        gptel-stream t
        gptel-default-mode 'org-mode
        )
  (gptel-make-gpt4all "GPT4All"
              :protocol "http"
              :host "localhost:4891"
              :models '("mistral-7b-openorca.Q4_0.gguf"
                "rift-coder-v0-7b-q4_0.gguf"))

  ;; Together.ai offers an OpenAI compatible API
  (gptel-make-openai "TogetherAI" ;Any name you want
             :host "api.together.xyz"
             :key "xxxxxxxxxredactedxxxxxxxxxxx"  ;can be a function that returns the key
             :models '(;; has many more, check together.ai
                   "mistralai/Mixtral-8x7B-Instruct-v0.1"
                   "codellama/CodeLlama-13b-Instruct-hf"
                   "codellama/CodeLlama-34b-Instruct-hf"
                   "togethercomputer/llama-2-70b-chat")))

any help is appreciated as I cant rule out errors on my side. thanks in advance

[karthink]

1. Run (setq gptel--debug t)
2. Try to use together.ai
3. A message will be printed to the messages buffer, and a buffer will pop up with the response (including the error). Please paste those here after removing your API key from them.

[trekta]

Thanks for the quick response! Here is the output of gptel-error:

HTTP/2 401 date: Wed, 24 Jan 2024 04:52:40 GMT content-length: 15 access-control-allow-origin: * set-cookie: Host-next-auth.csrf-token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx%xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx; Path=/; HttpOnly; Secure; SameSite=Lax set-cookie: Secure-next-auth.callback-url=https%3A%2F%2Fapi.together.xyz; Path=/; HttpOnly; Secure; SameSite=Lax etag: "e8oxfpna2sf" cf-cache-status: DYNAMIC server: cloudflare cf-ray: 84a5ab15de48901e-FRA alt-svc: h3=":443"; ma=86400

Missing API key(xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx . 560)

[karthink]

Looks like the API key isn't being read.

How are you setting it here:

  :key "xxxxxxxxxredactedxxxxxxxxxxx"  ;can be a function that returns the key

1. Are you using a string, a variable, or a function?
2. Can you do the following:

• In any buffer, switch to the together.ai backend (from the transient menu or however you switch backends)
• Run (gptel-backend-key gptel-backend)
• Run (gptel--get-api-key) The result of the first form should be whatever you set :key to above, and the second should give you the actual API key. Does this work?

[trekta]

the :key value is set as string/plain text. I ran both commands thru eval-expression and they both output the correct api key as plain text surrounded by quotation marks.

[karthink]

I tried setting the together.ai API key to a string (as you do) but couldn't reproduce the issue. My config is identical to yours.

Could you set gptel--debug to true again, and paste whatever is added to the messages buffer here? Instead of the gptel-error buffer, I mean. Be sure to remove your API key from the message.

[trekta]

this is the message buffer after setting gtpel--debug t

Querying TogetherAI... ("--disable" "--location" "--silent" "--compressed" "-XPOST" "-y300" "-Y1" "-D-" "-w(bd9e8774d6f4e1e1d4225af5d65702dd . %{size_header})" "-d{\"model\":\"togethercomputer/llama-2-70b-chat\",\"messages\":[{\"role\":\"system\",\"content\":\"You are a large language model living in Emacs and a helpful assistant. Respond concisely.\"},{\"role\":\"user\",\"content\":\"hi\"}],\"stream\":false,\"temperature\":1.0,\"max_tokens\":500}" "-HContent-Type: application/json" "https://api.together.xyz/v1/chat/completions") ChatGPT response error: ((HTTP/2 401) Malformed JSON in response.) Malformed JSON in response

correct me if I am wrong but I dont see an API key in that query :/

[karthink]

Your API key is not being sent at all. Please update gptel.

[trekta]

Updating did the job. I feel stupid now. Sorry to waste your time and thanks for the quick and easy help you provided. not sure if related but jsonrpc updated aswell..

[karthink]

Updating did the job. I feel stupid now. Sorry to waste your time and thanks for the quick and easy help you provided. not sure if related but jsonrpc updated aswell..

No worries, I should have pegged it after the first couple of messages. jsonrpc is not a dependency of gptel (direct or transitive) so it's unrelated.

[karthink]

(I really need to add better logging to gptel to catch these issues, this serves as a good reminder.)

[ryankask]

I'm seeing the same issue. I think it started after adding the key default to gptel-make-openai.

gptel-backend
#s(gptel-openai "Anyscale" "api.endpoints.anyscale.com" #<subr F616e6f6e796d6f75732d6c616d626461_anonymous_lambda_9> "https" t "/v1/chat/completions" nil ("meta-llama/Llama-2-7b-chat-hf" "meta-llama/Llama-2-13b-chat-hf" "meta-llama/Llama-2-70b-chat-hf" "codellama/CodeLlama-34b-Instruct-hf" "mistralai/Mistral-7B-Instruct-v0.1" "mistralai/Mixtral-8x7B-Instruct-v0.1" "HuggingFaceH4/zephyr-7b-beta" "Open-Orca/Mistral-7B-OpenOrca") "https://api.endpoints.anyscale.com/v1/chat/completions")

(gptel-backend-key gptel-backend)
nil

[karthink]

@ryankask Your gptel-backend does not include a key field.

1. Make sure you've updated gptel
2. Paste the exact configuration you used to install this backend here.

[ryankask]

@karthink You are right. I misread a recent commit and thought that the key argument had a default (ie (key 'gptel-api-key) ) but it’s the headers argument that now has a default.

I use the gptel-api-key function for the custom open ai backends I’ve created so that could be a useful default too.

[karthink]

I use the gptel-api-key function for the custom open ai backends I’ve created so that could be a useful default too.

This cannot be a default since there are openAI-compatible local APIs (llama.cpp and gpt4all among others) that do not require an api key. Setting it to gptel-api-key by default will cause an error of the opposite kind from yours.