Spurious OpenAI API Key requirement for Llama.cpp

[karthink]

Discussed in https://github.com/karthink/gptel/discussions/169

^{Originally posted by **eastbowl** January 4, 2024} Hi, This is probably a very naive question (I'm a novice and non-programmer). I'm trying to use your module/package with a llama server on my computer. When invoking "M-x gptel" I am asked for my OpenAI API key. (My API key has no credits, so it fails to permit any access). My questions is, Why is it necessary to even contact OpenAI when using a completely different LLM on my own computer? I'm sure you have a good reason for it. I'm just curious what that reason is. It would help me understand things (possibly). Thanks for your time.

[karthink]

@eastbowl: I updated the llama.cpp instructions in the README. Basically you have to specify that no key is to be used for Llama (:key nil) when defining the backend. Please try it and let me know if it works correctly now.

[eastbowl]

@karthink: I updated my init by adding ":key nil" as instructed in the README. It stilled asked for my ChatGPT API key, unfortunately.

[karthink]

(use-package gptel
  :if (is-mbp-p)
  :config
  (setq-default gtpel-backend
                (gptel-make-openai      ;Not a typo, same API as OpenAI
                 "llama-cpp"            ;Any name
                 :stream t              ;Stream responses
                 :protocol "http"
                 :host "127.0.0.1:8080" ;Llama.cpp server location, typically localhost:8080 for Llamafile
                 :key nil               ;No key needed
                 :models '("test")))) ;Any names, doesn't matter for Llama

^ Just checking, does your config look like this?

[eastbowl]

Yes, exactly.

[karthink]

Hmm, I'm not able to reproduce this behavior. It works correctly once I set :key to nil.

[karthink]

While I try to figure this out, you could try using a dummy key: :key "something". It doesn't matter since it's not used by Llama.

[eastbowl]

I tried =:key "nothing"=, but still asking for API key. Let me know if I can be of any further help. Happy to do so.

[karthink]

Okay, should actually be fixed now. Please update, and don't hesitate to reopen this topic if the issue persists. I'm known to declare things fixed before they're tested!

[eastbowl]

I hate to say it, but it's still happening. I deleted the gptel folders in .emacs.d/straight/build and .emacs.d/straight/repos and then re-ran my config. Same behaviour (asking for my OpenAI key).

I'll paste my config below again:

(use-package gptel :if (is-mbp-p) :config (setq-default gtpel-backend (gptel-make-openai ;Not a typo, same API as OpenAI "llama-cpp" ;Any name :stream t ;Stream responses :protocol "http" :host "localhost:8080" ;Llama.cpp server location, typically localhost:8080 for Llamafile :key nil :models '("test")))) ;Any names, doesn't matter for Llama

Maybe this is just something on my end that is messing with things?

[karthink]

Again, your configuration looks fine. You can delete the :key nil line now, BTW.

I hate to say it, but it's still happening.

No worries, we'll figure it out. Reopening the issue for now.

[karthink]

On that note, I hope this issue isn't keeping you from using Llama. (You can enter anything (or nothing at all) when it asks you for an API key and the Llama connection should work normally afterwards.)

[eastbowl]

I can't use Llama with your package, if that's what you mean. After I enter a key (anything) if I submit a prompt, it will be rejected as being invalid. It is clearly contacting OpenAI. Maybe that is new info? Anyway, thanks for looking into it.

[karthink]

I can't use Llama with your package, if that's what you mean. After I enter a key (anything) if I submit a prompt, it will be rejected as being invalid. It is clearly contacting OpenAI. Maybe that is new info?

That's new info indeed. This is even more puzzling now.

[karthink]

I'm now doubting if the use-package block you are using is being loaded at all. Could you try the following:

1. Run M-x eval-expression and press return
2. Type in gptel-backend and press return
3. Paste the output here.

[eastbowl]

Ok, here is the message after doing that:

s(gptel-openai "ChatGPT" "api.openai.com" #[0 "��� PBC�" ["Authorization" "Bearer " gptel--get-api-key] 3] "https" t "/v1/chat/completions" gptel-api-key ("gpt-3.5-turbo" "gpt-3.5-turbo-16k" "gpt-4" "gpt-4-1106-preview") "https://api.openai.com/v1/chat/completions")

[karthink]

Your use-package configuration is not being loaded by Emacs.

What does the function is-mbp-p in :if (is-mbp-p) do? Does it return true on this computer?

[eastbowl]

Yes it does return true - it's just a test to see if we're running on my macbook.

[karthink]

Very strange. Your use-package block for gptel is clearly not being read, or being overriden somehow. Is gptel-backend still set to the wrong value after you manually evaluate your use-package block? (C-M-x or C-x C-e at the end of the block)

[eastbowl]

My value for gptel-backend is as follows (pasted from helpful variable: buffer:

gtpel-backend is a variable without a source file.

Value

s(gptel-openai "llama-cpp" "localhost:8080" nil "http" t "/v1/chat/completions" nil

    ("test")
    "http://localhost:8080/v1/chat/completions")

References Could not find source file.

Find all references

Debugging Forget

[karthink]

Okay, that looks good.

What happens when you run M-x gptel now?

[eastbowl]

It opens the chat buffer, but when I enter a prompt I get the same error I was getting before:

ChatGPT error: (HTTP/2 401) Incorrect API key provided: d. You can find your API key at https://platform.openai.com/account/api-keys.

[eastbowl]

After I did the above, I quit and then reloaded emacs. Now when I run gptel it asks for my OpenAI key (I had earlier just typed in "d" when asked for my key, as you can see in the error message). The variable gptel-backend is as I posted above (so use-package is loading it, I think).

[bapfeld]

I ran into similar issues when trying to load using use-package. I'm not an expert in how loading works with that (or in elisp generally) but I found that setting gptel-backend using either :init or :config blocks failed. I was able to resolve using a :custom block instead. This doesn't really explain what is happening or why, but if you're committed to using use-package then an easy solution is as simple as:

(use-package gptel
    :custom
    (gptel-backend (gptel-make-openai
                    "llama-cpp"
                    :stream t
                    :protocol "http"
                    :host "localhost:8080"
                    :key nil
                    :models '("test")))
    )

[karthink]

I ran into similar issues when trying to load using use-package. I'm not an expert in how loading works with that (or in elisp generally) but I found that setting gptel-backend using either :init or :config blocks failed. I was able to resolve using a :config block instead. This doesn't really explain what is happening or why, but if you're committed to using use-package then an easy solution is as simple as:

(use-package gptel
    :custom
    (gptel-backend (gptel-make-openai
                    "llama-cpp"
                    :stream t
                    :protocol "http"
                    :host "localhost:8080"
                    :key nil
                    :models '("test")))
    )

Thank you for this info. I'm assuming you used setq-default and not setq to set the default backend when placing this code in the :init or :config blocks? (Using setq will not work since gptel-backend has a buffer-local value.)

[bapfeld]

I just confirmed that it does work for me using setq-default in a config: but NOT an init: block. I could have sworn I tested that.

[eastbowl]

Sorry to resurface this but I found the "solution" to my issue so thought I would post it. I noticed in the ChatGPT buffer that the top-left part of the buffer has two "buttons": one is "[Send: buffer]" and the other is "[gpt-3.5-turbo]". I pressed the [Send: buffer] button and noticed an option (-m) to choose the model. I hit that option and selected "llama.cpp". The second button now says [test] (which corresponds to the name given to it in the configuration block), and bingo, it is working.

Perhaps this info will be useful to someone.

[nameiwillforget]

I ran into similar problems. My config is:

(use-package gptel
  :config
  (gptel-make-openai "testai"          ;Any name
    :stream t                             ;Stream responses
    :protocol "http"
    :host "localhost:8080"                ;Llama.cpp server location
    :models '("test")
    :key nil)

  (gptel-make-openai "desktop"          ;Any name
    :stream t                             ;Stream responses
    :protocol "http"
    :host "1.0.0.8:8080"                ;Llama.cpp server location
    :models '("test")
    :key nil)

  (setq-default gptel-backend
                (gptel-make-openai "testai")
                gptel-model   "local"))

If I use this config, call gptel-menu and choose one of these two AIs I get the response

testai response error: ((HTTP/2 401) invalid_request_error) You didn't provide an API key. You need to provide your API key in an Authorization header using Bearer auth (i.e. Authorization: Bearer YOUR_KEY), or as the password field (with blank username) if you're accessing the API from your browser and are prompted for a username and password. You can obtain an API key from https://platform.openai.com/account/api-keys.

Using :custom instead of :config led to the arguments not read at all.

[karthink]

@nameiwillforget

(use-package gptel
  :config
  (gptel-make-openai "desktop"          ;Any name
    :stream t                             ;Stream responses
    :protocol "http"
    :host "1.0.0.8:8080"                ;Llama.cpp server location
    :models '("test")
    :key nil)

  (setq-default gptel-backend
                (gptel-make-openai "testai"          ;Any name
                  :stream t                             ;Stream responses
                  :protocol "http"
                  :host "localhost:8080"                ;Llama.cpp server location
                  :models '("test")
                  :key nil)
                gptel-model "test"))

I think the setq-default description in the README is confusing users, I will change it to be more explicit.

[karthink]

I've made the code in the README more explicit to avoid this confusion.

[nameiwillforget]

I've tried the new code, but the problem remains. It's also not the case that another model was chosen, I've always looked at -m to ensure it was one of the local models.

Edit: apparently, the evaluation didn't update the menu defaults at the first attempt, now it changed :testai to :testai:test and I don't get the API error anymore. I get a malformed JSON error, but I'll put this into the corresponding issue.

[karthink]

I get a malformed JSON error, but I'll put this into the corresponding issue.

@nameiwillforget Did you create or add to an issue about this error?

[nameiwillforget]

"karthink" @.***> writes:

I get a malformed JSON error, but I'll put this into the corresponding issue.

@nameiwillforget Did you create or add to an issue about this error?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

No, I actually found the source of that error, it was a bad config on my part.