Ollama (or other OpenAI API compatible local LLMs) support

[micheleoletti]

Knowing that Ollama server supports OpenAI API(https://ollama.com/blog/openai-compatibility), the goal is to point Cursor to query the local Ollama server.

My setup is pretty simple:

• Macbook Air M1
• Ollama running locally + llama2

I added a llama2 model, set "ollama" as API key(not used but needed apparently), and overridden the base URL to point to localhost.

But it does not work:

If I try to verify the API key it seems like it cannot reach localhost:

But if I try the provided test snippet in the terminal, it works correctly:

So it seems like Cursor internal service is not able to perform the fetch to localhost.

Is there something conceptually wrong with my plan and its implementation? Did anybody manage to make this configuration work?

[tosh]

I also ran into this just now, is there a way to make it work?

[andreaspoldi]

+1

[TazzerMAN]

Same here :/

[kcolemangt]

Ollama Phi3 in the ⌘ K 🚀

Takes some effort but it's fast and works well!

[micheleoletti]

@kcolemangt looks amazing! what did you put in the settings to do that? I can't get it to work 🤔

[kcolemangt]

Had to write a custom router and point Cursor’s OpenAI Base URL to that. Thinking of releasing it—want to try?

[micheleoletti]

I see... yeah I'd be interested in trying that out! On what hardware are you running ollama by the way?

[kcolemangt]

released llm-router. lmk how it works for you @micheleoletti

[yeralin]

Cursor does not like when you specify a port in the "Override OpenAI Base URL" field.

If you serve ollama on the default http (80) port, it starts working: OLLAMA_ORIGINS=* OLLAMA_HOST=127.0.0.1:80 (sudo -E) ollama serve

Then you can put http://localhost/v1 under "Override OpenAI Base URL" in Cursor.

---

UPD: For some reason, Cursor is trying to hit /v1/models endpoint which is not implemented in Ollama:

[GIN] 2024/05/30 - 11:45:40 | 404 |     200.375µs |       127.0.0.1 | GET      "/v1/models"

That causes this error message:

---

UPD 2: Even after creating a compatible /v1/models endpoint in Ollama, Cursor still refuses to work:

~ curl -k http://127.0.0.1/v1/models
{"data":[{"created":1715857935,"id":"llama3:latest","object":"model","owned_by":"organization-owner"}],"object":"list"}

[GIN] 2024/05/30 - 15:55:17 | 200 |    1.221584ms |       127.0.0.1 | GET      "/v1/models"

In Cursor's Dev Tools I get:

ConnectError: [not_found] Model llama3:latest not found

Tried both llama3 and llama3:latest.

Seems like Cursor has something hardcoded for localhost/127.0.0.1 address.

[flight505]

there is the repo https://github.com/ryoppippi/proxy-worker-for-cursor-with-ollama which is an option (but not for offline use). The readme also points out there is direct communication to cursor server, which is likely why cursor haven't enabled an easy to use option for true local LLM.