Using LocalGPT with external server which hosts ollama (via ngrok)

[pfrankov]

^{Originally posted by 0xSynth January 17, 2024} Hi, I can't run ollama mixtral model on my desktop, however I have a server to which I would like to connect (via ngrok), however for some reason I can't, is it a bug? Can you add this?

[pfrankov]

@0xSynth I moved the discussion to an issue. Please provide some more information: how do you run Ollama on your desktop? Have you tried to send regular curl to the server?

[ghost]

Thank you for quick response, I run obsydian with local-gpt plugin on my desktop. The server is running ollama, the server is accessible via a ngrok tunel and it works fine with https://github.com/kghandour/Ollama-SwiftUI, also requests are accessible via an API - I've tried requests via curl.

I had an issue with CORS, but I've managed to resolve it, however the problem still persists. The response I'm getting is

{error: "model 'orca-mini' not found, try pulling it first"} However I'm not able to configure it for some reason in the local-gpt settings, as the refresh button basically does nothing. No request to fetch model list is being sent. Having an input where the default model name could be typed in would help in those kind of situations.

[pfrankov]

Thank you for the additional info! I've just tried with ngrok ngrok http 11434 and it it works for me right from the box:

Please double check the settings, try to remove plugin and install again, try to update Ollama.

Having an input where the default model name could be typed in would help in those kind of situations.

It's pointless to specify a model name using input — if you can't get a list of available models, then Ollama is unavailable.

[ghost]

Supper strange, for me none of those GET /api/tags requests are hitting the server. I only get POST generate when trying to use the plugin in one of the articles I'm writting.

[ghost]

The console does not show any GET /api/tags request from the app being set to the server either.

[ghost]

I have downloaded the orca-mini and as a result I am able to use local-gpt, however the problem still persists - not being able to choose a model in the settings page.

[pfrankov]

Have you tried to kill Ollama client and run Ollama in serve mode OLLAMA_ORIGINS='*' ollama serve?

[ghost]

Yes I did, without OLLAMA_ORIGINS='*' the requests return with 403, for me the issue is that for some reason after updating the API URL, the request to /api/tags is not being sent, as a result I am not able to specify model different than orca-mini which is the default one.

[pfrankov]

the request to /api/tags is not being sent

That's expected: Obsidian's internal requests doesn't shown in Network tab. And errors from it also not visible in Console.

Have you tried with regular http://localhost:11434? It seems that even this is failing with an error (while it shouldn't with with OLLAMA_ORIGINS='*' ollama serve)

[pfrankov]

[ghost]

I was talking about the server side also, on your screenshots they are visible, however for some reason those are not sent to my server. Regarding the localhost, it just an error when I had the Ollama URL set to localhost

[pfrankov]

it just an error when I had the Ollama URL set to localhost

Nothing happen after that?

Let's exclude possible causes one by one: could you try to install and use the same scenario with the BMO Chatbot plugin?

[ghost]

Thank you for your recording, for me this refresh default model does not work for some reason, the request are not being sent to the server, at the same time I am able to run the plugin with orca-mini and requests are hitting the server

[ghost]

Ok, I think we are onto something, with BMO Chatbot I am getting an error

The website when visible with incognito mode looks like this, maybe this is the issue? On the other hand you are also on free plan and it does work for you... Maybe the solution would be to hit the GET api/tags directly without verifying the GET / if ngrok is used, or as they suggest ngrok-skip-browser-warning request?

[ghost]

Ok, in the end I've decided to fork your plugin and change the default model to the one I like to use - works fine, however the issue still remains.

[quantarion]

I would drop support for the Ollama protocol and retain only OpenAI. All engines used for locally deploying LLMs have OpenAI compatibility modes in one form or another, including Ollama. The development effort would be much more useful if focused on adding new features rather than debugging VPNs, firewalls, tunnels, and similar issues. Regarding tunneling, I confirm that SSH tunneling works fine with Ollama (in OpenAI compatibility mode), Exllama (TabbyAPI), Aphrodite, and llama.cpp (in OpenAI mode).