Add ability to listen on unix socket

ollama / ollama

Get up and running with Llama 3, Mistral, Gemma 2, and other large language models.

https://ollama.com

MIT License

77.07k stars 5.8k forks source link

Add ability to listen on unix socket #739

Open Sir-Photch opened 9 months ago

Sir-Photch commented 9 months ago

For usecases when there is a reverse-proxy in front of ollama, it would make sense to enable listening on a unix socket instead of a port, that is /path/to.socket. This would reduce unnecessary TCP/IP-overhead between reverse proxy and ollama, as well as enable security configurations with unix file permissions.

jmorganca commented 9 months ago

Thanks for the issue!

Stebalien commented 6 months ago

So, I'm looking into implementing this (if that's OK). My plan was to reuse the OLLAMA_HOST variable but model it after DOCKER_HOST:

Keep support for IP:PORT and IP.
Add support for tcp://IP:PORT.
Add support for unix:///path/to/socket.

Thoughts?

(eventually I'd like to add systemd socket activation)

silvergasp commented 4 months ago

@Stebalien how are you getting on with this, I'm also interested and potentially can justify putting some time towards it if you need some help?

Stebalien commented 4 months ago

I was waiting for feedback but missed the :+1:. I have no plans to tackle it immediately (I'll wait till I get a free weekend) so go for it.

RangerMauve commented 1 month ago

This'd be nice to have. I was thinking it'd be useful in combination with some sort of listener from systemd to start the server on-demand.

fvillena commented 4 weeks ago

It would be helpful for security concerns on multi-user machines.