Proposal: Launching LLM server as a daemon

[blmarket]

Proof-of-concept: https://github.com/blmarket/llm-ls

Hi,

I'm wondering llm-ls can incorporate dedicated LLM server provider within LSP server, preferably as a shared instance via daemonize. Idea was inspired by bazel client/server.

It works as following:

• There's a lock file to ensure only 1 LLM server is running
• If there's no usage of LLM (i.e. LSP exits) for 10 seconds, LLM server is terminated
  • There is a heartbeat to let LLM process know there is a client
• Multiple LSP instances will share single LLM server

Still many things are hardcoded (such as LLM path, various file path, model path), but it's usable with existing llm.XXX editor plugins.

Want to know whether this is something llm-ls would like to support as one of its backends, or better be kept as different project.

[blmarket]

I used my prototype since then. Lessons learned - llm-ls make too many requests and llama.cpp server did not support cancellation. I had to implement my own lock to make the server process only the latest request.

As it is beyond the proof-of-concept I'll keep this version separated. Maybe I better just fork as these features may not align with huggingface's interest?

[McPatate]

Hello @blmarket thank you for opening the issue. This is definitely interesting to me, as it simplifies installation (to some extent) and operation on you local machine (cf my comment in https://github.com/huggingface/llm.nvim/issues/87).

I would be happy for you to contribute your work on this topic, if possible, in the form of small incremental PRs.