stanier
commented
4 months ago Personally, I would go a different route. I have similarly asked myself repeatedly over the past year or so what the best way to do simple LLM inference in a cluster compute environment is. But I never felt the explicit need to integrate with the job scheduler for a couple reasons:
- additional complexity with zero inherent value gain
- resource allocation for a short-lived or very-short-lived job such as running inference against a single query makes absolutely no sense, even for a dedicated interactive queue plentiful in GRES. Even if this were not to matter to the parties involved, I'd imagine sysops being naturally miffed by the overhead on the scheduler. Instead of allocating for such short-lived jobs, you'd get better mileage just invoking ollama from an interactive job
- single user inference against dedicated hardware for most LLMs is actually inherently wasteful of resources from my understanding, because you will most likely not be exploiting the benefits of inference batching
- ollama, and the llama.cpp/GGML library it is built upon are more appropriate for enabling inference on consumer-grade hardware rather than a cluster, as I believe is part of the separate intentions of both. Most centers already have HF Transformers, ONNX, and other backends installed as modules for inference against various network architectures, which are built with enterprise hardware and scaling in mind.
Generally I think it is an ill fit and would advise against, only because you have other options that are much more practical which I would highly recommend considering. But I could also be looking at this the wrong way somehow, as admittedly is sometimes characteristic of myself... I'm open to hear what others have to think on this and hear if there are maybe some needs/constraints I'm not taking into account.
edit: On second thought, I may have spoke too hastily about not using llama.cpp/GGML for inference in a cluster environment, as it would be entirely valid in use cases where you have maybe ~4-40GB VRAM to work with. In the same way, ollama would make some sense for convenience. I still can't get behind the use of such short-lived jobs, but if it is a small (less than a dozen nodes) cluster dedicated to one group, it may make sense for them and I have no room to speak in that case.
wwjCMP
Slurm is a utility to manage and schedule workloads on a cluster of computers. Many academic institutions use it for distributing computation.
I was wondering if it would be a good idea to implement an interface that allows use of chat interface with the model loaded via Slurm jobs. This way your request gets queued, and when the computation is done, ollama will pipe the output.
There could be a number of ways to do this :
Allow single-response subcommands which start the server, run the query and kill the server when the output is received, for eg:
integrate the slurm scheduling within Ollama
Write a wrapper around ollama that does point 1.
Im hoping to have a discussion on what the community thinks about this topic.