ENH: enable multiple models running on a single device

[UranusSeven]

Is your feature request related to a problem? Please describe

Currently, our system assigns each model to a unique GPU device. While this approach ensures protection against out-of-memory (OOM) errors, it isn't optimal in terms of resource allocation and can lead to underutilization of GPU memory.

To enhance our GPU resource management and utilization:

• Implement fine-grained GPU memory control mechanisms.
• For Large Language Models (LLM):
  • Consider the following components for GPU memory allocation:
    1. Model weights
    2. Activations
    3. Key-value (kv) cache
• For non-LLM models:
  • GPU memory allocation should take into account:
    1. Model weights
    2. Activations

By calculating and monitoring the GPU memory requirements of each model, we can intelligently allocate multiple models to run on a single GPU device when memory allows. This will ensure efficient GPU resource utilization without compromising on performance or risking OOM errors.

[BradKML]

Would like to see this being developed further as default or easily manageable through Docker. Currently this feature (use with Docker) is not in the docs, and alternatives like Ollama does not include rerankers. Cross-referencing these two issue: https://github.com/xorbitsai/inference/issues/503 https://github.com/xorbitsai/inference/issues/1228

[github-actions[bot]]

This issue is stale because it has been open for 7 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 5 days since being marked as stale.