Models can be loaded with different accelerators, for example, llama2-70b can be located with 2 A100 80GB or 4 A100 40 GB, we should support this. And usually high-end GPUs will be stockout frequently, this can help improve the SLO of services.
Why is this needed:
Cost saving and SLO consideration.
Completion requirements:
This enhancement requires the following artifacts:
[ ] Design doc
[ ] API change
[x] Docs update
The artifacts should be linked in subsequent comments.
kerthcet
commented
1 month ago
What would you like to be added:
Models can be loaded with different accelerators, for example, llama2-70b can be located with 2 A100 80GB or 4 A100 40 GB, we should support this. And usually high-end GPUs will be stockout frequently, this can help improve the SLO of services.
Why is this needed:
Cost saving and SLO consideration.
Completion requirements:
This enhancement requires the following artifacts:
The artifacts should be linked in subsequent comments.