Closed kerthcet closed 3 weeks ago
/kind question
This maybe useful for really really big models because some nodes are dedicated for them, like with high end GPUs preset.
However, this can be solved with cache systems like fluid, llmaz should not touch this because caching to the local host makes little sense. /close
What would you like to be added:
Support to downloads models in prior of inference service, and it's useful when new nodes privisioned.
Why is this needed:
Accelerate the service startup, however, if the cluster has P2P accelerating system like dragonfly, this seems not that necessary.
Completion requirements:
This enhancement requires the following artifacts:
The artifacts should be linked in subsequent comments.