Download models in prior

[kerthcet]

What would you like to be added:

Support to downloads models in prior of inference service, and it's useful when new nodes privisioned.

Why is this needed:

Accelerate the service startup, however, if the cluster has P2P accelerating system like dragonfly, this seems not that necessary.

Completion requirements:

This enhancement requires the following artifacts:

• [x] Design doc
• [ ] API change
• [x] Docs update

The artifacts should be linked in subsequent comments.

[kerthcet]

/kind question

[kerthcet]

This maybe useful for really really big models because some nodes are dedicated for them, like with high end GPUs preset.

[kerthcet]

However, this can be solved with cache systems like fluid, llmaz should not touch this because caching to the local host makes little sense. /close