Accelerate the model weight loading in kubernetes

InftyAI / llmaz-dashboard

A web console for llmaz.

https://github.com/InftyAI/llmaz

Apache License 2.0

3 stars 5 forks source link

Open kerthcet opened 12 months ago

kerthcet commented 12 months ago

Because models are quite big with GB size, we should think of an efficient way to load models, e.g.

localPath for experiment, but won't work in a multi nodes cluster
fluid, which can help to speed up the data accessing by caching the data in advance
NAS

kerthcet commented 12 months ago

For the first trial, let's use localPath with a pvc mounted. :)