InftyAI / llmaz

☸️ Easy, advanced inference platform for large language models on Kubernetes
Apache License 2.0
13 stars 5 forks source link

Always download the model weights when pod starts #88

Closed kerthcet closed 3 weeks ago

kerthcet commented 3 weeks ago

What happened:

Because of https://github.com/InftyAI/omnistore/issues/12, we'll always download the model weights even we cached them in the host machine.

What you expected to happen:

When model weights downloaded, we should not reload again.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

kerthcet commented 3 weeks ago

/milestone v0.1.0

kerthcet commented 3 weeks ago

Let's not cache the models to the host right now because it makes little sense.