search

opea-project / GenAIInfra

Containerization and cloud native suite for OPEA
Apache License 2.0
25 stars 48 forks source link

Support production level volume types for model data #128

Closed lianhao closed 1 month ago

lianhao commented 3 months ago

Currently, our k8s installation (either helm-charts or manifest) have hostPath volume type to store the model data to save model download time. This is NOT ready for production environment, and is NOT suitable for scale outing with multiple nodes.

In production environment, users often use persistent volume(PV/PVC) to store those kind of data. We should give users the options to choose which volume type they want to use.

For PVs, it should have annotation "pv.beta.kubernetes.io/gid: " so non root user in pod can have write access to the volume mounted from this PV. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-persistent-volume-storage/#access-control as reference.

endomorphosis commented 2 months ago

I agree with this, and this is one of the reasons why previous to OPEA I built huggingface model manager, that pulls models from ipfs / s3 / huggingface, and evicts models as needed.

daisy-ycguo commented 2 months ago

@lianhao can this issue be closed?

lianhao commented 2 months ago

nope, we didn't have time to resolve this in M3. should target for next release

moting9 commented 1 month ago

hostpath is easy for getting started, maybe we can have a "sample" dir in the repo, and add diverse nfs, s3 and so on backend pv/pvc to store model files for user reference.