Downsize the model-loader image

[kerthcet]

What would you like to be added:

Currently, the model-runner is about 56MB, however, the model-loader is about 466MB, we should try to smaller the size.

Why is this needed:

Fast startup.

Completion requirements:

This enhancement requires the following artifacts:

• [ ] Design doc
• [ ] API change
• [ ] Docs update

The artifacts should be linked in subsequent comments.

[kerthcet]

/kind feature /help

[qinguoyi]

AFAIK, Poetry is almost a combination of pip + venv. When building an image, do we really need to use a virtual environment?

Can we only install python and necessary package dependencies, which may reduce the size of the image.

[kerthcet]

I think we didn't install the virtual env in the image, see command in Dockerfile.loader:

RUN poetry config virtualenvs.create false

Maybe this is just because of the dependency problem and python is an interpreted language.

[kerthcet]

Actually, at first, I want to use rust instead, however, modelScope doesn't support this, huggingface has a rust lib, moreover, the object store providers doesn't support rust language as well, so I have to use python instead.

According to huggingface post, rust could be 2x(IIRC) faster than python because of the lock problem in python. 😄 related document: https://huggingface.co/docs/huggingface_hub/v0.25.1/package_reference/environment_variables#hfhubenablehftransfer

[qinguoyi]

https://github.com/InftyAI/llmaz/pull/179#issue-2554672963

hi, i push a request to downsize the model-loader image, PTAL, tks.

[qinguoyi]

Actually, at first, I want to use rust instead, however, modelScope doesn't support this, huggingface has a rust lib, moreover, the object store providers doesn't support rust language as well, so I have to use python instead.

According to huggingface post, rust could be 2x(IIRC) faster than python because of the lock problem in python. 😄 related document: https://huggingface.co/docs/huggingface_hub/v0.25.1/package_reference/environment_variables#hfhubenablehftransfer

in fact，this module only downloads the image, not inference and training, and does not require strong binding to python, so why not use go?

[kerthcet]

Because huggingface_hub doesn't have a golang binding, I just don't want to rewrite the whole logic. Am I misunderstanding you here?

[kerthcet]

fixed by https://github.com/InftyAI/llmaz/pull/179, close then. /close