docker images are too large

meta-llama / llama-stack

Composable building blocks to build Llama Apps

MIT License

4.68k stars 599 forks source link

docker images are too large #254

Open nikolaydubina opened 1 month ago

nikolaydubina commented 1 month ago

how come docker image is 9GB?

this is not model itself, right?

it is odd to have docker image 20x larger than model itself (e.g. 1B/3B INT4)

yanxi0830 commented 4 weeks ago

Thanks for raising the issue! The reason why distribution-meta-reference-gpu is 7GB is because it uses pytorch as base image which is ~6.18 GB, due to CUDA library for GPU inference. For distributions with remote providers, the size is ~2GB. We are also looking for ways to reduce the size of docker images.

stevegrubb commented 3 weeks ago

This is likely fixed by #349.