PygmalionAI / aphrodite-engine

Large-scale LLM inference engine
https://aphrodite.pygmalion.chat
GNU Affero General Public License v3.0
1.09k stars 120 forks source link

[Installation]: Docker runs out of CPU swap size on 8 GPUs. How to lower swap_space to be less than 4GB per GPU? #483

Closed elabz closed 1 month ago

elabz commented 5 months ago

Your current environment

The output of `python env.py`

My apologies, everyone, not sure how to provide environment data from a docker container that cannot start. But the hw setup is:

32GB RAM 8 x 6GB VRAM (P106-100 GPUs)

Getting this error on trying to start docker container:

aphrodite-engine  | ValueError: Too large swap space. 32.00 GiB out of the 31.27 GiB total CPU memory is allocated for the swap space.

Please note that the docker run command in Readme did not work since it would not pass HF_TOKEN, and the model cannot be downloaded. So, I had to set values in .env and then do docker compose build and docker compose up in order for it to get as far as try to start.

I can see a reference to swap_space in aphrodite-engine/aphrodite/common/config.py but the value seems to be hardcoded at 4GB elsewhere. Is there a way to lower CPU RAM Per GPU requirement by passing a config value to the docker container? If so, how? Would much appreciate help in getting past this. Thanks!

How did you install Aphrodite?

Docker install from aphrodite-engine/docker
elabz commented 5 months ago

Also, starting it on 4 GPUs I get this warning, which makes sense, but how do I configure the size?

WARNING:  Possibly too large swap space. 16.00 GiB out of the 31.27 GiB total
AlpinDale commented 1 month ago

--swap-space should handle this