Closed Permafacture closed 2 months ago
Hi, try setting env variable GPU_MEMORY_UTILIZATION
to 0.9
That seems to have fixed it. Thanks! The README says the default is .95 though in the container there was no value for that environment variable set until I set it.
I'm using MODEL_NAME=TheBloke/dolphin-2.2.1-mistral-7B-AWQ and QUANTIZATION=awq on a runpod serverless instance with a network drive, RTX 4090 which should be plenty of VRAM for this, and docker image runpod/worker-vllm:stable-cuda12.1.0
My first request completes successfully, but the second request to the same worker (sent after the first has completed) always crashes with OOM. If I log into the web terminal, nvidia-smi says all the vram is taken but lists no process as responsible.
Here's the code I'm using. I just run this once, wait for it to complete, and then run it again.
Additional information