Closed dannysemi closed 6 months ago
Try lowering GPU_MEMORY_UTILIZATION
first
Try lowering
GPU_MEMORY_UTILIZATION
first
I set it to 0.8 and it worked. Does this reduce the model's performance?
It should lower the total number of requests you can handle concurrently. Try setting a higher number (but lower than the default 0.95), and work your way down until it doesn't crash.
Disabling CUDA Graphs by setting enforce eager to true would lower performance though, so lowering memory utilization is better.
Thank you. I'll figure out a number that works.
I get this error when trying to run gptq models. I've built the worker myself with enforce_eager set to True and it works. Maybe an environment variable? Or could there be something else preventing this model from utilizing the proper amount of vram?