Closed willsamu closed 7 months ago
This is pretty useful to have, some models like Capybara-34B have ludicrous context lengths that are difficult to fit in a workers VRAM.
Using this stopped me hitting Out Of Memory errors in those cases.
Thank you for your feedback and taking the time to make this, this feature has now been added in the latest update!
Fixes this message received when trying to run a mistral-7b finetune:
gpu_memory_utilization
is already set to0.98
by default. With this PRMAX_MODEL_LEN
can be set as environment variable. Expects an integer, defaults to 4096.