Closed nepeee closed 1 year ago
Maybe it a bug of load_llama_model_4bit_low_ram_and_offload function in which it does not call model.half(), causing kv cache to be fp32 format. You need to call model.half() manually. Or you can use exllama for inferencing, which is much faster and need less VRAM so that you can use full 2048 context length.
I switched to the autogptq/hf version, seems to be ok now, thx ❤️
Hi!
I finetuned a 30B llama1 model with full 2048 token context using xformers with a 3090. But when i try to do inference i get cuda oom after only 866(inc. generated) tokens. Is this normal?
I used the same code as in inference.py: