Open keymonzhang opened 1 week ago
@keymonzhang Nothing really officially recommended. Powerful gpu's like h100 should be the fastest right now especially with fp8, however most people don't have access to them. A 4090 with the gguf q8 version will be decently fast and perform nicely.
The lowest you can run it comfortably is with a good 8gb vram gpu and a nf4 quantized version. It will be slightly worse then the original one(bf16) but can fit in 8gb vram unlike bf16 which requires at least 24gb at the least.
I hope someone can help me answer it. Thanks!