I want to run llama 3 8B model (Pretrained) I don't want to quantize it, it takes 16gb of vram with 8192 ctx and every optimization,
i have 2 T4 gpus but the issue is that if i run on 1 t4 gpu (15Gb) it crashed but if i run on both, Both gpu usuage is 14Gb, though i want to use 7 gb vram portion of other gpu for Stable Diffusion, i've tried using the gpu vram percent parameter but it limits gpu vram on both gpu and model crashes (COM) error. (cuda out of memory)
Due to the nature of tensor parallelism, it is not possible. You may consider using exllamav2/tabbyapi for this case, they can run unquantized models in addition to exl2.
Your current environment
i7-13th Gen, 30GB ram ddr4, 2 T4 (15x2)GB
How would you like to use Aphrodite?
I want to run llama 3 8B model (Pretrained) I don't want to quantize it, it takes 16gb of vram with 8192 ctx and every optimization, i have 2 T4 gpus but the issue is that if i run on 1 t4 gpu (15Gb) it crashed but if i run on both, Both gpu usuage is 14Gb, though i want to use 7 gb vram portion of other gpu for Stable Diffusion, i've tried using the gpu vram percent parameter but it limits gpu vram on both gpu and model crashes (COM) error. (cuda out of memory)