Run the LLM's over multiple GPUS Using 8bit models to compress the vram footprint. "facebook/opt-30b" runs on 2 nvidia rtx 3090's. "facebook/opt-66b" might squeeze onto bigger GPUs or you can use float16 to and CPU or nvme/ssd offload.
This uses Huggingface accelerate and bitsandbytes.
Run the LLM's over multiple GPUS Using 8bit models to compress the vram footprint. "facebook/opt-30b" runs on 2 nvidia rtx 3090's. "facebook/opt-66b" might squeeze onto bigger GPUs or you can use float16 to and CPU or nvme/ssd offload.
This uses Huggingface accelerate and bitsandbytes.