Closed raj-khare closed 7 months ago
@raj-khare yes it does, but GPU/CUDA memory can't be swapped, and a 4-bit 70B model takes up ~35GB GPU memory just for the weights. It might work in 32GB memory if you use llama.cpp and don't offload all the layers to the GPU, but for 70B models I would really recommend using Jetson AGX Orin 64GB.
got it
I'm trying to load a 70B 4-bit quantized model on a 32 GB jetson kit. However my process gets killed even when i have a lot of swap. Does MLC support swap out of the box? Any advice is highly appreciated