Swap memory not working

[raj-khare]

I'm trying to load a 70B 4-bit quantized model on a 32 GB jetson kit. However my process gets killed even when i have a lot of swap. Does MLC support swap out of the box? Any advice is highly appreciated

root@tegra-ubuntu:/# python3 /opt/mlc-llm/benchmark.py --model /data/models/mlc/dist/models/q4f16_ft/params/ --prompt /data/prompts/completion_16.json --max-new-tokens 128
Namespace(model='/data/models/mlc/dist/models/q4f16_ft/params/', prompt=['/data/prompts/completion_16.json'], chat=False, streaming=False, max_new_tokens=128, max_num_prompts=None, save='')
-- loading /data/models/mlc/dist/models/q4f16_ft/params/
Killed
root@tegra-ubuntu:/# 

[dusty-nv]

@raj-khare yes it does, but GPU/CUDA memory can't be swapped, and a 4-bit 70B model takes up ~35GB GPU memory just for the weights. It might work in 32GB memory if you use llama.cpp and don't offload all the layers to the GPU, but for 70B models I would really recommend using Jetson AGX Orin 64GB.

[raj-khare]

got it