Closed mahmoodn closed 1 month ago
Hi @mahmoodn , the 10GB device memory is not enough to quantize the GPT-J model. Please refer to the answer of a similar issue: https://github.com/NVIDIA/TensorRT-LLM/issues/1932#issuecomment-2227560712
Thanks for the reply. Unfortunately, I don't have access to A100 (Ampere). If there is no option to create smaller chunks (if any) in order to reduce the GPU memory at a given time, then that is bad...
Or, I would like to ask if there are quantized files publicly available for those who don't have the computing resource?
Hi @mahmoodn , we do have plans to upload pre-quantized weights to HF model hub in the future.
I am using quantize.py according to GPTJ inference guide and the command is
That command however fails with an out-of-memory error message:
I searched for
PYTORCH_CUDA_ALLOC_CONF
to see how to use that. I tried different values and even when I use 32 (minimum is 20) withexport 'PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:32'
prior to running the Python command, I still get the same error.I have one RTX3080 device with 10GB of memory. Any idea on how to fix that? I don't know if that is a TensortRT-LLM issue or Pytorch issue, so any idea would greatly help.