llama 3.2 1B instruct spinquant int 4 as .pte

facebookresearch / SpinQuant

Code repo for the paper "SpinQuant LLM quantization with learned rotations"

Other

170 stars 15 forks source link

llama 3.2 1B instruct spinquant int 4 as .pte #16

Open v-prgmr opened 1 month ago

v-prgmr commented 1 month ago

Hi everyone,

Has anyone successfully created a .pte of llama 3.2 1B instruct spinquant from https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct-SpinQuant_INT4_EO8/tree/main

Asking because I am facing issues trying to convert it using spinquant and also while trying to use the executorch export scripts.

Thank you.

kuantuna commented 4 days ago

Yes, I went through these steps. Installed executorch and built relevant stuff, downloaded the folder from here, ran the export_llama script (the one specifically for spinquant) by replacing LLAMA_QUANTIZED_CHECKPOINT and LLAMA_PARAMS with the paths of the corresponding files downloaded. And then I obtained a .pte file.