search

mit-han-lab / llm-awq

[MLSys 2024 Best Paper Award] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
MIT License
2.38k stars 184 forks source link

awq_inference_engine is missing from source, so quantizing custom models fails #173

Closed RDouglasSharp closed 5 months ago

RDouglasSharp commented 5 months ago

python -m awq.entry --model_path /home/doug/checkpoints/dh/latest/checkpoint-3335 --w_bit 4 --q_group_size 128 --run_awq --dump_awq awq_cache/checkpoint-3335.pt /home/doug/.local/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead. _torch_pytree._register_pytree_node( /home/doug/.local/lib/python3.10/site-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead. _torch_pytree._register_pytree_node( Traceback (most recent call last): File "/home/doug/miniconda3/envs/awq/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/doug/miniconda3/envs/awq/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/home/doug/llm-awq/awq/entry.py", line 15, in from awq.quantize.pre_quant import run_awq, apply_awq File "/home/doug/llm-awq/awq/quantize/pre_quant.py", line 12, in from tinychat.models import LlavaLlamaForCausalLM File "/home/doug/llm-awq/tinychat/models/init.py", line 1, in from .falcon import FalconForCausalLM File "/home/doug/llm-awq/tinychat/models/falcon.py", line 11, in import awq_inference_engine ModuleNotFoundError: No module named 'awq_inference_engine'

RDouglasSharp commented 5 months ago

I think this may be related to issue #118 as well. ImportError: DLL load failed while importing awq_inference_engine #118

RDouglasSharp commented 5 months ago

Never mind... I realize now that I was missing an installation step