AniZpZ / AutoSmoothQuant

An easy-to-use package for implementing SmoothQuant for LLMs
MIT License
82 stars 7 forks source link

When quantizing baichuan2-7B, I encountered an error: AttributeError: 'RMSNorm' object has no attribute 'epsilon'. #20

Open Flyipig opened 6 months ago

Flyipig commented 6 months ago

Here are the details of the traceback: Traceback (most recent call last):
File "/data/run01/scz0oo5/AutoSmoothQuant/autosmoothquant/examples/smoothquant_model.py", line 70, in
main()
File "/HOME/scz0oo5/.conda/envs/AUTOSmooth/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/data/run01/scz0oo5/AutoSmoothQuant/autosmoothquant/examples/smoothquant_model.py", line 65, in main
int8_model = quant_model_class.from_float(model, decoder_layer_scales, quant_config)
File "/data/run01/scz0oo5/AutoSmoothQuant/autosmoothquant/models/baichuan.py", line 561, in from_float
int8_module.model = Int8BaichuanModel.from_float(
File "/data/run01/scz0oo5/AutoSmoothQuant/autosmoothquant/models/baichuan.py", line 363, in from_float
int8_module.layers[i] = Int8BaichuanLayer.from_float(
File "/data/run01/scz0oo5/AutoSmoothQuant/autosmoothquant/models/baichuan.py", line 284, in from_float
int8_module.input_layernorm = Int8BaichuanRMSNorm.from_float(
File "/data/run01/scz0oo5/AutoSmoothQuant/autosmoothquant/models/baichuan.py", line 54, in from_float
int8_norm = Int8BaichuanRMSNorm(module.weight.numel(), module.epsilon)
File "/HOME/scz0oo5/.conda/envs/AUTOSmooth/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1688, in getattr
raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")
AttributeError: 'RMSNorm' object has no attribute 'epsilon'

I guess this issue is related to the version of PyTorch being used. Could you please advise on the specific version of PyTorch required for using AUTOSmoothQuant?