AttributeError: 'HQQLinear' object has no attribute 'weight'

[foduucom]

Thank you for creating this amazing package. It looks very promising. But i am facing some issue on installation on Linux Mint (Ubuntu) desktop. Where i have a NVIDIA RTX 3060 GPU with 12GB VRAM. I am just trying the same code as provided in the reference readme.

It can download the YouTube video and convert to MP3, also model is downloaded. It loaded the model into memory (not sure) but then i am stuck. Below the error. If need any trace log i can share.

2024-05-10 16:10:34,324 - INFO - We will use 90% of the memory on device 0 for storing the model, and 10% for the buffer to avoid OOM. You can set `max_memory` in to a higher value to use more memory (at your own risk).
Traceback (most recent call last):
  File "/home/ml/whisperplus/demo-test.py", line 24, in <module>
    pipeline = SpeechToTextPipeline(
  File "/home/ml/.local/lib/python3.10/site-packages/whisperplus/pipelines/whisper.py", line 26, in __init__
    self.load_model(model_id, quant_config)
  File "/home/ml/.local/lib/python3.10/site-packages/whisperplus/pipelines/whisper.py", line 43, in load_model
    model = AutoModelForSpeechSeq2Seq.from_pretrained(
  File "/home/ml/.local/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 563, in from_pretrained
    return model_class.from_pretrained(
  File "/home/ml/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3689, in from_pretrained
    ) = cls._load_pretrained_model(
  File "/home/ml/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4123, in _load_pretrained_model
    new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
  File "/home/ml/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 847, in _load_state_dict_into_meta_model
    old_param = getattr(old_param, split)
  File "/home/ml/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1695, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
**AttributeError: 'HQQLinear' object has no attribute 'weight'**

[kadirnar]

It's a problem with the Transformers library. https://github.com/huggingface/transformers/issues/30727

Would you try this?

pip install git+https://github.com/huggingface/transformers.git@d0c72c15c25e100860b6af692d084f06546a0b7a

[foduucom]

@kadirnar Thank you for your quick response. But I am still facing same issue. 😔

[kadirnar]

Transformers library solved the error. Can you try deleting the Transformers library and reinstalling it? https://github.com/huggingface/transformers/pull/30748

pip install git+https://github.com/huggingface/transformers

[foduucom]

I think the issue from transformers has been solved. But further there is something else :

2024-05-11 14:51:45,554 - INFO - We will use 90% of the memory on device 0 for storing the model, and 10% for the buffer to avoid OOM. You can set `max_memory` in to a higher value to use more memory (at your own risk).
2024-05-11 14:51:55,127 - INFO - Model loaded successfully.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-05-11 14:51:56,829 - INFO - Transcribing audio...
Traceback (most recent call last):
  File "/home/ml/whisperplus/demo-test.py", line 31, in <module>
    transcript = pipeline(
  File "/home/ml/.local/lib/python3.10/site-packages/whisperplus/pipelines/whisper.py", line 92, in __call__
    result = pipe(audio_path)
  File "/home/ml/.local/lib/python3.10/site-packages/transformers/pipelines/automatic_speech_recognition.py", line 285, in __call__
    return super().__call__(inputs, **kwargs)
  File "/home/ml/.local/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1234, in __call__
    return next(
  File "/home/ml/.local/lib/python3.10/site-packages/transformers/pipelines/pt_utils.py", line 124, in __next__
    item = next(self.iterator)
  File "/home/ml/.local/lib/python3.10/site-packages/transformers/pipelines/pt_utils.py", line 269, in __next__
    processed = self.infer(next(self.iterator), **self.params)
  File "/home/ml/.local/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1149, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
  File "/home/ml/.local/lib/python3.10/site-packages/transformers/pipelines/automatic_speech_recognition.py", line 506, in _forward
    generate_kwargs["encoder_outputs"] = encoder(inputs, attention_mask=attention_mask)
  File "/home/ml/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/ml/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ml/.local/lib/python3.10/site-packages/accelerate/hooks.py", line 166, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/ml/.local/lib/python3.10/site-packages/transformers/models/whisper/modeling_whisper.py", line 1209, in forward
    layer_outputs = encoder_layer(
  File "/home/ml/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/ml/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ml/.local/lib/python3.10/site-packages/accelerate/hooks.py", line 166, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/ml/.local/lib/python3.10/site-packages/transformers/models/whisper/modeling_whisper.py", line 765, in forward
    hidden_states, attn_weights, _ = self.self_attn(
  File "/home/ml/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/ml/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ml/.local/lib/python3.10/site-packages/transformers/models/whisper/modeling_whisper.py", line 435, in forward
    query_states = self._reshape(self.q_proj(hidden_states), -1, bsz)
  File "/home/ml/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/ml/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ml/.local/lib/python3.10/site-packages/accelerate/hooks.py", line 166, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/ml/.local/lib/python3.10/site-packages/hqq/core/quantize.py", line 790, in forward_aten_backprop
    return HQQMatmulNoCacheDeq.apply(x, self.dequantize_aten, self.bias)
  File "/home/ml/.local/lib/python3.10/site-packages/torch/autograd/function.py", line 539, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/ml/.local/lib/python3.10/site-packages/hqq/core/quantize.py", line 257, in forward
    out = torch.matmul(x, dequantize().t())
  File "/home/ml/.local/lib/python3.10/site-packages/hqq/core/quantize.py", line 713, in dequantize_aten
    W_est = self.dequantize_Wq_aten(W_q, meta)
  File "/home/ml/.local/lib/python3.10/site-packages/hqq/core/quantize.py", line 661, in dequantize_Wq_aten
    return hqq_aten.dequantize(
AttributeError: 'NoneType' object has no attribute 'dequantize'

[kadirnar]

Tested and working with RTX 3090. Did you do this?

pip uninstall transformers
pip install git+https://github.com/huggingface/transformers

[foduucom]

i am trying it on RTX 3060. Still, i am facing the same issue. I will try it on Colab or somewhere else. Then update you!

[kadirnar]

Tomorrow I will update colab notebook and test it.