Error about fairseq when inference hubert

KakaruHayate commented 3 months ago

muDNN(v2400) 2024-04-02 15:54:04.407026 0d:0h:0m:35s TID=0x7128101b227cf70d GPU=0 Handle=0xaacc340 ERROR# INVALID_PARAMETER in ScaledDotProductAttention::RunMath, Reason:
    Internal error 4d mask shape
/home/kakaru/anaconda3/envs/musatest/lib/python3.9/site-packages/torch/nn/functional.py:5334: UserWarning: Flash attention only supports architecture with mp version 2.2. But now attempts to run on a mp 2.1 gpu. (Triggered internally at /home/torch_musa/torch_musa/csrc/aten/ops/attention/mudnn/SDPUtils.h:28.)
  attn_output = scaled_dot_product_attention(q, k, v, attn_mask, dropout_p, is_causal)
  0%|                                                   | 0/509 [00:13<?, ?it/s]
Traceback (most recent call last):
  File "/home/kakaru/ReFlow-VAE-SVC-main/preprocess.py", line 207, in <module>
    preprocess(args.data.train_path, f0_extractor, volume_extractor, mel_extractor, units_encoder, sample_rate, hop_size, device = device, use_pitch_aug = use_pitch_aug, extensions = extensions)
  File "/home/kakaru/ReFlow-VAE-SVC-main/preprocess.py", line 137, in preprocess
    process(file)
  File "/home/kakaru/ReFlow-VAE-SVC-main/preprocess.py", line 102, in process
    units_t = units_encoder.encode(audio_t, sample_rate, hop_size)
  File "/home/kakaru/ReFlow-VAE-SVC-main/reflow/extractors.py", line 238, in encode
    units = self.model(audio_res)
  File "/home/kakaru/ReFlow-VAE-SVC-main/reflow/extractors.py", line 343, in __call__
    logits = self.hubert.extract_features(**inputs)
  File "/home/kakaru/anaconda3/envs/musatest/lib/python3.9/site-packages/fairseq/models/hubert/hubert.py", line 535, in extract_features
    res = self.forward(
  File "/home/kakaru/anaconda3/envs/musatest/lib/python3.9/site-packages/fairseq/models/hubert/hubert.py", line 467, in forward
    x, _ = self.encoder(
  File "/home/kakaru/anaconda3/envs/musatest/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/kakaru/anaconda3/envs/musatest/lib/python3.9/site-packages/fairseq/models/wav2vec/wav2vec2.py", line 1003, in forward
    x, layer_results = self.extract_features(x, padding_mask, layer)
  File "/home/kakaru/anaconda3/envs/musatest/lib/python3.9/site-packages/fairseq/models/wav2vec/wav2vec2.py", line 1049, in extract_features
    x, (z, lr) = layer(
  File "/home/kakaru/anaconda3/envs/musatest/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/kakaru/anaconda3/envs/musatest/lib/python3.9/site-packages/fairseq/models/wav2vec/wav2vec2.py", line 1260, in forward
    x, attn = self.self_attn(
  File "/home/kakaru/anaconda3/envs/musatest/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/kakaru/anaconda3/envs/musatest/lib/python3.9/site-packages/fairseq/modules/multihead_attention.py", line 538, in forward
    return F.multi_head_attention_forward(
  File "/home/kakaru/anaconda3/envs/musatest/lib/python3.9/site-packages/torch/nn/functional.py", line 5334, in multi_head_attention_forward
    attn_output = scaled_dot_product_attention(q, k, v, attn_mask, dropout_p, is_causal)
RuntimeError: MuDNNMathSDPAFwd MUDNN failed in: Run SDPA

不使用fairseq去推理实际上是没问题的，或许是个bug？

KakaruHayate commented 2 months ago

nn.TransformerEncoderLayer基本都有这个问题

hanhaowen-mt commented 1 week ago

请提供复现源码

KakaruHayate commented 1 week ago

请提供复现源码

https://github.com/yxlllc/ReFlow-VAE-SVC/blob/main/reflow/extractors.py

可以参考这里，line335开始

fmo-mt commented 4 days ago

可能是SDP接口不适配导致的，麻烦提供下具体运行的代码（不是module源码），下个release中会修复

MooreThreads / torch_musa

Error about fairseq when inference hubert #45