add alternative to support torch.stft in RPi(aarch64)

[BrownsugarZeer]

First

Thank you for building these lifesavers :D\ I found that someone posted a same issue#6, and also noticed that fixed in #60976 with third-party then merged into lastest pytorch version (here) few days ago.

For now, I still faced the same problem about MKL(the following for more details). The first thing I tried to do was building from source, but it was my first time to build and everything was so strange to me :(\ Could you help me to check whether works on RPi4?\ Thanks a lot!!!

Bug

fft: ATen not compiled with MKL support\ RuntimeError raised when trying to compute STFT on RPi4.

To Reproduce

My code

import os
import torch
import torchaudio
from speechbrain.pretrained import EncoderClassifier

classifier = EncoderClassifier.from_hparams(
    source="pretrained_model/best_model",
    hparams_file='xvector_interface.yaml',
    savedir="pretrained_model/best_model"
)

print(os.uname().machine)
print(torch.__version__)

# Perform classification
audio_file = '/home/sugar/Desktop/vscode_workspace/ASR/repos/GSC10/pretrained_model/022cd682_nohash_0.wav'
signal, fs = torchaudio.load(audio_file)  # id: 022cd682, lb: yes
output_probs, score, index, text_lab = classifier.classify_batch(signal)
print(output_probs, score, index, text_lab)
print('Target: 022cd682_yes, Predicted: ' + text_lab[0])

Error trace:

aarch64
1.9.0
Traceback (most recent call last):
  File "/home/sugar/Desktop/vscode_workspace/ASR/repos/GSC10/pretrained_model/train_interface.py", line 18, in <module>
    output_probs, score, index, text_lab = classifier.classify_batch(signal)
  File "/home/sugar/Desktop/vscode_workspace/ASR/repos/speechbrain/speechbrain/pretrained/interfaces.py", line 608, in classify_batch
    emb = self.encode_batch(wavs, wav_lens)
  File "/home/sugar/Desktop/vscode_workspace/ASR/repos/speechbrain/speechbrain/pretrained/interfaces.py", line 570, in encode_batch
    feats = self.modules.compute_features(wavs)
  File "/home/sugar/Desktop/vscode_workspace/ASR/repos/asr/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/sugar/Desktop/vscode_workspace/ASR/repos/speechbrain/speechbrain/lobes/features.py", line 136, in forward
    STFT = self.compute_STFT(wav)
  File "/home/sugar/Desktop/vscode_workspace/ASR/repos/asr/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/sugar/Desktop/vscode_workspace/ASR/repos/speechbrain/speechbrain/processing/features.py", line 161, in forward
    stft = torch.stft(
  File "/home/sugar/Desktop/vscode_workspace/ASR/repos/asr/lib/python3.9/site-packages/torch/functional.py", line 471, in stft
    return _VF.stft(input, n_fft, hop_length, win_length, window,  # type: ignore[attr-defined]
RuntimeError: fft: ATen not compiled with MKL support

---

Simple sample:

import os
import torch
print(os.uname().machine)
print(torch.__version__)
print(torch.fft.fft(torch.randn(3,3)))

Error trace:

aarch64
1.9.0
Traceback (most recent call last):
  File "/home/sugar/Desktop/vscode_workspace/ASR/repos/mkl_error/simple_sample.py", line 11, in <module>
    main()
  File "/home/sugar/Desktop/vscode_workspace/ASR/repos/mkl_error/simple_sample.py", line 7, in main
    print(torch.fft.fft(torch.randn(3, 3)))
RuntimeError: fft: ATen not compiled with MKL support

Expected behavior

Everything run successfully.

Environment

PyTorch version: 1.9.0\ OS: Ubuntu 21.04 64bit\ Python version: 3.9.5

[KumaTea]

Hi, thank you for telling this! I'll try to build from the main branch later, to see if it works.

Note: this fix was merged into mainstream after tag v1.9.0 was released, so the version built will be 1.10.0a0.

---

Update:

There might be something wrong with Google's breakpad:

[2146/4295] Building CXX object third_party/breakpad/CMakeFil.../breakpad.dir/src/client/linux/handler/exception_handler.cc.o
FAILED: third_party/breakpad/CMakeFiles/breakpad.dir/src/client/linux/handler/exception_handler.cc.o
/opt/rh/devtoolset-8/root/usr/bin/c++ -DHAVE_A_OUT_H -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DUSE_EXTERNAL_MZCRC -D_FILE_OFFSET_BITS=64 -I/opt/pytorch/torch/build/caffe2/contrib/aten -I/opt/pytorch/torch/third_party/onnx -I/opt/pytorch/torch/build/third_party/onnx -I/opt/pytorch/torch/third_party/foxi -I/opt/pytorch/torch/build/third_party/foxi -I/opt/pytorch/torch/third_party/breakpad/src -I/opt/pytorch/torch/third_party/breakpad/src/third_party/linux/include -isystem /opt/pytorch/torch/build/third_party/gloo -isystem /opt/pytorch/torch/cmake/../third_party/gloo -isystem /opt/pytorch/torch/third_party/protobuf/src -isystem /opt/pytorch/torch/third_party/gemmlowp -isystem /opt/pytorch/torch/third_party/neon2sse -isystem /opt/pytorch/torch/third_party/XNNPACK/include -isystem /opt/pytorch/torch/cmake/../third_party/eigen -isystem /opt/python/cp39-cp39/include/python3.9 -isystem /opt/python/cp39-cp39/lib/python3.9/site-packages/numpy/core/include -isystem /opt/pytorch/torch/cmake/../third_party/pybind11/include -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -O3 -DNDEBUG -DNDEBUG -fPIC -DCAFFE2_USE_GLOO -D__NEON__ -DUSE_GCC_GET_CPUID -DTH_HAVE_THREAD -std=gnu++14 -MD -MT third_party/breakpad/CMakeFiles/breakpad.dir/src/client/linux/handler/exception_handler.cc.o -MF third_party/breakpad/CMakeFiles/breakpad.dir/src/client/linux/handler/exception_handler.cc.o.d -o third_party/breakpad/CMakeFiles/breakpad.dir/src/client/linux/handler/exception_handler.cc.o -c /opt/pytorch/torch/third_party/breakpad/src/client/linux/handler/exception_handler.cc
In file included from /opt/pytorch/torch/third_party/breakpad/src/client/linux/minidump_writer/linux_dumper.h:54,
                 from /opt/pytorch/torch/third_party/breakpad/src/client/linux/minidump_writer/minidump_writer.h:42,
                 from /opt/pytorch/torch/third_party/breakpad/src/client/linux/handler/exception_handler.h:42,
                 from /opt/pytorch/torch/third_party/breakpad/src/client/linux/handler/exception_handler.cc:66:
/opt/pytorch/torch/third_party/breakpad/src/client/linux/dump_writer_common/thread_info.h:69:27: error: field ‘regs’ has incomplete type ‘google_breakpad::user_regs_struct’
   struct user_regs_struct regs;
                           ^~~~
/opt/pytorch/torch/third_party/breakpad/src/client/linux/dump_writer_common/thread_info.h:69:10: note: forward declaration of ‘struct google_breakpad::user_regs_struct’
   struct user_regs_struct regs;
          ^~~~~~~~~~~~~~~~
/opt/pytorch/torch/third_party/breakpad/src/client/linux/dump_writer_common/thread_info.h:70:29: error: field ‘fpregs’ has incomplete type ‘google_breakpad::user_fpsimd_struct’
   struct user_fpsimd_struct fpregs;
                             ^~~~~~
/opt/pytorch/torch/third_party/breakpad/src/client/linux/dump_writer_common/thread_info.h:70:10: note: forward declaration of ‘struct google_breakpad::user_fpsimd_struct’
   struct user_fpsimd_struct fpregs;
          ^~~~~~~~~~~~~~~~~~
ninja: build stopped: subcommand failed.

I have suspended building and wait for upstream patches...

[BrownsugarZeer]

Thank you for the rapid response and testing!!!

Sadly to hear the bad news, but I will keep track of this. If there are any further information, I will let you know :D

[KumaTea]

Hi.

Still unable to solve the error, but I just realized that since breakpad is an error reporting tool and is newly add in PyTorch, simply disabling it (export USE_BREAKPAD=0) should be okay.

By this way I've managed to build the wheel of v1.10.0a0:

torch-1.10.0a0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

torch-1.10.0a0-cp39-cp39-linux_aarch64.whl

details...

GitHub won't allow me to upload `whl` here, so I have uploaded to AWS S3 instead. The link will live for about 1 month.

Python 3.9.6 (default, Aug 26 2021, 06:38:01)
[GCC 10.2.1 20210130 (Red Hat 10.2.1-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> import torch
>>> print(os.uname().machine)
aarch64
>>> print(torch.__version__)
1.10.0a0
>>> print(torch.fft.fft(torch.randn(3,3)))
tensor([[-0.4325+0.0000j, -0.0342-0.3125j, -0.0342+0.3125j],
        [ 0.1138+0.0000j, -2.5345-0.4279j, -2.5345+0.4279j],
        [ 3.2972+0.0000j,  0.4888-1.5191j,  0.4888+1.5191j]])

[BrownsugarZeer]

Hi, I had already installed and it worked!!!

But unfortunately, there was another problem happened. I'm not sure that torchaudio should be also built as latest version or what. Since the latest ver. of pytorch and torchaudio are still under developing(perhaps there are more uncertain errors).

I guess this error means pytorch does not contain the function that torchaudio needs (version mismatch between torchaudio and pytorch).

Python 3.9.5 (default, May 11 2021, 08:20:37) 
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.fft.fft(torch.randn(3, 3))
tensor([[ 1.6230+0.0000j, -0.5391+0.9954j, -0.5391-0.9954j],
        [-1.5692+0.0000j,  2.3500+0.4336j,  2.3500-0.4336j],
        [ 0.4668+0.0000j, -1.4437+1.2189j, -1.4437-1.2189j]])
>>> 
>>> import torchaudio
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/sugar/test/test/lib/python3.9/site-packages/torchaudio/__init__.py", line 1, in <module>
    from . import extension  # noqa: F401
  File "/home/sugar/test/test/lib/python3.9/site-packages/torchaudio/extension/__init__.py", line 5, in <module>
    _init_extension()
  File "/home/sugar/test/test/lib/python3.9/site-packages/torchaudio/extension/extension.py", line 11, in _init_extension
    _init_script_module(ext)
  File "/home/sugar/test/test/lib/python3.9/site-packages/torchaudio/extension/extension.py", line 19, in _init_script_module
    torch.classes.load_library(path)
  File "/home/sugar/test/test/lib/python3.9/site-packages/torch/_classes.py", line 48, in load_library
    torch.ops.load_library(path)
  File "/home/sugar/test/test/lib/python3.9/site-packages/torch/_ops.py", line 110, in load_library
    ctypes.CDLL(path)
  File "/usr/lib/python3.9/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /home/sugar/test/test/lib/python3.9/site-packages/torchaudio/_torchaudio.so: undefined symbol: _ZNK2at6Tensor9index_putERKN3c104ListINS1_8optionalIS0_EEEERKS0_b
>>> 

It's my personal project and I don't want to bother you too much. But if you have any ideas, please let me know.

Thanks again, sincerely.

[KumaTea]

Hi, just as you expected, all secondary packages (torchaudio, vision, text, etc.) require a specific PyTorch version.

In your situation torchaudio is built on top of PyTorch v1.9.0, so it won't work on v1.10.0 unless recompiled.

[BrownsugarZeer]

Hi, I have a great news!!! After installed some dependencies and ninja, I ran the script in pytorch-aarch64/build/manylinux/audio.sh with a following command. The the wheel was built successfully.

bash audio.sh 0.10.0a0 master

(Although faced some problems, and finally realized that I needed a RPi4 with 8G RAM. maybe you can mention this on the README)

In short, here is the result and works well !!

code:

import os 
import torch
import torchaudio
from speechbrain.pretrained import EncoderClassifier

classifier = EncoderClassifier.from_hparams(
    source="/home/sugar/Desktop/vscode_workspace/ASR/repos/GSC10/pretrained_model/best_model",
    hparams_file='xvector_interface.yaml',
    savedir="/home/sugar/Desktop/vscode_workspace/ASR/repos/GSC10/pretrained_model/best_model"
)

print(os.uname().machine)
print(torch.__version__)
print(torchaudio.__version__)

# Perform classification
audio_file = '/home/sugar/Desktop/vscode_workspace/ASR/repos/GSC10/pretrained_model/022cd682_nohash_0.wav'
signal, fs = torchaudio.load(audio_file)  # id: 022cd682, lb: yes
output_probs, score, index, text_lab = classifier.classify_batch(signal)
print('Target: 022cd682_yes, Predicted: ' + text_lab[0])

result

aarch64
1.10.0a0
0.10.0a0
Target: 022cd682_yes, Predicted: yes

[KumaTea]

Hi, great to hear that!

(Although faced some problems, and finally realized that I needed a RPi4 with 8G RAM. maybe you can mention this on the README)

You may try using zram to extend your RAM.

# sudo

modprobe zram num_devices=4
sleep 5

zramctl --find --size 2G
mkswap /dev/zram0
swapon -p 5 /dev/zram0
sleep 1

zramctl --find --size 2G
mkswap /dev/zram1
swapon -p 5 /dev/zram1
sleep 1

zramctl --find --size 2G
mkswap /dev/zram2
swapon -p 5 /dev/zram2
sleep 1

zramctl --find --size 2G
mkswap /dev/zram3
swapon -p 5 /dev/zram3

[hk0901]

Hello, I've met the question as you too when use stft in torch 1.7.1 。But compile failed using torch-1.10.0a0, My gcc version is 7.3.0 。

I try to compile libtorch many times in my own machine but failed at last ! Can you provide a solved version of torch1.7.1?
Thanks a lot

[KumaTea]

Hi @hk0901 ,

Can you provide a solved version of torch1.7.1?

I'm afraid I can't, because MKL support for aarch64 will be added since 1.10, patching older versions is likely not feasible.

But compile failed using torch-1.10.0a0, My gcc version is 7.3.0

Although GCC 5 should be able to support C++14 which is needed by PyTorch, you might need gcc 8 or newer based on my experience. It might be okay to use a newer gcc version.

[hk0901]

你好， torch-1.10.0a0-cp39-cp39-linux_aarch64.whl 的运行结果好像不正确， 我在x64平台的master版本，arm的1.9.0的版本，得到的结果都是相同，但是你的这个whl模型输出结果和stft的结果都和他们不一样， 请问是哪里有问题吗

另外，我自己编译总是报错，所以希望能用您提供的arm版本最新的pytorch

[KumaTea]

但是你的这个whl模型输出结果和stft的结果都和他们不一样

我没有用过 stft 算法，不甚清楚具体原因，或许可以提供下实例代码和运行结果

自己编译总是报错

armv8.2-a+dotprod 要求 GCC 版本在 8 以上