Closed jzysl423 closed 1 year ago
Dec1lent
commented
1 year ago So is it fixed? Google colab doesnt install the latest version of Transformers !pip install -U transformers to upgrade.
If thats not your problem, please send a snippet or full error message.
jzysl423
commented
1 year ago !pip install -r ./requirements.txt Error
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/, https://download.pytorch.org/whl/cu116
Collecting nemo_toolkit[asr]
Cloning https://github.com/NVIDIA/NeMo.git (to revision r1.14.0) to /tmp/pip-install-_fa2rbiq/nemo-toolkit_3ab91ff1859249e99cff6c11a94a0b0c
Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA/NeMo.git /tmp/pip-install-_fa2rbiq/nemo-toolkit_3ab91ff1859249e99cff6c11a94a0b0c
Running command git checkout -b r1.14.0 --track origin/r1.14.0
Switched to a new branch 'r1.14.0'
Branch 'r1.14.0' set up to track remote branch 'r1.14.0' from 'origin'.
Resolved https://github.com/NVIDIA/NeMo.git to commit 0a0b8a1bbaf9f884cf3abce2ceeb149f68ed114c
Preparing metadata (setup.py) ... done
Collecting git+https://github.com/openai/whisper.git (from -r ./requirements.txt (line 12))
Cloning https://github.com/openai/whisper.git to /tmp/pip-req-build-btppygk9
Running command git clone --filter=blob:none --quiet https://github.com/openai/whisper.git /tmp/pip-req-build-btppygk9
Resolved https://github.com/openai/whisper.git to commit 7858aa9c08d98f75575035ecd6481f462d66ca27
Preparing metadata (setup.py) ... done
Collecting git+https://github.com/m-bain/whisperX.git@d687cf33588dac51536d7d0edb840ac92a69df11 (from -r ./requirements.txt (line 13))
Cloning https://github.com/m-bain/whisperX.git (to revision d687cf33588dac51536d7d0edb840ac92a69df11) to /tmp/pip-req-build-cycjvi7s
Running command git clone --filter=blob:none --quiet https://github.com/m-bain/whisperX.git /tmp/pip-req-build-cycjvi7s
Running command git rev-parse -q --verify 'sha^d687cf33588dac51536d7d0edb840ac92a69df11'
Running command git fetch -q https://github.com/m-bain/whisperX.git d687cf33588dac51536d7d0edb840ac92a69df11
Running command git checkout -q d687cf33588dac51536d7d0edb840ac92a69df11
Resolved https://github.com/m-bain/whisperX.git to commit d687cf33588dac51536d7d0edb840ac92a69df11
Preparing metadata (setup.py) ... done
Collecting demucs
Cloning https://github.com/facebookresearch/demucs to /tmp/pip-install-_fa2rbiq/demucs_7cba59e7ae7b4770aaa74f0e2667f528
Running command git clone --filter=blob:none --quiet https://github.com/facebookresearch/demucs /tmp/pip-install-_fa2rbiq/demucs_7cba59e7ae7b4770aaa74f0e2667f528
Resolved https://github.com/facebookresearch/demucs to commit 3b8430c12242bbbba48769eed6da5190c6ff3c2d
Preparing metadata (setup.py) ... done
Requirement already satisfied: torch==1.13.1+cu116 in /usr/local/lib/python3.8/dist-packages (from -r ./requirements.txt (line 2)) (1.13.1+cu116)
Requirement already satisfied: torchvision==0.14.1+cu116 in /usr/local/lib/python3.8/dist-packages (from -r ./requirements.txt (line 3)) (0.14.1+cu116)
Requirement already satisfied: torchaudio==0.13.1+cu116 in /usr/local/lib/python3.8/dist-packages (from -r ./requirements.txt (line 4)) (0.13.1+cu116)
Requirement already satisfied: torchtext==0.14.1 in /usr/local/lib/python3.8/dist-packages (from -r ./requirements.txt (line 5)) (0.14.1)
Collecting pytorch-lightning==1.8.6
Downloading pytorch_lightning-1.8.6-py3-none-any.whl (800 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 800.3/800.3 KB 13.0 MB/s eta 0:00:00
Collecting sanic==22.6.2
Downloading sanic-22.6.2-py3-none-any.whl (271 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 271.9/271.9 KB 34.3 MB/s eta 0:00:00
Requirement already satisfied: transformers in /usr/local/lib/python3.8/dist-packages (from -r ./requirements.txt (line 8)) (4.26.1)
Collecting accelerate
Downloading accelerate-0.16.0-py3-none-any.whl (199 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 199.7/199.7 KB 29.2 MB/s eta 0:00:00
Requirement already satisfied: text-unidecode in /usr/local/lib/python3.8/dist-packages (from -r ./requirements.txt (line 10)) (1.3)
Collecting deepmultilingualpunctuation
Downloading deepmultilingualpunctuation-1.0.1-py3-none-any.whl (5.4 kB)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.8/dist-packages (from torch==1.13.1+cu116->-r ./requirements.txt (line 2)) (4.4.0)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.8/dist-packages (from torchvision==0.14.1+cu116->-r ./requirements.txt (line 3)) (7.1.2)
Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from torchvision==0.14.1+cu116->-r ./requirements.txt (line 3)) (2.25.1)
Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from torchvision==0.14.1+cu116->-r ./requirements.txt (line 3)) (1.21.6)
Requirement already satisfied: tqdm in /usr/local/lib/python3.8/dist-packages (from torchtext==0.14.1->-r ./requirements.txt (line 5)) (4.64.1)
Collecting tensorboardX>=2.2
Downloading tensorboardX-2.6-py2.py3-none-any.whl (114 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 114.5/114.5 KB 17.6 MB/s eta 0:00:00
Collecting lightning-utilities!=0.4.0,>=0.3.0
Downloading lightning_utilities-0.6.0.post0-py3-none-any.whl (18 kB)
Requirement already satisfied: PyYAML>=5.4 in /usr/local/lib/python3.8/dist-packages (from pytorch-lightning==1.8.6->-r ./requirements.txt (line 6)) (6.0)
Requirement already satisfied: packaging>=17.0 in /usr/local/lib/python3.8/dist-packages (from pytorch-lightning==1.8.6->-r ./requirements.txt (line 6)) (23.0)
Requirement already satisfied: fsspec[http]>2021.06.0 in /usr/local/lib/python3.8/dist-packages (from pytorch-lightning==1.8.6->-r ./requirements.txt (line 6)) (2023.1.0)
Collecting torchmetrics>=0.7.0
Downloading torchmetrics-0.11.1-py3-none-any.whl (517 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 517.2/517.2 KB 49.0 MB/s eta 0:00:00
Collecting sanic-routing<22.6.0,>=22.3.0
Downloading sanic_routing-22.3.0-py3-none-any.whl (25 kB)
Collecting aiofiles>=0.6.0
Downloading aiofiles-23.1.0-py3-none-any.whl (14 kB)
Requirement already satisfied: multidict<7.0,>=5.0 in /usr/local/lib/python3.8/dist-packages (from sanic==22.6.2->-r ./requirements.txt (line 7)) (6.0.4)
Collecting websockets>=10.0
Downloading websockets-10.4-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (106 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 107.0/107.0 KB 15.3 MB/s eta 0:00:00
Collecting ujson>=1.35
Downloading ujson-5.7.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (52 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 52.8/52.8 KB 8.0 MB/s eta 0:00:00
Collecting uvloop>=0.5.3
Downloading uvloop-0.17.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.6/4.6 MB 80.3 MB/s eta 0:00:00
Collecting httptools>=0.0.10
Downloading httptools-0.5.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (427 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 427.8/427.8 KB 45.5 MB/s eta 0:00:00
Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from transformers->-r ./requirements.txt (line 8)) (3.9.0)
Requirement already satisfied: huggingface-hub<1.0,>=0.11.0 in /usr/local/lib/python3.8/dist-packages (from transformers->-r ./requirements.txt (line 8)) (0.12.0)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.8/dist-packages (from transformers->-r ./requirements.txt (line 8)) (2022.6.2)
Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.8/dist-packages (from transformers->-r ./requirements.txt (line 8)) (0.13.2)
Requirement already satisfied: psutil in /usr/local/lib/python3.8/dist-packages (from accelerate->-r ./requirements.txt (line 9)) (5.4.8)
Requirement already satisfied: numba in /usr/local/lib/python3.8/dist-packages (from nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.56.4)
Collecting onnx>=1.7.0
Downloading onnx-1.13.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.5/13.5 MB 99.7 MB/s eta 0:00:00
Requirement already satisfied: python-dateutil in /usr/local/lib/python3.8/dist-packages (from nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (2.8.2)
Collecting ruamel.yaml
Downloading ruamel.yaml-0.17.21-py3-none-any.whl (109 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 109.5/109.5 KB 16.6 MB/s eta 0:00:00
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.8/dist-packages (from nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (1.0.2)
Collecting setuptools==59.5.0
Downloading setuptools-59.5.0-py3-none-any.whl (952 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 952.4/952.4 KB 74.8 MB/s eta 0:00:00
Requirement already satisfied: tensorboard in /usr/local/lib/python3.8/dist-packages (from nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (2.11.2)
Collecting wget
Downloading wget-3.2.zip (10 kB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: wrapt in /usr/local/lib/python3.8/dist-packages (from nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (1.14.1)
Collecting braceexpand
Downloading braceexpand-0.1.7-py2.py3-none-any.whl (5.9 kB)
Requirement already satisfied: editdistance in /usr/local/lib/python3.8/dist-packages (from nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.5.3)
Collecting g2p_en
Downloading g2p_en-2.1.0-py3-none-any.whl (3.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.1/3.1 MB 100.9 MB/s eta 0:00:00
Requirement already satisfied: inflect in /usr/local/lib/python3.8/dist-packages (from nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (2.1.0)
Requirement already satisfied: ipywidgets in /usr/local/lib/python3.8/dist-packages (from nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (7.7.1)
Collecting jiwer
Downloading jiwer-2.5.1-py3-none-any.whl (15 kB)
Collecting kaldi-python-io
Downloading kaldi-python-io-1.2.2.tar.gz (8.8 kB)
Preparing metadata (setup.py) ... done
Collecting kaldiio
Downloading kaldiio-2.17.2.tar.gz (24 kB)
Preparing metadata (setup.py) ... done
Collecting librosa>=0.9.0
Downloading librosa-0.9.2-py3-none-any.whl (214 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 214.3/214.3 KB 27.3 MB/s eta 0:00:00
Requirement already satisfied: marshmallow in /usr/local/lib/python3.8/dist-packages (from nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (3.19.0)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.8/dist-packages (from nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (3.2.2)
Collecting pyannote.core
Downloading pyannote.core-5.0.0-py3-none-any.whl (58 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.5/58.5 KB 5.1 MB/s eta 0:00:00
Collecting pyannote.metrics
Downloading pyannote.metrics-3.2.1-py3-none-any.whl (51 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 51.4/51.4 KB 7.7 MB/s eta 0:00:00
Collecting pydub
Downloading pydub-0.25.1-py2.py3-none-any.whl (32 kB)
Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.8/dist-packages (from nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (1.7.3)
Requirement already satisfied: soundfile in /usr/local/lib/python3.8/dist-packages (from nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.11.0)
Collecting sox
Downloading sox-1.4.1-py2.py3-none-any.whl (39 kB)
Collecting texterrors
Downloading texterrors-0.4.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 58.3 MB/s eta 0:00:00
Collecting hydra-core<1.3,>=1.2.0
Downloading hydra_core-1.2.0-py3-none-any.whl (151 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 151.1/151.1 KB 19.2 MB/s eta 0:00:00
Collecting omegaconf<2.3,>=2.2
Downloading omegaconf-2.2.3-py3-none-any.whl (79 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.3/79.3 KB 10.7 MB/s eta 0:00:00
Collecting PyYAML>=5.4
Downloading PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl (662 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 662.4/662.4 KB 53.3 MB/s eta 0:00:00
Collecting torchmetrics>=0.7.0
Downloading torchmetrics-0.10.3-py3-none-any.whl (529 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 529.7/529.7 KB 44.9 MB/s eta 0:00:00
Collecting transformers
Downloading transformers-4.21.2-py3-none-any.whl (4.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.7/4.7 MB 70.9 MB/s eta 0:00:00
Collecting wandb
Downloading wandb-0.13.10-py3-none-any.whl (2.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 76.5 MB/s eta 0:00:00
Collecting webdataset<=0.1.62,>=0.1.48
Downloading webdataset-0.1.62-py3-none-any.whl (32 kB)
Requirement already satisfied: pandas in /usr/local/lib/python3.8/dist-packages (from nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (1.3.5)
Collecting sacremoses>=0.0.43
Downloading sacremoses-0.0.53.tar.gz (880 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 880.6/880.6 KB 4.5 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Collecting sentencepiece<1.0.0
Downloading sentencepiece-0.1.97-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 61.7 MB/s eta 0:00:00
Collecting youtokentome>=1.0.5
Downloading youtokentome-1.0.6-cp38-cp38-manylinux2010_x86_64.whl (1.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 66.5 MB/s eta 0:00:00
Collecting tokenizers!=0.11.3,<0.14,>=0.11.1
Downloading tokenizers-0.12.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.6/6.6 MB 109.3 MB/s eta 0:00:00
Requirement already satisfied: more-itertools in /usr/local/lib/python3.8/dist-packages (from openai-whisper==20230124->-r ./requirements.txt (line 12)) (9.0.0)
Collecting ffmpeg-python==0.2.0
Downloading ffmpeg_python-0.2.0-py3-none-any.whl (25 kB)
Requirement already satisfied: future in /usr/local/lib/python3.8/dist-packages (from ffmpeg-python==0.2.0->openai-whisper==20230124->-r ./requirements.txt (line 12)) (0.16.0)
Collecting pyannote.audio
Downloading pyannote.audio-2.1.1-py2.py3-none-any.whl (390 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 390.7/390.7 KB 43.3 MB/s eta 0:00:00
Collecting dora-search
Downloading dora_search-0.1.11.tar.gz (87 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 87.0/87.0 KB 12.8 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting diffq>=0.2.1
Downloading diffq-0.2.3-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (446 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 446.3/446.3 KB 42.1 MB/s eta 0:00:00
Collecting einops
Downloading einops-0.6.0-py3-none-any.whl (41 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.6/41.6 KB 6.1 MB/s eta 0:00:00
Collecting julius>=0.2.3
Downloading julius-0.2.7.tar.gz (59 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 59.6/59.6 KB 8.6 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Collecting lameenc>=1.2
Downloading lameenc-1.4.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (189 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 189.6/189.6 KB 24.9 MB/s eta 0:00:00
Collecting openunmix
Downloading openunmix-1.2.1-py3-none-any.whl (46 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.7/46.7 KB 7.2 MB/s eta 0:00:00
Requirement already satisfied: Cython in /usr/local/lib/python3.8/dist-packages (from diffq>=0.2.1->demucs->-r ./requirements.txt (line 14)) (0.29.33)
Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /usr/local/lib/python3.8/dist-packages (from fsspec[http]>2021.06.0->pytorch-lightning==1.8.6->-r ./requirements.txt (line 6)) (3.8.3)
Collecting antlr4-python3-runtime==4.9.*
Downloading antlr4-python3-runtime-4.9.3.tar.gz (117 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 117.0/117.0 KB 16.0 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Requirement already satisfied: importlib-resources in /usr/local/lib/python3.8/dist-packages (from hydra-core<1.3,>=1.2.0->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (5.10.2)
Requirement already satisfied: decorator>=4.0.10 in /usr/local/lib/python3.8/dist-packages (from librosa>=0.9.0->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (4.4.2)
Requirement already satisfied: resampy>=0.2.2 in /usr/local/lib/python3.8/dist-packages (from librosa>=0.9.0->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.4.2)
Requirement already satisfied: pooch>=1.0 in /usr/local/lib/python3.8/dist-packages (from librosa>=0.9.0->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (1.6.0)
Requirement already satisfied: joblib>=0.14 in /usr/local/lib/python3.8/dist-packages (from librosa>=0.9.0->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (1.2.0)
Requirement already satisfied: audioread>=2.1.9 in /usr/local/lib/python3.8/dist-packages (from librosa>=0.9.0->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (3.0.0)
Requirement already satisfied: llvmlite<0.40,>=0.39.0dev0 in /usr/local/lib/python3.8/dist-packages (from numba->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.39.1)
Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.8/dist-packages (from numba->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (6.0.0)
Collecting protobuf<4,>=3.20.2
Downloading protobuf-3.20.3-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 59.6 MB/s eta 0:00:00
Requirement already satisfied: six in /usr/local/lib/python3.8/dist-packages (from sacremoses>=0.0.43->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (1.15.0)
Requirement already satisfied: click in /usr/local/lib/python3.8/dist-packages (from sacremoses>=0.0.43->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (7.1.2)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from scikit-learn->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (3.1.0)
Requirement already satisfied: cffi>=1.0 in /usr/local/lib/python3.8/dist-packages (from soundfile->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (1.15.1)
Collecting retrying
Downloading retrying-1.3.4-py3-none-any.whl (11 kB)
Collecting treetable
Downloading treetable-0.2.5.tar.gz (10 kB)
Preparing metadata (setup.py) ... done
Collecting submitit
Downloading submitit-1.4.5-py3-none-any.whl (73 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 73.1/73.1 KB 10.1 MB/s eta 0:00:00
Requirement already satisfied: nltk>=3.2.4 in /usr/local/lib/python3.8/dist-packages (from g2p_en->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (3.7)
Collecting distance>=0.1.3
Downloading Distance-0.1.3.tar.gz (180 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 180.3/180.3 KB 22.1 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Requirement already satisfied: widgetsnbextension~=3.6.0 in /usr/local/lib/python3.8/dist-packages (from ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (3.6.1)
Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /usr/local/lib/python3.8/dist-packages (from ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (3.0.5)
Requirement already satisfied: ipython-genutils~=0.2.0 in /usr/local/lib/python3.8/dist-packages (from ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.2.0)
Requirement already satisfied: traitlets>=4.3.1 in /usr/local/lib/python3.8/dist-packages (from ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (5.7.1)
Requirement already satisfied: ipython>=4.0.0 in /usr/local/lib/python3.8/dist-packages (from ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (7.9.0)
Requirement already satisfied: ipykernel>=4.5.1 in /usr/local/lib/python3.8/dist-packages (from ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (5.3.4)
Collecting levenshtein==0.20.2
Downloading Levenshtein-0.20.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 64.9 MB/s eta 0:00:00
Collecting rapidfuzz<3.0.0,>=2.3.0
Downloading rapidfuzz-2.13.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 96.7 MB/s eta 0:00:00
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.8/dist-packages (from matplotlib->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.11.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (1.4.4)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (2022.7.1)
Collecting hmmlearn<0.3,>=0.2.7
Downloading hmmlearn-0.2.8-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (217 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 217.1/217.1 KB 27.8 MB/s eta 0:00:00
Collecting speechbrain<0.6,>=0.5.12
Downloading speechbrain-0.5.13-py3-none-any.whl (498 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 499.0/499.0 KB 53.1 MB/s eta 0:00:00
Collecting pytorch-metric-learning<2.0,>=1.0.0
Downloading pytorch_metric_learning-1.7.3-py3-none-any.whl (112 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 112.2/112.2 KB 17.0 MB/s eta 0:00:00
Collecting soundfile
Downloading SoundFile-0.10.3.post1-py2.py3-none-any.whl (21 kB)
Collecting backports.cached-property
Downloading backports.cached_property-1.0.2-py3-none-any.whl (6.1 kB)
Collecting pyannote.database<5.0,>=4.1.1
Downloading pyannote.database-4.1.3-py3-none-any.whl (41 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.6/41.6 KB 5.9 MB/s eta 0:00:00
Collecting networkx<3.0,>=2.6
Downloading networkx-2.8.8-py3-none-any.whl (2.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 88.8 MB/s eta 0:00:00
Collecting torch-audiomentations>=0.11.0
Downloading torch_audiomentations-0.11.0-py3-none-any.whl (47 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.9/47.9 KB 7.2 MB/s eta 0:00:00
Collecting pyannote.audio
Downloading pyannote.audio-2.0.1-py2.py3-none-any.whl (385 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 385.9/385.9 KB 43.9 MB/s eta 0:00:00
Downloading pyannote.audio-1.1.2-py3-none-any.whl (231 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 231.2/231.2 KB 29.5 MB/s eta 0:00:00
Collecting pescador>=2.1.0
Downloading pescador-2.1.0.tar.gz (20 kB)
Preparing metadata (setup.py) ... done
Collecting sortedcollections>=1.0.1
Downloading sortedcollections-2.1.0-py3-none-any.whl (9.5 kB)
Requirement already satisfied: cachetools>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from pyannote.audio->whisperx==1.0->-r ./requirements.txt (line 13)) (5.3.0)
Requirement already satisfied: sortedcontainers>=2.0.4 in /usr/local/lib/python3.8/dist-packages (from pyannote.audio->whisperx==1.0->-r ./requirements.txt (line 13)) (2.4.0)
Collecting pyannote.pipeline<2.0.0,>=1.5.2
Downloading pyannote.pipeline-1.5.2-py3-none-any.whl (25 kB)
Requirement already satisfied: sympy>=1.1 in /usr/local/lib/python3.8/dist-packages (from pyannote.metrics->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (1.7.1)
Requirement already satisfied: tabulate>=0.7.7 in /usr/local/lib/python3.8/dist-packages (from pyannote.metrics->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.8.10)
Collecting docopt>=0.6.2
Downloading docopt-0.6.2.tar.gz (25 kB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.8/dist-packages (from tensorboard->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (1.4.0)
Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from tensorboard->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.6.1)
Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from tensorboard->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (1.0.1)
Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.8/dist-packages (from tensorboard->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.38.4)
Requirement already satisfied: grpcio>=1.24.3 in /usr/local/lib/python3.8/dist-packages (from tensorboard->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (1.51.1)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.8/dist-packages (from tensorboard->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (1.8.1)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.8/dist-packages (from tensorboard->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.4.6)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.8/dist-packages (from tensorboard->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (3.4.1)
Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.8/dist-packages (from tensorboard->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (2.16.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->torchvision==0.14.1+cu116->-r ./requirements.txt (line 3)) (1.24.3)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->torchvision==0.14.1+cu116->-r ./requirements.txt (line 3)) (2.10)
Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests->torchvision==0.14.1+cu116->-r ./requirements.txt (line 3)) (4.0.0)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->torchvision==0.14.1+cu116->-r ./requirements.txt (line 3)) (2022.12.7)
Collecting ruamel.yaml.clib>=0.2.6
Downloading ruamel.yaml.clib-0.2.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (555 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 555.3/555.3 KB 54.6 MB/s eta 0:00:00
Collecting plac
Downloading plac-1.3.5-py2.py3-none-any.whl (22 kB)
Collecting pybind11
Downloading pybind11-2.10.3-py3-none-any.whl (222 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 222.4/222.4 KB 31.7 MB/s eta 0:00:00
Collecting loguru
Downloading loguru-0.6.0-py3-none-any.whl (58 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.3/58.3 KB 9.3 MB/s eta 0:00:00
Requirement already satisfied: termcolor in /usr/local/lib/python3.8/dist-packages (from texterrors->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (2.2.0)
Collecting GitPython>=1.0.0
Downloading GitPython-3.1.30-py3-none-any.whl (184 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 184.0/184.0 KB 25.2 MB/s eta 0:00:00
Collecting pathtools
Downloading pathtools-0.1.2.tar.gz (11 kB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: appdirs>=1.4.3 in /usr/local/lib/python3.8/dist-packages (from wandb->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (1.4.4)
Collecting setproctitle
Downloading setproctitle-1.3.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (31 kB)
Collecting docker-pycreds>=0.4.0
Downloading docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)
Collecting sentry-sdk>=1.0.0
Downloading sentry_sdk-1.15.0-py2.py3-none-any.whl (181 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 181.3/181.3 KB 24.6 MB/s eta 0:00:00
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.8.6->-r ./requirements.txt (line 6)) (1.3.3)
Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.8.6->-r ./requirements.txt (line 6)) (2.1.1)
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.8/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.8.6->-r ./requirements.txt (line 6)) (1.3.1)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.8/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.8.6->-r ./requirements.txt (line 6)) (4.0.2)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.8.6->-r ./requirements.txt (line 6)) (1.8.2)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.8.6->-r ./requirements.txt (line 6)) (22.2.0)
Requirement already satisfied: pycparser in /usr/local/lib/python3.8/dist-packages (from cffi>=1.0->soundfile->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (2.21)
Collecting gitdb<5,>=4.0.1
Downloading gitdb-4.0.10-py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.7/62.7 KB 9.0 MB/s eta 0:00:00
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.8/dist-packages (from google-auth<3,>=1.6.3->tensorboard->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (4.9)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.8/dist-packages (from google-auth<3,>=1.6.3->tensorboard->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.2.8)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.8/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (1.3.1)
Requirement already satisfied: jupyter-client in /usr/local/lib/python3.8/dist-packages (from ipykernel>=4.5.1->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (6.1.12)
Requirement already satisfied: tornado>=4.2 in /usr/local/lib/python3.8/dist-packages (from ipykernel>=4.5.1->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (6.0.4)
Requirement already satisfied: pickleshare in /usr/local/lib/python3.8/dist-packages (from ipython>=4.0.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.7.5)
Requirement already satisfied: prompt-toolkit<2.1.0,>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from ipython>=4.0.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (2.0.10)
Collecting jedi>=0.10
Downloading jedi-0.18.2-py2.py3-none-any.whl (1.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 87.9 MB/s eta 0:00:00
Requirement already satisfied: pexpect in /usr/local/lib/python3.8/dist-packages (from ipython>=4.0.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (4.8.0)
Requirement already satisfied: pygments in /usr/local/lib/python3.8/dist-packages (from ipython>=4.0.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (2.6.1)
Requirement already satisfied: backcall in /usr/local/lib/python3.8/dist-packages (from ipython>=4.0.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.2.0)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.8/dist-packages (from importlib-metadata->numba->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (3.12.1)
Requirement already satisfied: pyzmq>=15.0 in /usr/local/lib/python3.8/dist-packages (from pescador>=2.1.0->pyannote.audio->whisperx==1.0->-r ./requirements.txt (line 13)) (23.2.1)
Requirement already satisfied: typer[all]>=0.2.1 in /usr/local/lib/python3.8/dist-packages (from pyannote.database<5.0,>=4.1.1->pyannote.audio->whisperx==1.0->-r ./requirements.txt (line 13)) (0.7.0)
Collecting optuna>=1.4
Downloading optuna-3.1.0-py3-none-any.whl (365 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 365.3/365.3 KB 29.4 MB/s eta 0:00:00
Collecting urllib3<1.27,>=1.21.1
Downloading urllib3-1.26.14-py2.py3-none-any.whl (140 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.6/140.6 KB 18.6 MB/s eta 0:00:00
Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.8/dist-packages (from sympy>=1.1->pyannote.metrics->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (1.2.1)
Requirement already satisfied: notebook>=4.4.1 in /usr/local/lib/python3.8/dist-packages (from widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (5.7.16)
Requirement already satisfied: cloudpickle>=1.2.1 in /usr/local/lib/python3.8/dist-packages (from submitit->dora-search->demucs->-r ./requirements.txt (line 14)) (2.2.1)
Collecting smmap<6,>=3.0.1
Downloading smmap-5.0.0-py3-none-any.whl (24 kB)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in /usr/local/lib/python3.8/dist-packages (from jedi>=0.10->ipython>=4.0.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.8.3)
Requirement already satisfied: terminado>=0.8.1 in /usr/local/lib/python3.8/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.13.3)
Requirement already satisfied: nbconvert<6.0 in /usr/local/lib/python3.8/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (5.6.1)
Requirement already satisfied: Send2Trash in /usr/local/lib/python3.8/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (1.8.0)
Requirement already satisfied: nbformat in /usr/local/lib/python3.8/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (5.7.3)
Requirement already satisfied: prometheus-client in /usr/local/lib/python3.8/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.16.0)
Requirement already satisfied: jinja2<=3.0.0 in /usr/local/lib/python3.8/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (2.11.3)
Requirement already satisfied: jupyter-core>=4.4.0 in /usr/local/lib/python3.8/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (5.2.0)
Requirement already satisfied: sqlalchemy>=1.3.0 in /usr/local/lib/python3.8/dist-packages (from optuna>=1.4->pyannote.pipeline<2.0.0,>=1.5.2->pyannote.audio->whisperx==1.0->-r ./requirements.txt (line 13)) (1.4.46)
Collecting alembic>=1.5.0
Downloading alembic-1.9.3-py3-none-any.whl (210 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 210.6/210.6 KB 26.0 MB/s eta 0:00:00
Collecting colorlog
Downloading colorlog-6.7.0-py2.py3-none-any.whl (11 kB)
Collecting cmaes>=0.9.1
Downloading cmaes-0.9.1-py3-none-any.whl (21 kB)
Requirement already satisfied: wcwidth in /usr/local/lib/python3.8/dist-packages (from prompt-toolkit<2.1.0,>=2.0.0->ipython>=4.0.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.2.6)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.8/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.4.8)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (3.2.2)
Collecting rich<13.0.0,>=10.11.0
Downloading rich-12.6.0-py3-none-any.whl (237 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 237.5/237.5 KB 28.8 MB/s eta 0:00:00
Collecting shellingham<2.0.0,>=1.3.0
Downloading shellingham-1.5.0.post1-py2.py3-none-any.whl (9.4 kB)
Collecting colorama<0.5.0,>=0.4.3
Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.8/dist-packages (from pexpect->ipython>=4.0.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.7.0)
Collecting Mako
Downloading Mako-1.2.4-py3-none-any.whl (78 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.7/78.7 KB 11.5 MB/s eta 0:00:00
Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.8/dist-packages (from jinja2<=3.0.0->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (2.0.1)
Requirement already satisfied: platformdirs>=2.5 in /usr/local/lib/python3.8/dist-packages (from jupyter-core>=4.4.0->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (3.0.0)
Requirement already satisfied: bleach in /usr/local/lib/python3.8/dist-packages (from nbconvert<6.0->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (6.0.0)
Requirement already satisfied: defusedxml in /usr/local/lib/python3.8/dist-packages (from nbconvert<6.0->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.7.1)
Requirement already satisfied: pandocfilters>=1.4.1 in /usr/local/lib/python3.8/dist-packages (from nbconvert<6.0->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (1.5.0)
Requirement already satisfied: entrypoints>=0.2.2 in /usr/local/lib/python3.8/dist-packages (from nbconvert<6.0->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.4)
Requirement already satisfied: mistune<2,>=0.8.1 in /usr/local/lib/python3.8/dist-packages (from nbconvert<6.0->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.8.4)
Requirement already satisfied: testpath in /usr/local/lib/python3.8/dist-packages (from nbconvert<6.0->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.6.0)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (2.16.2)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (4.3.3)
Collecting commonmark<0.10.0,>=0.9.0
Downloading commonmark-0.9.1-py2.py3-none-any.whl (51 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 51.1/51.1 KB 7.7 MB/s eta 0:00:00
Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.8/dist-packages (from sqlalchemy>=1.3.0->optuna>=1.4->pyannote.pipeline<2.0.0,>=1.5.2->pyannote.audio->whisperx==1.0->-r ./requirements.txt (line 13)) (2.0.2)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.19.3)
Requirement already satisfied: webencodings in /usr/local/lib/python3.8/dist-packages (from bleach->nbconvert<6.0->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.5.1)
Building wheels for collected packages: openai-whisper, whisperx, demucs, antlr4-python3-runtime, julius, sacremoses, dora-search, kaldi-python-io, kaldiio, nemo_toolkit, wget, distance, docopt, pescador, pathtools, treetable
Building wheel for openai-whisper (setup.py) ... done
Created wheel for openai-whisper: filename=openai_whisper-20230124-py3-none-any.whl size=1179424 sha256=2fa8bdec7d75beccd028120641bc3ae2082a0b293f85a3e84068f99efb6f5d01
Stored in directory: /tmp/pip-ephem-wheel-cache-19duztex/wheels/a7/70/18/b7693c07b1d18b3dafb328f5d0496aa0d41a9c09ef332fd8e6
Building wheel for whisperx (setup.py) ... done
Created wheel for whisperx: filename=whisperx-1.0-py3-none-any.whl size=1191159 sha256=7196216a703c43575fbe84597530d820e112b486bcfef22fb757311ad372b33d
Stored in directory: /root/.cache/pip/wheels/0f/a6/90/e4493eb2653ff56e74cb52c7f1af8c6197171c9d4cd8022d24
Building wheel for demucs (setup.py) ... done
Created wheel for demucs: filename=demucs-4.0.0-py3-none-any.whl size=76630 sha256=535757e245d34d44eaa7add7e1465c7f944ca298ce99ea519c5c4897b3b898ed
Stored in directory: /tmp/pip-ephem-wheel-cache-19duztex/wheels/aa/e1/ef/dd00c4df7b3e6dc06c9afa5048c7ab3aaf043cb63b97e78ccf
Building wheel for antlr4-python3-runtime (setup.py) ... done
Created wheel for antlr4-python3-runtime: filename=antlr4_python3_runtime-4.9.3-py3-none-any.whl size=144575 sha256=7e423ba65a8c5c1eede0927cc8e41c807880b59b3edfac5accabce652de1bb8e
Stored in directory: /root/.cache/pip/wheels/b1/a3/c2/6df046c09459b73cc9bb6c4401b0be6c47048baf9a1617c485
Building wheel for julius (setup.py) ... done
Created wheel for julius: filename=julius-0.2.7-py3-none-any.whl size=21895 sha256=6060a7ef0c51c4e5b4b790ac70c7d9abbc4e0526cc5bf353449a5ae5986ee601
Stored in directory: /root/.cache/pip/wheels/85/89/4f/88596b58a42ee452100fe1cd6ac31265bb192e597cf85908da
Building wheel for sacremoses (setup.py) ... done
Created wheel for sacremoses: filename=sacremoses-0.0.53-py3-none-any.whl size=895260 sha256=009ab97aa28b0e92c490f6d558cad416ee9ba9671acd2806862538d0b8e67c90
Stored in directory: /root/.cache/pip/wheels/82/ab/9b/c15899bf659ba74f623ac776e861cf2eb8608c1825ddec66a4
Building wheel for dora-search (pyproject.toml) ... done
Created wheel for dora-search: filename=dora_search-0.1.11-py3-none-any.whl size=75008 sha256=9fe36c48f2949a52855551b32fd77e3ee9d4425b22e9be074298685dd5048a40
Stored in directory: /root/.cache/pip/wheels/9b/42/69/2259709315acf43bd7b9876fb35454db01f770b63519966ac9
Building wheel for kaldi-python-io (setup.py) ... done
Created wheel for kaldi-python-io: filename=kaldi_python_io-1.2.2-py3-none-any.whl size=8970 sha256=c1c09698d19e01ed06cb791f4bc204c7746337385d417fcddbae6363d085ef1b
Stored in directory: /root/.cache/pip/wheels/8c/da/46/4afb7e26376c33af41c3ec388d5b63d34d186f6df1545cac30
Building wheel for kaldiio (setup.py) ... done
Created wheel for kaldiio: filename=kaldiio-2.17.2-py3-none-any.whl size=24471 sha256=db3c71d6bfa21785c6db92bf8a13081649a77d4c25ee4f7121931c8deac04166
Stored in directory: /root/.cache/pip/wheels/49/b3/00/af2103b510836161326bb51e27795407b07fda6969d0ae5967
Building wheel for nemo_toolkit (setup.py) ... done
Created wheel for nemo_toolkit: filename=nemo_toolkit-1.14.0-py3-none-any.whl size=3945956 sha256=354269d1b738f60a6ae4ea36980dd78e8385cbff2f391b5a1e9c8e0389531d5d
Stored in directory: /tmp/pip-ephem-wheel-cache-19duztex/wheels/ba/61/5e/9ff0ec204906506c099bb02de2193625082e939dd782cfea9b
Building wheel for wget (setup.py) ... done
Created wheel for wget: filename=wget-3.2-py3-none-any.whl size=9674 sha256=9cb12498e8d2ee7cd3ba085d3070a5c8d3efec06253643a0d924984c0a48c187
Stored in directory: /root/.cache/pip/wheels/bd/a8/c3/3cf2c14a1837a4e04bd98631724e81f33f462d86a1d895fae0
Building wheel for distance (setup.py) ... done
Created wheel for distance: filename=Distance-0.1.3-py3-none-any.whl size=16275 sha256=19d61b31efa1c84985417c56f90649eefa22a82c97be7ded351760607c2cdf62
Stored in directory: /root/.cache/pip/wheels/fb/a8/64/6edcab63ec51512a87cacf9b3563c711ad6b7b05d61b704493
Building wheel for docopt (setup.py) ... done
Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13723 sha256=179a7a7ee46692737e169d0bcb8c7506a7cca236d68480272a2586b618785290
Stored in directory: /root/.cache/pip/wheels/56/ea/58/ead137b087d9e326852a851351d1debf4ada529b6ac0ec4e8c
Building wheel for pescador (setup.py) ... done
Created wheel for pescador: filename=pescador-2.1.0-py3-none-any.whl size=21105 sha256=80aecfdc19c950cf923a760e2453ab862f2249f987ec5871797b1784966654e0
Stored in directory: /root/.cache/pip/wheels/04/04/bb/78c0daaafa679c767a2c5be25e36390f33ffecad7aee7ad10e
Building wheel for pathtools (setup.py) ... done
Created wheel for pathtools: filename=pathtools-0.1.2-py3-none-any.whl size=8806 sha256=5e4ec552f7cce27e9d24eaf26e127debe3cec69ad4c6602e813eeffb37825c80
Stored in directory: /root/.cache/pip/wheels/4c/8e/7e/72fbc243e1aeecae64a96875432e70d4e92f3d2d18123be004
Building wheel for treetable (setup.py) ... done
Created wheel for treetable: filename=treetable-0.2.5-py3-none-any.whl size=7347 sha256=b216741017440d59a07a6093846fa15dbfc1f2c34a9bd2762b3c6f963a479ed4
Stored in directory: /root/.cache/pip/wheels/20/ac/31/490968d6fc824620f9f04f03a2f90149bbbbcdb6c6e614909c
Successfully built openai-whisper whisperx demucs antlr4-python3-runtime julius sacremoses dora-search kaldi-python-io kaldiio nemo_toolkit wget distance docopt pescador pathtools treetable
Installing collected packages: wget, tokenizers, sentencepiece, sanic-routing, pydub, plac, pathtools, lameenc, docopt, distance, commonmark, braceexpand, antlr4-python3-runtime, youtokentome, websockets, webdataset, uvloop, urllib3, ujson, treetable, submitit, sox, sortedcollections, smmap, shellingham, setuptools, setproctitle, sacremoses, ruamel.yaml.clib, rich, retrying, rapidfuzz, PyYAML, pybind11, protobuf, pescador, Mako, loguru, lightning-utilities, kaldiio, kaldi-python-io, jedi, httptools, ffmpeg-python, einops, docker-pycreds, colorlog, colorama, cmaes, aiofiles, torchmetrics, tensorboardX, sentry-sdk, sanic, ruamel.yaml, pyannote.core, onnx, omegaconf, levenshtein, julius, gitdb, g2p_en, diffq, alembic, accelerate, texterrors, pyannote.database, optuna, openunmix, jiwer, hydra-core, GitPython, dora-search, wandb, transformers, pytorch-lightning, pyannote.pipeline, pyannote.metrics, librosa, demucs, openai-whisper, deepmultilingualpunctuation, pyannote.audio, nemo_toolkit, whisperx
Attempting uninstall: tokenizers
Found existing installation: tokenizers 0.13.2
Uninstalling tokenizers-0.13.2:
Successfully uninstalled tokenizers-0.13.2
Attempting uninstall: urllib3
Found existing installation: urllib3 1.24.3
Uninstalling urllib3-1.24.3:
Successfully uninstalled urllib3-1.24.3
Attempting uninstall: setuptools
Found existing installation: setuptools 57.4.0
Uninstalling setuptools-57.4.0:
Successfully uninstalled setuptools-57.4.0
Attempting uninstall: PyYAML
Found existing installation: PyYAML 6.0
Uninstalling PyYAML-6.0:
Successfully uninstalled PyYAML-6.0
Attempting uninstall: protobuf
Found existing installation: protobuf 3.19.6
Uninstalling protobuf-3.19.6:
Successfully uninstalled protobuf-3.19.6
Attempting uninstall: transformers
Found existing installation: transformers 4.26.1
Uninstalling transformers-4.26.1:
Successfully uninstalled transformers-4.26.1
Attempting uninstall: librosa
Found existing installation: librosa 0.8.1
Uninstalling librosa-0.8.1:
Successfully uninstalled librosa-0.8.1
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tensorflow 2.11.0 requires protobuf<3.20,>=3.9.2, but you have protobuf 3.20.3 which is incompatible.
Successfully installed GitPython-3.1.30 Mako-1.2.4 PyYAML-5.4.1 accelerate-0.16.0 aiofiles-23.1.0 alembic-1.9.3 antlr4-python3-runtime-4.9.3 braceexpand-0.1.7 cmaes-0.9.1 colorama-0.4.6 colorlog-6.7.0 commonmark-0.9.1 deepmultilingualpunctuation-1.0.1 demucs-4.0.0 diffq-0.2.3 distance-0.1.3 docker-pycreds-0.4.0 docopt-0.6.2 dora-search-0.1.11 einops-0.6.0 ffmpeg-python-0.2.0 g2p_en-2.1.0 gitdb-4.0.10 httptools-0.5.0 hydra-core-1.2.0 jedi-0.18.2 jiwer-2.5.1 julius-0.2.7 kaldi-python-io-1.2.2 kaldiio-2.17.2 lameenc-1.4.2 levenshtein-0.20.2 librosa-0.9.2 lightning-utilities-0.6.0.post0 loguru-0.6.0 nemo_toolkit-1.14.0 omegaconf-2.2.3 onnx-1.13.0 openai-whisper-20230124 openunmix-1.2.1 optuna-3.1.0 pathtools-0.1.2 pescador-2.1.0 plac-1.3.5 protobuf-3.20.3 pyannote.audio-1.1.2 pyannote.core-5.0.0 pyannote.database-4.1.3 pyannote.metrics-3.2.1 pyannote.pipeline-1.5.2 pybind11-2.10.3 pydub-0.25.1 pytorch-lightning-1.8.6 rapidfuzz-2.13.7 retrying-1.3.4 rich-12.6.0 ruamel.yaml-0.17.21 ruamel.yaml.clib-0.2.7 sacremoses-0.0.53 sanic-22.6.2 sanic-routing-22.3.0 sentencepiece-0.1.97 sentry-sdk-1.15.0 setproctitle-1.3.2 setuptools-59.5.0 shellingham-1.5.0.post1 smmap-5.0.0 sortedcollections-2.1.0 sox-1.4.1 submitit-1.4.5 tensorboardX-2.6 texterrors-0.4.4 tokenizers-0.12.1 torchmetrics-0.10.3 transformers-4.21.2 treetable-0.2.5 ujson-5.7.0 urllib3-1.26.14 uvloop-0.17.0 wandb-0.13.10 webdataset-0.1.62 websockets-10.4 wget-3.2 whisperx-1.0 youtokentome-1.0.6
[ ]
!python diarize.py -a cs.m4a Error
[NeMo W 2023-02-15 08:16:44 optimizers:55] Apex was not found. Using the lamb or fused_adam optimizer will error out.
[NeMo W 2023-02-15 08:16:44 nemo_logging:349] /usr/local/lib/python3.8/dist-packages/torch/jit/annotations.py:309: UserWarning: TorchScript will treat type annotations of Tensor dtype-specific subtypes as if they are normal Tensors. dtype constraints are not enforced in compilation either.
warnings.warn("TorchScript will treat type annotations of Tensor "
2023-02-15 08:16:44.842075: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-02-15 08:16:45.758284: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2023-02-15 08:16:45.758432: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2023-02-15 08:16:45.758454: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
Downloading: "https://dl.fbaipublicfiles.com/demucs/hybrid_transformer/f7e0c4bc-ba3fe64a.th" to /root/.cache/torch/hub/checkpoints/f7e0c4bc-ba3fe64a.th
100% 80.2M/80.2M [00:01<00:00, 47.3MB/s]
Downloading: "https://dl.fbaipublicfiles.com/demucs/hybrid_transformer/d12395a8-e57c48e6.th" to /root/.cache/torch/hub/checkpoints/d12395a8-e57c48e6.th
100% 80.2M/80.2M [00:01<00:00, 46.7MB/s]
Downloading: "https://dl.fbaipublicfiles.com/demucs/hybrid_transformer/92cfc3b6-ef3bcb9c.th" to /root/.cache/torch/hub/checkpoints/92cfc3b6-ef3bcb9c.th
100% 80.2M/80.2M [00:01<00:00, 49.4MB/s]
Downloading: "https://dl.fbaipublicfiles.com/demucs/hybrid_transformer/04573f0d-f3cf25b2.th" to /root/.cache/torch/hub/checkpoints/04573f0d-f3cf25b2.th
100% 80.2M/80.2M [00:01<00:00, 53.0MB/s]
Selected model is a bag of 4 models. You will see that many progress bars per track.
Separated tracks will be stored in /content/whisper-diarization/temp_outputs/htdemucs_ft
Separating track cs.m4a
100%|██████████████████████████████████████████████████████████████████████| 602.55/602.55 [00:32<00:00, 18.43seconds/s]
100%|██████████████████████████████████████████████████████████████████████| 602.55/602.55 [00:26<00:00, 22.90seconds/s]
100%|██████████████████████████████████████████████████████████████████████| 602.55/602.55 [00:26<00:00, 22.97seconds/s]
100%|██████████████████████████████████████████████████████████████████████| 602.55/602.55 [00:26<00:00, 22.69seconds/s]
100%|██████████████████████████████████████| 1.42G/1.42G [00:10<00:00, 146MiB/s]
100% 60000/60000 [00:13<00:00, 4384.90frames/s]
Downloading: "https://download.pytorch.org/torchaudio/models/wav2vec2_fairseq_base_ls960_asr_ls960.pth" to /root/.cache/torch/hub/checkpoints/wav2vec2_fairseq_base_ls960_asr_ls960.pth
100% 360M/360M [00:04<00:00, 93.3MB/s]
Failed to align segment ("."): no characters in this segment found in model dictionary, resorting to original...
Failed to align segment ("."): no characters in this segment found in model dictionary, resorting to original...
Failed to align segment ("."): no characters in this segment found in model dictionary, resorting to original...
Failed to align segment ("."): no characters in this segment found in model dictionary, resorting to original...
Failed to align segment ("."): no characters in this segment found in model dictionary, resorting to original...
Failed to align segment ("."): no characters in this segment found in model dictionary, resorting to original...
Failed to align segment ("."): no characters in this segment found in model dictionary, resorting to original...
Failed to align segment ("."): no characters in this segment found in model dictionary, resorting to original...
Failed to align segment ("."): no characters in this segment found in model dictionary, resorting to original...
Failed to align segment ("."): no characters in this segment found in model dictionary, resorting to original...
Failed to align segment ("."): no characters in this segment found in model dictionary, resorting to original...
Failed to align segment ("."): no characters in this segment found in model dictionary, resorting to original...
Failed to align segment ("."): no characters in this segment found in model dictionary, resorting to original...
Failed to align segment ("."): no characters in this segment found in model dictionary, resorting to original...
Failed to align segment ("."): no characters in this segment found in model dictionary, resorting to original...
Failed to align segment ("."): no characters in this segment found in model dictionary, resorting to original...
Failed to align segment ("."): no characters in this segment found in model dictionary, resorting to original...
Failed to align segment ("."): no characters in this segment found in model dictionary, resorting to original...
Failed to align segment ("."): no characters in this segment found in model dictionary, resorting to original...
Failed to align segment ("."): no characters in this segment found in model dictionary, resorting to original...
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /content/whisper-diarization/diarize.py:104 in <module> │
│ │
│ 101 alignment_model, metadata = whisperx.load_align_model( │
│ 102 │ language_code=whisper_results["language"], device=device │
│ 103 ) │
│ ❱ 104 result_aligned = whisperx.align( │
│ 105 │ whisper_results["segments"], alignment_model, metadata, vocal_targ │
│ 106 ) │
│ 107 │
│ │
│ /usr/local/lib/python3.8/dist-packages/whisperx/alignment.py:302 in align │
│ │
│ 299 │ not_space = char_segments_arr["char"] != " " │
│ 300 │ │
│ 301 │ per_seg_grp = char_segments_arr.groupby(["segment-idx", "subsegmen │
│ ❱ 302 │ char_segments_arr = per_seg_grp.apply(lambda x: x.reset_index(drop │
│ 303 │ per_word_grp = char_segments_arr[not_space].groupby(["segment-idx" │
│ 304 │ per_subseg_grp = char_segments_arr[not_space].groupby(["segment-id │
│ 305 │ per_seg_grp = char_segments_arr[not_space].groupby(["segment-idx"] │
│ │
│ /usr/local/lib/python3.8/dist-packages/pandas/util/_decorators.py:311 in │
│ wrapper │
│ │
│ 308 │ │ │ │ │ FutureWarning, │
│ 309 │ │ │ │ │ stacklevel=stacklevel, │
│ 310 │ │ │ │ ) │
│ ❱ 311 │ │ │ return func(*args, **kwargs) │
│ 312 │ │ │
│ 313 │ │ return wrapper │
│ 314 │
│ │
│ /usr/local/lib/python3.8/dist-packages/pandas/core/frame.py:5799 in │
│ reset_index │
│ │
│ 5796 │ │ │ │ │ │ level_values, lab, allow_fill=True, fill_val │
│ 5797 │ │ │ │ │ ) │
│ 5798 │ │ │ │ │
│ ❱ 5799 │ │ │ │ new_obj.insert(0, name, level_values) │
│ 5800 │ │ │
│ 5801 │ │ new_obj.index = new_index │
│ 5802 │ │ if not inplace: │
│ │
│ /usr/local/lib/python3.8/dist-packages/pandas/core/frame.py:4414 in insert │
│ │
│ 4411 │ │ │ ) │
│ 4412 │ │ if not allow_duplicates and column in self.columns: │
│ 4413 │ │ │ # Should this be a different kind of error?? │
│ ❱ 4414 │ │ │ raise ValueError(f"cannot insert {column}, already exist │
│ 4415 │ │ if not isinstance(loc, int): │
│ 4416 │ │ │ raise TypeError("loc must be int") │
│ 4417 │
╰──────────────────────────────────────────────────────────────────────────────╯
ValueError: cannot insert subsegment-idx, already exists
MahmoudAshraf97
commented
1 year ago Hi @jzysl423, colab is supported but it's constantly changing its environment so I cant track all changes, since this error happens in alignment part, try upgrading whisperx using the following command:
!pip install -U git+https://github.com/m-bain/whisperx.git
jzysl423
commented
1 year ago @MahmoudAshraf97 New error
[NeMo W 2023-02-15 11:03:38 optimizers:55] Apex was not found. Using the lamb or fused_adam optimizer will error out.
[NeMo W 2023-02-15 11:03:38 nemo_logging:349] /usr/local/lib/python3.8/dist-packages/torch/jit/annotations.py:309: UserWarning: TorchScript will treat type annotations of Tensor dtype-specific subtypes as if they are normal Tensors. dtype constraints are not enforced in compilation either.
warnings.warn("TorchScript will treat type annotations of Tensor "
2023-02-15 11:03:39.335406: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-02-15 11:03:41.433293: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2023-02-15 11:03:41.433547: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2023-02-15 11:03:41.433581: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
Selected model is a bag of 4 models. You will see that many progress bars per track.
Separated tracks will be stored in /content/whisper-diarization/temp_outputs/htdemucs_ft
Separating track cs.m4a
100%|██████████████████████████████████████████████████████████████████████| 602.55/602.55 [00:30<00:00, 19.90seconds/s]
100%|██████████████████████████████████████████████████████████████████████| 602.55/602.55 [00:29<00:00, 20.74seconds/s]
100%|██████████████████████████████████████████████████████████████████████| 602.55/602.55 [00:28<00:00, 21.10seconds/s]
100%|██████████████████████████████████████████████████████████████████████| 602.55/602.55 [00:28<00:00, 21.00seconds/s]
100%|█████████████████████████████████████| 1.42G/1.42G [00:49<00:00, 30.8MiB/s]
100% 60000/60000 [01:18<00:00, 764.31frames/s]
Downloading: "https://download.pytorch.org/torchaudio/models/wav2vec2_fairseq_base_ls960_asr_ls960.pth" to /root/.cache/torch/hub/checkpoints/wav2vec2_fairseq_base_ls960_asr_ls960.pth
100% 360M/360M [00:01<00:00, 212MB/s]
Failed to align segment (" I don't understand why, if you're into chicks anyway,"): backtrack failed, resorting to original...
[NeMo I 2023-02-15 11:08:51 msdd_models:1081] Loading pretrained diar_msdd_telephonic model from NGC
[NeMo I 2023-02-15 11:08:51 cloud:66] Downloading from: https://api.ngc.nvidia.com/v2/models/nvidia/nemo/diar_msdd_telephonic/versions/1.0.0/files/diar_msdd_telephonic.nemo to /root/.cache/torch/NeMo/NeMo_1.14.0/diar_msdd_telephonic/9c319f27168dc4980b8ba9a4ddd711bc/diar_msdd_telephonic.nemo
[NeMo I 2023-02-15 11:09:10 common:912] Instantiating model from pre-trained checkpoint
[NeMo W 2023-02-15 11:09:11 modelPT:142] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
Train config :
manifest_filepath: null
emb_dir: null
sample_rate: 16000
num_spks: 2
soft_label_thres: 0.5
labels: null
batch_size: 15
emb_batch_size: 0
shuffle: true
[NeMo W 2023-02-15 11:09:11 modelPT:149] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s).
Validation config :
manifest_filepath: null
emb_dir: null
sample_rate: 16000
num_spks: 2
soft_label_thres: 0.5
labels: null
batch_size: 15
emb_batch_size: 0
shuffle: false
[NeMo W 2023-02-15 11:09:11 modelPT:155] Please call the ModelPT.setup_test_data() or ModelPT.setup_multiple_test_data() method and provide a valid configuration file to setup the test data loader(s).
Test config :
manifest_filepath: null
emb_dir: null
sample_rate: 16000
num_spks: 2
soft_label_thres: 0.5
labels: null
batch_size: 15
emb_batch_size: 0
shuffle: false
seq_eval_mode: false
[NeMo I 2023-02-15 11:09:11 features:267] PADDING: 16
[NeMo I 2023-02-15 11:09:11 label_models:126] Setting angular: true/false in decoder is deprecated and will be removed in 1.13 version, use specific loss with _target_
[NeMo I 2023-02-15 11:09:11 features:267] PADDING: 16
[NeMo I 2023-02-15 11:09:12 save_restore_connector:243] Model EncDecDiarLabelModel was successfully restored from /root/.cache/torch/NeMo/NeMo_1.14.0/diar_msdd_telephonic/9c319f27168dc4980b8ba9a4ddd711bc/diar_msdd_telephonic.nemo.
[NeMo I 2023-02-15 11:09:12 label_models:126] Setting angular: true/false in decoder is deprecated and will be removed in 1.13 version, use specific loss with _target_
[NeMo I 2023-02-15 11:09:12 features:267] PADDING: 16
[NeMo I 2023-02-15 11:09:12 speaker_utils:92] Number of files to diarize: 1
[NeMo I 2023-02-15 11:09:12 clustering_diarizer:129] Loading pretrained vad_multilingual_marblenet model from NGC
[NeMo I 2023-02-15 11:09:12 cloud:66] Downloading from: https://api.ngc.nvidia.com/v2/models/nvidia/nemo/vad_multilingual_marblenet/versions/1.10.0/files/vad_multilingual_marblenet.nemo to /root/.cache/torch/NeMo/NeMo_1.14.0/vad_multilingual_marblenet/670f425c7f186060b7a7268ba6dfacb2/vad_multilingual_marblenet.nemo
[NeMo I 2023-02-15 11:09:15 common:912] Instantiating model from pre-trained checkpoint
[NeMo W 2023-02-15 11:09:15 modelPT:142] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
Train config :
manifest_filepath: /manifests/ami_train_0.63.json,/manifests/freesound_background_train.json,/manifests/freesound_laughter_train.json,/manifests/fisher_2004_background.json,/manifests/fisher_2004_speech_sampled.json,/manifests/google_train_manifest.json,/manifests/icsi_all_0.63.json,/manifests/musan_freesound_train.json,/manifests/musan_music_train.json,/manifests/musan_soundbible_train.json,/manifests/mandarin_train_sample.json,/manifests/german_train_sample.json,/manifests/spanish_train_sample.json,/manifests/french_train_sample.json,/manifests/russian_train_sample.json
sample_rate: 16000
labels:
- background
- speech
batch_size: 256
shuffle: true
is_tarred: false
tarred_audio_filepaths: null
tarred_shard_strategy: scatter
augmentor:
shift:
prob: 0.5
min_shift_ms: -10.0
max_shift_ms: 10.0
white_noise:
prob: 0.5
min_level: -90
max_level: -46
norm: true
noise:
prob: 0.5
manifest_path: /manifests/noise_0_1_musan_fs.json
min_snr_db: 0
max_snr_db: 30
max_gain_db: 300.0
norm: true
gain:
prob: 0.5
min_gain_dbfs: -10.0
max_gain_dbfs: 10.0
norm: true
num_workers: 16
pin_memory: true
[NeMo W 2023-02-15 11:09:15 modelPT:149] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s).
Validation config :
manifest_filepath: /manifests/ami_dev_0.63.json,/manifests/freesound_background_dev.json,/manifests/freesound_laughter_dev.json,/manifests/ch120_moved_0.63.json,/manifests/fisher_2005_500_speech_sampled.json,/manifests/google_dev_manifest.json,/manifests/musan_music_dev.json,/manifests/mandarin_dev.json,/manifests/german_dev.json,/manifests/spanish_dev.json,/manifests/french_dev.json,/manifests/russian_dev.json
sample_rate: 16000
labels:
- background
- speech
batch_size: 256
shuffle: false
val_loss_idx: 0
num_workers: 16
pin_memory: true
[NeMo W 2023-02-15 11:09:15 modelPT:155] Please call the ModelPT.setup_test_data() or ModelPT.setup_multiple_test_data() method and provide a valid configuration file to setup the test data loader(s).
Test config :
manifest_filepath: null
sample_rate: 16000
labels:
- background
- speech
batch_size: 128
shuffle: false
test_loss_idx: 0
[NeMo I 2023-02-15 11:09:15 features:267] PADDING: 16
[NeMo I 2023-02-15 11:09:16 save_restore_connector:243] Model EncDecClassificationModel was successfully restored from /root/.cache/torch/NeMo/NeMo_1.14.0/vad_multilingual_marblenet/670f425c7f186060b7a7268ba6dfacb2/vad_multilingual_marblenet.nemo.
[NeMo I 2023-02-15 11:09:16 msdd_models:855] Multiscale Weights: [1, 1, 1, 1, 1]
[NeMo I 2023-02-15 11:09:16 msdd_models:856] Clustering Parameters: {
"oracle_num_speakers": false,
"max_num_speakers": 8,
"enhanced_count_thres": 80,
"max_rp_threshold": 0.25,
"sparse_search_volume": 30,
"maj_vote_spk_count": false
}
[NeMo I 2023-02-15 11:09:16 speaker_utils:92] Number of files to diarize: 1
[NeMo I 2023-02-15 11:09:16 clustering_diarizer:303] Split long audio file to avoid CUDA memory issue
splitting manifest: 100% 1/1 [00:27<00:00, 27.92s/it]
[NeMo I 2023-02-15 11:09:43 classification_models:247] Perform streaming frame-level VAD
[NeMo I 2023-02-15 11:09:43 collections:296] Filtered duration for loading collection is 0.000000.
[NeMo I 2023-02-15 11:09:43 collections:300] # 12 files loaded accounting to # 1 labels
vad: 100% 12/12 [00:28<00:00, 2.36s/it]
[NeMo I 2023-02-15 11:10:12 clustering_diarizer:246] Generating predictions with overlapping input segments
generating preds: 100% 1/1 [00:09<00:00, 9.08s/it]
[NeMo I 2023-02-15 11:10:21 clustering_diarizer:258] Converting frame level prediction to speech/no-speech segment in start and end times format.
creating speech segments: 100% 1/1 [00:01<00:00, 1.33s/it]
[NeMo I 2023-02-15 11:10:22 clustering_diarizer:281] Subsegmentation for embedding extraction: scale0, nemo_outputs/speaker_outputs/subsegments_scale0.json
[NeMo I 2023-02-15 11:10:22 clustering_diarizer:336] Extracting embeddings for Diarization
[NeMo I 2023-02-15 11:10:22 collections:296] Filtered duration for loading collection is 0.000000.
[NeMo I 2023-02-15 11:10:22 collections:300] # 199 files loaded accounting to # 1 labels
[1/5] extract embeddings: 100% 4/4 [00:04<00:00, 1.10s/it]
[NeMo I 2023-02-15 11:10:27 clustering_diarizer:380] Saved embedding files to nemo_outputs/speaker_outputs/embeddings
[NeMo I 2023-02-15 11:10:27 clustering_diarizer:281] Subsegmentation for embedding extraction: scale1, nemo_outputs/speaker_outputs/subsegments_scale1.json
[NeMo I 2023-02-15 11:10:27 clustering_diarizer:336] Extracting embeddings for Diarization
[NeMo I 2023-02-15 11:10:27 collections:296] Filtered duration for loading collection is 0.000000.
[NeMo I 2023-02-15 11:10:27 collections:300] # 211 files loaded accounting to # 1 labels
[2/5] extract embeddings: 100% 4/4 [00:04<00:00, 1.15s/it]
[NeMo I 2023-02-15 11:10:31 clustering_diarizer:380] Saved embedding files to nemo_outputs/speaker_outputs/embeddings
[NeMo I 2023-02-15 11:10:31 clustering_diarizer:281] Subsegmentation for embedding extraction: scale2, nemo_outputs/speaker_outputs/subsegments_scale2.json
[NeMo I 2023-02-15 11:10:31 clustering_diarizer:336] Extracting embeddings for Diarization
[NeMo I 2023-02-15 11:10:31 collections:296] Filtered duration for loading collection is 0.000000.
[NeMo I 2023-02-15 11:10:31 collections:300] # 229 files loaded accounting to # 1 labels
[3/5] extract embeddings: 100% 4/4 [00:08<00:00, 2.23s/it]
[NeMo I 2023-02-15 11:10:40 clustering_diarizer:380] Saved embedding files to nemo_outputs/speaker_outputs/embeddings
[NeMo I 2023-02-15 11:10:40 clustering_diarizer:281] Subsegmentation for embedding extraction: scale3, nemo_outputs/speaker_outputs/subsegments_scale3.json
[NeMo I 2023-02-15 11:10:40 clustering_diarizer:336] Extracting embeddings for Diarization
[NeMo I 2023-02-15 11:10:40 collections:296] Filtered duration for loading collection is 0.000000.
[NeMo I 2023-02-15 11:10:40 collections:300] # 261 files loaded accounting to # 1 labels
[4/5] extract embeddings: 100% 5/5 [00:05<00:00, 1.09s/it]
[NeMo I 2023-02-15 11:10:46 clustering_diarizer:380] Saved embedding files to nemo_outputs/speaker_outputs/embeddings
[NeMo I 2023-02-15 11:10:46 clustering_diarizer:281] Subsegmentation for embedding extraction: scale4, nemo_outputs/speaker_outputs/subsegments_scale4.json
[NeMo I 2023-02-15 11:10:46 clustering_diarizer:336] Extracting embeddings for Diarization
[NeMo I 2023-02-15 11:10:46 collections:296] Filtered duration for loading collection is 0.000000.
[NeMo I 2023-02-15 11:10:46 collections:300] # 350 files loaded accounting to # 1 labels
[5/5] extract embeddings: 100% 6/6 [00:06<00:00, 1.16s/it]
[NeMo I 2023-02-15 11:10:53 clustering_diarizer:380] Saved embedding files to nemo_outputs/speaker_outputs/embeddings
clustering: 100% 1/1 [00:04<00:00, 4.08s/it]
[NeMo W 2023-02-15 11:10:57 der:105] Check if each ground truth RTTMs were present in the provided manifest file. Skipping calculation of Diariazation Error Rate
[NeMo I 2023-02-15 11:10:57 clustering_diarizer:455] Outputs are saved in /content/whisper-diarization/temp_outputs/nemo_outputs directory
[NeMo I 2023-02-15 11:10:57 msdd_models:951] Loading embedding pickle file of scale:0 at nemo_outputs/speaker_outputs/embeddings/subsegments_scale0_embeddings.pkl
[NeMo I 2023-02-15 11:10:57 msdd_models:951] Loading embedding pickle file of scale:1 at nemo_outputs/speaker_outputs/embeddings/subsegments_scale1_embeddings.pkl
[NeMo I 2023-02-15 11:10:57 msdd_models:951] Loading embedding pickle file of scale:2 at nemo_outputs/speaker_outputs/embeddings/subsegments_scale2_embeddings.pkl
[NeMo I 2023-02-15 11:10:57 msdd_models:951] Loading embedding pickle file of scale:3 at nemo_outputs/speaker_outputs/embeddings/subsegments_scale3_embeddings.pkl
[NeMo I 2023-02-15 11:10:57 msdd_models:951] Loading embedding pickle file of scale:4 at nemo_outputs/speaker_outputs/embeddings/subsegments_scale4_embeddings.pkl
[NeMo I 2023-02-15 11:10:57 msdd_models:929] Loading cluster label file from nemo_outputs/speaker_outputs/subsegments_scale4_cluster.label
[NeMo I 2023-02-15 11:10:57 collections:611] Filtered duration for loading collection is 0.000000.
[NeMo I 2023-02-15 11:10:57 collections:614] Total 6 session files loaded accounting to # 6 audio clips
100% 1/1 [00:00<00:00, 1.91it/s]
[NeMo I 2023-02-15 11:10:57 msdd_models:1393] [Threshold: 0.7000] [use_clus_as_main=False] [diar_window=50]
[NeMo I 2023-02-15 11:10:57 speaker_utils:92] Number of files to diarize: 1
[NeMo W 2023-02-15 11:10:57 der:105] Check if each ground truth RTTMs were present in the provided manifest file. Skipping calculation of Diariazation Error Rate
[NeMo I 2023-02-15 11:10:57 speaker_utils:92] Number of files to diarize: 1
[NeMo W 2023-02-15 11:10:58 der:105] Check if each ground truth RTTMs were present in the provided manifest file. Skipping calculation of Diariazation Error Rate
[NeMo I 2023-02-15 11:10:58 speaker_utils:92] Number of files to diarize: 1
[NeMo W 2023-02-15 11:10:58 der:105] Check if each ground truth RTTMs were present in the provided manifest file. Skipping calculation of Diariazation Error Rate
[NeMo I 2023-02-15 11:10:58 msdd_models:1414]
Downloading config.json: 100% 914/914 [00:00<00:00, 927kB/s]
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py:10 │
│ 02 in _get_module │
│ │
│ 999 │ │
│ 1000 │ def _get_module(self, module_name: str): │
│ 1001 │ │ try: │
│ ❱ 1002 │ │ │ return importlib.import_module("." + module_name, self.__ │
│ 1003 │ │ except Exception as e: │
│ 1004 │ │ │ raise RuntimeError( │
│ 1005 │ │ │ │ f"Failed to import {self.__name__}.{module_name} beca │
│ │
│ /usr/lib/python3.8/importlib/__init__.py:127 in import_module │
│ │
│ 124 │ │ │ if character != '.': │
│ 125 │ │ │ │ break │
│ 126 │ │ │ level += 1 │
│ ❱ 127 │ return _bootstrap._gcd_import(name[level:], package, level) │
│ 128 │
│ 129 │
│ 130 _RELOADING = {} │
│ <frozen importlib._bootstrap>:1014 in _gcd_import │
│ <frozen importlib._bootstrap>:991 in _find_and_load │
│ <frozen importlib._bootstrap>:975 in _find_and_load_unlocked │
│ <frozen importlib._bootstrap>:671 in _load_unlocked │
│ <frozen importlib._bootstrap_external>:848 in exec_module │
│ <frozen importlib._bootstrap>:219 in _call_with_frames_removed │
│ │
│ /usr/local/lib/python3.8/dist-packages/transformers/models/xlm_roberta/model │
│ ing_tf_xlm_roberta.py:19 in <module> │
│ │
│ 16 """ TF 2.0 XLM-RoBERTa model.""" │
│ 17 │
│ 18 from ...utils import add_start_docstrings, logging │
│ ❱ 19 from ..roberta.modeling_tf_roberta import ( │
│ 20 │ TFRobertaForCausalLM, │
│ 21 │ TFRobertaForMaskedLM, │
│ 22 │ TFRobertaForMultipleChoice, │
│ │
│ /usr/local/lib/python3.8/dist-packages/transformers/models/roberta/modeling_ │
│ tf_roberta.py:36 in <module> │
│ │
│ 33 │ TFSequenceClassifierOutput, │
│ 34 │ TFTokenClassifierOutput, │
│ 35 ) │
│ ❱ 36 from ...modeling_tf_utils import ( │
│ 37 │ TFCausalLanguageModelingLoss, │
│ 38 │ TFMaskedLanguageModelingLoss, │
│ 39 │ TFModelInputType, │
│ │
│ /usr/local/lib/python3.8/dist-packages/transformers/modeling_tf_utils.py:38 │
│ in <module> │
│ │
│ 35 from tensorflow.python.keras.saving import hdf5_format │
│ 36 │
│ 37 from huggingface_hub import Repository, list_repo_files │
│ ❱ 38 from keras.saving.hdf5_format import save_attributes_to_hdf5_group │
│ 39 from requests import HTTPError │
│ 40 from transformers.utils.hub import convert_file_size_to_int, get_chec │
│ 41 │
╰──────────────────────────────────────────────────────────────────────────────╯
ModuleNotFoundError: No module named 'keras.saving.hdf5_format'
The above exception was the direct cause of the following exception:
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /content/whisper-diarization/diarize.py:145 in <module> │
│ │
│ 142 │
│ 143 if whisper_results["language"] in punct_model_langs: │
│ 144 │ # restoring punctuation in the transcript to help realign the sent │
│ ❱ 145 │ punct_model = PunctuationModel(model="kredor/punctuate-all") │
│ 146 │ │
│ 147 │ words_list = list(map(lambda x: x["word"], wsm)) │
│ 148 │
│ │
│ /usr/local/lib/python3.8/dist-packages/deepmultilingualpunctuation/punctuati │
│ onmodel.py:9 in __init__ │
│ │
│ 6 class PunctuationModel(): │
│ 7 │ def __init__(self, model = "oliverguhr/fullstop-punctuation-multila │
│ 8 │ │ if torch.cuda.is_available(): │
│ ❱ 9 │ │ │ self.pipe = pipeline("ner",model, grouped_entities=False, d │
│ 10 │ │ else: │
│ 11 │ │ │ self.pipe = pipeline("ner",model, grouped_entities=False) │
│ 12 │
│ │
│ /usr/local/lib/python3.8/dist-packages/transformers/pipelines/__init__.py:65 │
│ 0 in pipeline │
│ │
│ 647 │ # Forced if framework already defined, inferred if it's None │
│ 648 │ # Will load the correct model if possible │
│ 649 │ model_classes = {"tf": targeted_task["tf"], "pt": targeted_task["p │
│ ❱ 650 │ framework, model = infer_framework_load_model( │
│ 651 │ │ model, │
│ 652 │ │ model_classes=model_classes, │
│ 653 │ │ config=config, │
│ │
│ /usr/local/lib/python3.8/dist-packages/transformers/pipelines/base.py:233 in │
│ infer_framework_load_model │
│ │
│ 230 │ │ │ │ │ if _class is not None: │
│ 231 │ │ │ │ │ │ classes.append(_class) │
│ 232 │ │ │ │ if look_tf: │
│ ❱ 233 │ │ │ │ │ _class = getattr(transformers_module, f"TF{archit │
│ 234 │ │ │ │ │ if _class is not None: │
│ 235 │ │ │ │ │ │ classes.append(_class) │
│ 236 │ │ │ class_tuple = class_tuple + tuple(classes) │
│ │
│ /usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py:99 │
│ 3 in __getattr__ │
│ │
│ 990 │ │ │ value = self._get_module(name) │
│ 991 │ │ elif name in self._class_to_module.keys(): │
│ 992 │ │ │ module = self._get_module(self._class_to_module[name]) │
│ ❱ 993 │ │ │ value = getattr(module, name) │
│ 994 │ │ else: │
│ 995 │ │ │ raise AttributeError(f"module {self.__name__} has no attr │
│ 996 │
│ │
│ /usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py:99 │
│ 2 in __getattr__ │
│ │
│ 989 │ │ if name in self._modules: │
│ 990 │ │ │ value = self._get_module(name) │
│ 991 │ │ elif name in self._class_to_module.keys(): │
│ ❱ 992 │ │ │ module = self._get_module(self._class_to_module[name]) │
│ 993 │ │ │ value = getattr(module, name) │
│ 994 │ │ else: │
│ 995 │ │ │ raise AttributeError(f"module {self.__name__} has no attr │
│ │
│ /usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py:10 │
│ 04 in _get_module │
│ │
│ 1001 │ │ try: │
│ 1002 │ │ │ return importlib.import_module("." + module_name, self.__ │
│ 1003 │ │ except Exception as e: │
│ ❱ 1004 │ │ │ raise RuntimeError( │
│ 1005 │ │ │ │ f"Failed to import {self.__name__}.{module_name} beca │
│ 1006 │ │ │ │ f" traceback):\n{e}" │
│ 1007 │ │ │ ) from e │
╰──────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Failed to import
transformers.models.xlm_roberta.modeling_tf_xlm_roberta because of the following
error (look up to see its traceback):
No module named 'keras.saving.hdf5_format'@jzysl423 upgrade transformers library
!pip install -U transformers
KevinGeLe
commented
1 year ago The version should be transformers-4.26.1.
jzysl423
commented
1 year ago Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/, https://download.pytorch.org/whl/cu116
Collecting nemo_toolkit[asr]
Cloning https://github.com/NVIDIA/NeMo.git (to revision r1.14.0) to /tmp/pip-install-l93_7ris/nemo-toolkit_97f73647cae74c978c9b80bb790e2e61
Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA/NeMo.git /tmp/pip-install-l93_7ris/nemo-toolkit_97f73647cae74c978c9b80bb790e2e61
Running command git checkout -b r1.14.0 --track origin/r1.14.0
Switched to a new branch 'r1.14.0'
Branch 'r1.14.0' set up to track remote branch 'r1.14.0' from 'origin'.
Resolved https://github.com/NVIDIA/NeMo.git to commit 0a0b8a1bbaf9f884cf3abce2ceeb149f68ed114c
Preparing metadata (setup.py) ... done
Collecting git+https://github.com/openai/whisper.git (from -r ./requirements.txt (line 12))
Cloning https://github.com/openai/whisper.git to /tmp/pip-req-build-h5k8dndc
Running command git clone --filter=blob:none --quiet https://github.com/openai/whisper.git /tmp/pip-req-build-h5k8dndc
Resolved https://github.com/openai/whisper.git to commit 7858aa9c08d98f75575035ecd6481f462d66ca27
Preparing metadata (setup.py) ... done
Collecting git+https://github.com/m-bain/whisperX.git@d687cf33588dac51536d7d0edb840ac92a69df11 (from -r ./requirements.txt (line 13))
Cloning https://github.com/m-bain/whisperX.git (to revision d687cf33588dac51536d7d0edb840ac92a69df11) to /tmp/pip-req-build-6gtqrc9l
Running command git clone --filter=blob:none --quiet https://github.com/m-bain/whisperX.git /tmp/pip-req-build-6gtqrc9l
Running command git rev-parse -q --verify 'sha^d687cf33588dac51536d7d0edb840ac92a69df11'
Running command git fetch -q https://github.com/m-bain/whisperX.git d687cf33588dac51536d7d0edb840ac92a69df11
Running command git checkout -q d687cf33588dac51536d7d0edb840ac92a69df11
Resolved https://github.com/m-bain/whisperX.git to commit d687cf33588dac51536d7d0edb840ac92a69df11
Preparing metadata (setup.py) ... done
Collecting demucs
Cloning https://github.com/facebookresearch/demucs to /tmp/pip-install-l93_7ris/demucs_60fa2c82ef5545288aed8e4729807c19
Running command git clone --filter=blob:none --quiet https://github.com/facebookresearch/demucs /tmp/pip-install-l93_7ris/demucs_60fa2c82ef5545288aed8e4729807c19
Resolved https://github.com/facebookresearch/demucs to commit 3cb9d237c68b45d32719ee5e2e8fae6bf04e5088
Preparing metadata (setup.py) ... done
Requirement already satisfied: torch==1.13.1+cu116 in /usr/local/lib/python3.8/dist-packages (from -r ./requirements.txt (line 2)) (1.13.1+cu116)
Requirement already satisfied: torchvision==0.14.1+cu116 in /usr/local/lib/python3.8/dist-packages (from -r ./requirements.txt (line 3)) (0.14.1+cu116)
Requirement already satisfied: torchaudio==0.13.1+cu116 in /usr/local/lib/python3.8/dist-packages (from -r ./requirements.txt (line 4)) (0.13.1+cu116)
Requirement already satisfied: torchtext==0.14.1 in /usr/local/lib/python3.8/dist-packages (from -r ./requirements.txt (line 5)) (0.14.1)
Collecting pytorch-lightning==1.8.6
Downloading pytorch_lightning-1.8.6-py3-none-any.whl (800 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 800.3/800.3 KB 30.7 MB/s eta 0:00:00
Collecting sanic==22.6.2
Downloading sanic-22.6.2-py3-none-any.whl (271 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 271.9/271.9 KB 33.9 MB/s eta 0:00:00
Collecting transformers==4.26.1
Downloading transformers-4.26.1-py3-none-any.whl (6.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.3/6.3 MB 100.8 MB/s eta 0:00:00
Collecting accelerate
Downloading accelerate-0.16.0-py3-none-any.whl (199 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 199.7/199.7 KB 28.3 MB/s eta 0:00:00
Requirement already satisfied: text-unidecode in /usr/local/lib/python3.8/dist-packages (from -r ./requirements.txt (line 10)) (1.3)
Collecting deepmultilingualpunctuation
Downloading deepmultilingualpunctuation-1.0.1-py3-none-any.whl (5.4 kB)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.8/dist-packages (from torch==1.13.1+cu116->-r ./requirements.txt (line 2)) (4.4.0)
Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from torchvision==0.14.1+cu116->-r ./requirements.txt (line 3)) (1.21.6)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.8/dist-packages (from torchvision==0.14.1+cu116->-r ./requirements.txt (line 3)) (7.1.2)
Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from torchvision==0.14.1+cu116->-r ./requirements.txt (line 3)) (2.25.1)
Requirement already satisfied: tqdm in /usr/local/lib/python3.8/dist-packages (from torchtext==0.14.1->-r ./requirements.txt (line 5)) (4.64.1)
Requirement already satisfied: fsspec[http]>2021.06.0 in /usr/local/lib/python3.8/dist-packages (from pytorch-lightning==1.8.6->-r ./requirements.txt (line 6)) (2023.1.0)
Collecting tensorboardX>=2.2
Downloading tensorboardX-2.6-py2.py3-none-any.whl (114 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 114.5/114.5 KB 14.7 MB/s eta 0:00:00
Requirement already satisfied: packaging>=17.0 in /usr/local/lib/python3.8/dist-packages (from pytorch-lightning==1.8.6->-r ./requirements.txt (line 6)) (23.0)
Collecting lightning-utilities!=0.4.0,>=0.3.0
Downloading lightning_utilities-0.6.0.post0-py3-none-any.whl (18 kB)
Requirement already satisfied: PyYAML>=5.4 in /usr/local/lib/python3.8/dist-packages (from pytorch-lightning==1.8.6->-r ./requirements.txt (line 6)) (6.0)
Collecting torchmetrics>=0.7.0
Downloading torchmetrics-0.11.1-py3-none-any.whl (517 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 517.2/517.2 KB 48.2 MB/s eta 0:00:00
Requirement already satisfied: multidict<7.0,>=5.0 in /usr/local/lib/python3.8/dist-packages (from sanic==22.6.2->-r ./requirements.txt (line 7)) (6.0.4)
Collecting httptools>=0.0.10
Downloading httptools-0.5.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (427 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 427.8/427.8 KB 40.8 MB/s eta 0:00:00
Collecting aiofiles>=0.6.0
Downloading aiofiles-23.1.0-py3-none-any.whl (14 kB)
Collecting uvloop>=0.5.3
Downloading uvloop-0.17.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.6/4.6 MB 67.5 MB/s eta 0:00:00
Collecting ujson>=1.35
Downloading ujson-5.7.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (52 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 52.8/52.8 KB 7.0 MB/s eta 0:00:00
Collecting websockets>=10.0
Downloading websockets-10.4-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (106 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 107.0/107.0 KB 13.1 MB/s eta 0:00:00
Collecting sanic-routing<22.6.0,>=22.3.0
Downloading sanic_routing-22.3.0-py3-none-any.whl (25 kB)
Collecting huggingface-hub<1.0,>=0.11.0
Downloading huggingface_hub-0.12.0-py3-none-any.whl (190 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 190.3/190.3 KB 26.1 MB/s eta 0:00:00
Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from transformers==4.26.1->-r ./requirements.txt (line 8)) (3.9.0)
Collecting tokenizers!=0.11.3,<0.14,>=0.11.1
Downloading tokenizers-0.13.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.6/7.6 MB 103.7 MB/s eta 0:00:00
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.8/dist-packages (from transformers==4.26.1->-r ./requirements.txt (line 8)) (2022.6.2)
Requirement already satisfied: psutil in /usr/local/lib/python3.8/dist-packages (from accelerate->-r ./requirements.txt (line 9)) (5.4.8)
Requirement already satisfied: numba in /usr/local/lib/python3.8/dist-packages (from nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.56.4)
Collecting onnx>=1.7.0
Downloading onnx-1.13.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.5/13.5 MB 96.4 MB/s eta 0:00:00
Requirement already satisfied: python-dateutil in /usr/local/lib/python3.8/dist-packages (from nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (2.8.2)
Collecting ruamel.yaml
Downloading ruamel.yaml-0.17.21-py3-none-any.whl (109 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 109.5/109.5 KB 16.7 MB/s eta 0:00:00
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.8/dist-packages (from nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (1.0.2)
Collecting setuptools==59.5.0
Downloading setuptools-59.5.0-py3-none-any.whl (952 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 952.4/952.4 KB 72.4 MB/s eta 0:00:00
Requirement already satisfied: tensorboard in /usr/local/lib/python3.8/dist-packages (from nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (2.11.2)
Collecting wget
Downloading wget-3.2.zip (10 kB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: wrapt in /usr/local/lib/python3.8/dist-packages (from nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (1.14.1)
Collecting braceexpand
Downloading braceexpand-0.1.7-py2.py3-none-any.whl (5.9 kB)
Requirement already satisfied: editdistance in /usr/local/lib/python3.8/dist-packages (from nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.5.3)
Collecting g2p_en
Downloading g2p_en-2.1.0-py3-none-any.whl (3.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.1/3.1 MB 77.1 MB/s eta 0:00:00
Requirement already satisfied: inflect in /usr/local/lib/python3.8/dist-packages (from nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (2.1.0)
Requirement already satisfied: ipywidgets in /usr/local/lib/python3.8/dist-packages (from nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (7.7.1)
Collecting jiwer
Downloading jiwer-2.5.1-py3-none-any.whl (15 kB)
Collecting kaldi-python-io
Downloading kaldi-python-io-1.2.2.tar.gz (8.8 kB)
Preparing metadata (setup.py) ... done
Collecting kaldiio
Downloading kaldiio-2.17.2.tar.gz (24 kB)
Preparing metadata (setup.py) ... done
Collecting librosa>=0.9.0
Downloading librosa-0.9.2-py3-none-any.whl (214 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 214.3/214.3 KB 25.9 MB/s eta 0:00:00
Requirement already satisfied: marshmallow in /usr/local/lib/python3.8/dist-packages (from nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (3.19.0)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.8/dist-packages (from nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (3.2.2)
Collecting pyannote.core
Downloading pyannote.core-5.0.0-py3-none-any.whl (58 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.5/58.5 KB 8.2 MB/s eta 0:00:00
Collecting pyannote.metrics
Downloading pyannote.metrics-3.2.1-py3-none-any.whl (51 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 51.4/51.4 KB 7.6 MB/s eta 0:00:00
Collecting pydub
Downloading pydub-0.25.1-py2.py3-none-any.whl (32 kB)
Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.8/dist-packages (from nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (1.7.3)
Requirement already satisfied: soundfile in /usr/local/lib/python3.8/dist-packages (from nemo_toolkit[asr]->-r ./requirements.txt (line 11)) (0.11.0)
Collecting sox
Downloading sox-1.4.1-py2.py3-none-any.whl (39 kB)
Collecting texterrors
Downloading texterrors-0.4.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 65.0 MB/s eta 0:00:00
Collecting hydra-core<1.3,>=1.2.0
Downloading hydra_core-1.2.0-py3-none-any.whl (151 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 151.1/151.1 KB 11.4 MB/s eta 0:00:00
Collecting omegaconf<2.3,>=2.2
Downloading omegaconf-2.2.3-py3-none-any.whl (79 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.3/79.3 KB 12.4 MB/s eta 0:00:00
Collecting PyYAML>=5.4
Downloading PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl (662 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 662.4/662.4 KB 52.5 MB/s eta 0:00:00
Collecting torchmetrics>=0.7.0
Downloading torchmetrics-0.10.3-py3-none-any.whl (529 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 529.7/529.7 KB 58.3 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of text-unidecode to determine which version is compatible with other requirements. This could take a while.
Collecting text-unidecode
Downloading text_unidecode-1.3-py2.py3-none-any.whl (78 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.2/78.2 KB 11.3 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of accelerate to determine which version is compatible with other requirements. This could take a while.
Collecting accelerate
Downloading accelerate-0.15.0-py3-none-any.whl (191 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 191.5/191.5 KB 25.8 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of transformers to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of sanic to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of pytorch-lightning to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of torchtext to determine which version is compatible with other requirements. This could take a while.
Collecting torchtext==0.14.1
Downloading torchtext-0.14.1-cp38-cp38-manylinux1_x86_64.whl (2.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 87.2 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of torchaudio to determine which version is compatible with other requirements. This could take a while.
Collecting torchaudio==0.13.1+cu116
Downloading https://download.pytorch.org/whl/cu116/torchaudio-0.13.1%2Bcu116-cp38-cp38-linux_x86_64.whl (4.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.2/4.2 MB 31.5 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of torchvision to determine which version is compatible with other requirements. This could take a while.
Collecting torchvision==0.14.1+cu116
Downloading https://download.pytorch.org/whl/cu116/torchvision-0.14.1%2Bcu116-cp38-cp38-linux_x86_64.whl (24.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24.2/24.2 MB 17.2 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of sanic to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of pytorch-lightning to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of torchtext to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of torch to determine which version is compatible with other requirements. This could take a while.
Collecting torch==1.13.1+cu116
Downloading https://download.pytorch.org/whl/cu116/torch-1.13.1%2Bcu116-cp38-cp38-linux_x86_64.whl (1977.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 GB 866.4 kB/s eta 0:00:00
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
INFO: pip is looking at multiple versions of torchaudio to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install nemo-toolkit[asr]==1.14.0 and transformers==4.26.1 because these package versions have conflicting dependencies.
The conflict is caused by:
The user requested transformers==4.26.1
nemo-toolkit[asr] 1.14.0 depends on transformers<=4.21.2 and >=4.0.1
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts@jzysl423 for the time being please use !pip install --no-deps -r requirements.txt
1397v
commented
1 year ago Hi, I just executed it in Google Colab and got this error:
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /content/whisper-diarization/diarize.py:101 in <module> │
│ │
│ 98 torch.cuda.empty_cache() │
│ 99 │
│ 100 device = "cuda" │
│ ❱ 101 alignment_model, metadata = whisperx.load_align_model( │
│ 102 │ language_code=whisper_results["language"], device=device │
│ 103 ) │
│ 104 result_aligned = whisperx.align( │
│ │
│ /usr/local/lib/python3.8/dist-packages/whisperx/alignment.py:58 in │
│ load_align_model │
│ │
│ 55 │ if model_name in torchaudio.pipelines.__all__: │
│ 56 │ │ pipeline_type = "torchaudio" │
│ 57 │ │ bundle = torchaudio.pipelines.__dict__[model_name] │
│ ❱ 58 │ │ align_model = bundle.get_model().to(device) │
│ 59 │ │ labels = bundle.get_labels() │
│ 60 │ │ align_dictionary = {c.lower(): i for i, c in enumerate(labels) │
│ 61 │ else: │
│ │
│ /usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:989 in to │
│ │
│ 986 │ │ │ │ │ │ │ non_blocking, memory_format=convert_to_fo │
│ 987 │ │ │ return t.to(device, dtype if t.is_floating_point() or t.i │
│ 988 │ │ │
│ ❱ 989 │ │ return self._apply(convert) │
│ 990 │ │
│ 991 │ def register_backward_hook( │
│ 992 │ │ self, hook: Callable[['Module', _grad_t, _grad_t], Union[None │
│ │
│ /usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:641 in │
│ _apply │
│ │
│ 638 │ │
│ 639 │ def _apply(self, fn): │
│ 640 │ │ for module in self.children(): │
│ ❱ 641 │ │ │ module._apply(fn) │
│ 642 │ │ │
│ 643 │ │ def compute_should_use_set_data(tensor, tensor_applied): │
│ 644 │ │ │ if torch._has_compatible_shallow_copy_type(tensor, tensor │
│ │
│ /usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:641 in │
│ _apply │
│ │
│ 638 │ │
│ 639 │ def _apply(self, fn): │
│ 640 │ │ for module in self.children(): │
│ ❱ 641 │ │ │ module._apply(fn) │
│ 642 │ │ │
│ 643 │ │ def compute_should_use_set_data(tensor, tensor_applied): │
│ 644 │ │ │ if torch._has_compatible_shallow_copy_type(tensor, tensor │
│ │
│ /usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:641 in │
│ _apply │
│ │
│ 638 │ │
│ 639 │ def _apply(self, fn): │
│ 640 │ │ for module in self.children(): │
│ ❱ 641 │ │ │ module._apply(fn) │
│ 642 │ │ │
│ 643 │ │ def compute_should_use_set_data(tensor, tensor_applied): │
│ 644 │ │ │ if torch._has_compatible_shallow_copy_type(tensor, tensor │
│ │
│ /usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:641 in │
│ _apply │
│ │
│ 638 │ │
│ 639 │ def _apply(self, fn): │
│ 640 │ │ for module in self.children(): │
│ ❱ 641 │ │ │ module._apply(fn) │
│ 642 │ │ │
│ 643 │ │ def compute_should_use_set_data(tensor, tensor_applied): │
│ 644 │ │ │ if torch._has_compatible_shallow_copy_type(tensor, tensor │
│ │
│ /usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:664 in │
│ _apply │
│ │
│ 661 │ │ │ # track autograd history of `param_applied`, so we have t │
│ 662 │ │ │ # `with torch.no_grad():` │
│ 663 │ │ │ with torch.no_grad(): │
│ ❱ 664 │ │ │ │ param_applied = fn(param) │
│ 665 │ │ │ should_use_set_data = compute_should_use_set_data(param, │
│ 666 │ │ │ if should_use_set_data: │
│ 667 │ │ │ │ param.data = param_applied │
│ │
│ /usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:987 in │
│ convert │
│ │
│ 984 │ │ │ if convert_to_format is not None and t.dim() in (4, 5): │
│ 985 │ │ │ │ return t.to(device, dtype if t.is_floating_point() or │
│ 986 │ │ │ │ │ │ │ non_blocking, memory_format=convert_to_fo │
│ ❱ 987 │ │ │ return t.to(device, dtype if t.is_floating_point() or t.i │
│ 988 │ │ │
│ 989 │ │ return self._apply(convert) │
│ 990 │
│ │
│ /usr/local/lib/python3.8/dist-packages/torch/cuda/__init__.py:229 in │
│ _lazy_init │
│ │
│ 226 │ │ # are found or any other error occurs │
│ 227 │ │ if 'CUDA_MODULE_LOADING' not in os.environ: │
│ 228 │ │ │ os.environ['CUDA_MODULE_LOADING'] = 'LAZY' │
│ ❱ 229 │ │ torch._C._cuda_init() │
│ 230 │ │ # Some of the queued calls may reentrantly call _lazy_init(); │
│ 231 │ │ # we need to just return without initializing in that case. │
│ 232 │ │ # However, we must not let any *other* threads in! │
╰──────────────────────────────────────────────────────────────────────────────╯
RuntimeError: No CUDA GPUs are available
adamjonas
commented
1 year ago @1397v did you change the runtime type (runtime > change runtime type) to use GPUs? That's required in order to run it.
1397v
commented
1 year ago @adamjonas Hi! Yes, just found that solution in another thread. Unfortunately, I'm now solving all the dependencies issues. It seems that if I change the runtime, installation of the requirements.txt file dependencies aren't enough.
MahmoudAshraf97
commented
1 year ago @1397v can you tell me what extra requirements are needed?
1397v
commented
1 year ago @MahmoudAshraf97 For some reason, I don't know why, it's throwing the error ModuleNotFoundError with all these modules I am installing one by one as they come up... until now I have installed:
!pip install wget
!pip install omegaconf
!pip install ffmpeg
!pip install huggingface_hub
!pip install pyannote.audio
!pip install hydra-core
!pip install tokenizers
!pip install git+https://github.com/PyTorchLightning/pytorch-lightning
!pip install --upgrade pytorch-lightning # I had to upgrade after installing from github or else it didn't recognize the module
!pip install onnx
!pip install braceexpand
!pip install webdatasetI don't know what more modules it will need...
Before changing the runtime it was processing the file until the RuntimeError: No CUDA GPUs are available error.
MahmoudAshraf97
commented
1 year ago all of these are NeMo dependencies, can you try pulling the requirements file from the repo and install it? although that migh cause a problem with transformers library but worth a shot
1397v
commented
1 year ago @MahmoudAshraf97 It's the first thing I did:
Did I do it correctly?
1397v
commented
1 year ago I just succeded running it. Thanks so much for the quick answers!!
Unfortunately it doesn't work well with Spanish even using the --whisper-model large option.
This is the list of all dependencies I had to run:
!pip install wget
!pip install omegaconf
!pip install ffmpeg
!pip install huggingface_hub
!pip install pyannote.audio
!pip install hydra-core
!pip install tokenizers
!pip install git+https://github.com/PyTorchLightning/pytorch-lightning
!pip install --upgrade pytorch-lightning # I had to upgrade after installing from github or else it didn't recognize the module
!pip install onnx
!pip install braceexpand
!pip install webdataset
!pip install youtokentome
!pip install jiwer
!pip install ffmpeg-pythonAlso, here is a list of warnings I get at the beginning, I don't know if those might cause the diarization to be so inaccurate:
2023-02-27 22:22:50.702864: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-02-27 22:22:52.756825: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2023-02-27 22:22:52.756968: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2023-02-27 22:22:52.756989: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
[NeMo W 2023-02-27 22:22:59 optimizers:55] Apex was not found. Using the lamb or fused_adam optimizer will error out.
[NeMo W 2023-02-27 22:23:00 experimental:27] Module <class 'nemo.collections.asr.models.audio_to_audio_model.AudioToAudioModel'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-02-27 22:23:00 experimental:27] Module <class 'nemo.collections.asr.modules.audio_modules.SpectrogramToMultichannelFeatures'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-02-27 22:23:00 nemo_logging:349] /usr/local/lib/python3.8/dist-packages/torch/jit/annotations.py:309: UserWarning: TorchScript will treat type annotations of Tensor dtype-specific subtypes as if they are normal Tensors. dtype constraints are not enforced in compilation either.
warnings.warn("TorchScript will treat type annotations of Tensor "
[NeMo W 2023-02-27 22:23:01 experimental:27] Module <class 'nemo.collections.asr.data.audio_to_audio.BaseAudioDataset'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-02-27 22:23:01 experimental:27] Module <class 'nemo.collections.asr.data.audio_to_audio.AudioToTargetDataset'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-02-27 22:23:01 experimental:27] Module <class 'nemo.collections.asr.data.audio_to_audio.AudioToTargetWithReferenceDataset'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-02-27 22:23:01 experimental:27] Module <class 'nemo.collections.asr.data.audio_to_audio.AudioToTargetWithEmbeddingDataset'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-02-27 22:23:01 experimental:27] Module <class 'nemo.collections.asr.models.enhancement_models.EncMaskDecAudioToAudioModel'> is experimental, not ready for production and is not fully supported. Use at your own risk.
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.8/dist-packages/demucs/separate.py", line 12, in <module>
from dora.log import fatal
ModuleNotFoundError: No module named 'dora'
Source splitting failed, using original audio file. Use --no-stem argument to disable it.If the file contained music along the speech, this will cause inaccuracies, as I see in the log the source splitting failed which addresses this problem, it failed because it's missing a module called dora
1397v
commented
1 year ago Yes, I tried installing dora with this command and the error continued: !pip install dora.
I think the bad quality of the diarization was due to the 2 speakers having similar voice tones and speaking quick one after the other. I tried with another audio and it worked much better.
MahmoudAshraf97
commented
1 year ago @jzysl423 @adamjonas @1397v I added a colab notebook with all the requirements, it should work out of the box
!git clone https://github.com/MahmoudAshraf97/whisper-diarizationcd /content/whisper-diarization/!pip install -r ./requirements.txt!python diarize.py -a cs.m4aI had a problem running on the Colab platform and didn't know how to fix it.