PicklingError: Can't pickle <class 'nemo.collections.common.parts.preprocessing.collections.SpeechLabelEntity'>: attribute lookup SpeechLabelEntity on nemo.collections.common.parts.preprocessing.collections failed

mayur-neuralgarage commented 10 months ago

Initialize NeMo MSDD diarization model

msdd_model = NeuralDiarizer(cfg=create_config(temp_path)).to("cuda")

msdd_model.diarize()


PicklingError                             Traceback (most recent call last)
Cell In[37], line 3
      1 # Initialize NeMo MSDD diarization model
      2 msdd_model = NeuralDiarizer(cfg=create_config(temp_path)).to("cuda")
----> 3 msdd_model.diarize()
      5 del msdd_model
      6 torch.cuda.empty_cache()

PicklingError: Can't pickle <class 'nemo.collections.common.parts.preprocessing.collections.SpeechLabelEntity'>: attribute lookup SpeechLabelEntity on nemo.collections.common.parts.preprocessing.collections failed```

DanielArmyrConversy commented 10 months ago

I just saw this one too. Unfortunately, my example is deep in a rather complex piece of code, so I cannot extract a minimum viable example at this time. We will do some restructure on it, though, and then I can get back to you.

[...]
  File "/Users/danielarmyr/Library/Caches/pypoetry/virtualenvs/vbi-nIDHWX03-py3.10/lib/python3.10/site-packages/nemo/collections/asr/models/clustering_diarizer.py", line 447, in diarize
    self._extract_embeddings(self.subsegments_manifest_path, scale_idx, len(scales))

  File "/Users/danielarmyr/Library/Caches/pypoetry/virtualenvs/vbi-nIDHWX03-py3.10/lib/python3.10/site-packages/nemo/collections/asr/models/clustering_diarizer.py", line 350, in _extract_embeddings
    for test_batch in tqdm(

  File "/Users/danielarmyr/Library/Caches/pypoetry/virtualenvs/vbi-nIDHWX03-py3.10/lib/python3.10/site-packages/tqdm/std.py", line 1170, in __iter__
    for obj in iterable:

  File "/Users/danielarmyr/Library/Caches/pypoetry/virtualenvs/vbi-nIDHWX03-py3.10/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 435, in __iter__
    return self._get_iterator()

  File "/Users/danielarmyr/Library/Caches/pypoetry/virtualenvs/vbi-nIDHWX03-py3.10/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 381, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)

  File "/Users/danielarmyr/Library/Caches/pypoetry/virtualenvs/vbi-nIDHWX03-py3.10/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1034, in __init__
    w.start()

  File "/Users/danielarmyr/.pyenv/versions/3.10.13/lib/python3.10/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)

  File "/Users/danielarmyr/.pyenv/versions/3.10.13/lib/python3.10/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)

  File "/Users/danielarmyr/.pyenv/versions/3.10.13/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)

  File "/Users/danielarmyr/.pyenv/versions/3.10.13/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)

  File "/Users/danielarmyr/.pyenv/versions/3.10.13/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)

  File "/Users/danielarmyr/.pyenv/versions/3.10.13/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)

  File "/Users/danielarmyr/.pyenv/versions/3.10.13/lib/python3.10/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)

_pickle.PicklingError: Can't pickle <class 'nemo.collections.common.parts.preprocessing.collections.SpeechLabelEntity'>: attribute lookup SpeechLabelEntity on nemo.collections.common.parts.preprocessing.collections failed

MahmoudAshraf97 commented 10 months ago

Hello, how can I reproduce this? can you upload the audio file and list your versions of requirements?

AlbinGyllander commented 10 months ago

I have the same issue. First I get the error as mentioned above: Can't pickle <class 'nemo.collections.common.parts.preprocessing.collections.SpeechLabelEntity'>: attribute lookup SpeechLabelEntity on nemo.collections.common.parts.preprocessing.collections failed

And then the program continues and returns the error

RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that `you` are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

I am not sure if this is because I have implemented it incorrectly or if this is a bug but i hope it can help with debugging.

jiayihan2020 commented 10 months ago

I also have the same issue as @AlbinGyllander . I am using Python3.10.11, and the same issue also appeared with python 3.8,. No The packages that I have installed are as follows:

Package                     Version
--------------------------- ------------
absl-py                     2.0.0
aiohttp                     3.8.5
aiosignal                   1.3.1
alembic                     1.12.0
ansicon                     1.89.0
antlr4-python3-runtime      4.9.3
anyio                       3.7.1
appdirs                     1.4.4
arrow                       1.2.3
asteroid-filterbanks        0.4.0
asttokens                   2.4.0
async-timeout               4.0.3
attrs                       23.1.0
audioread                   3.0.0
av                          10.0.0
backcall                    0.2.0
backoff                     2.2.1
beautifulsoup4              4.12.2
blessed                     1.20.0
braceexpand                 0.1.7
cachetools                  5.3.1
certifi                     2023.7.22
cffi                        1.15.1
charset-normalizer          3.2.0
click                       8.1.7
cloudpickle                 2.2.1
cmaes                       0.10.0
colorama                    0.4.6
coloredlogs                 15.0.1
colorlog                    6.7.0
comm                        0.1.4
contourpy                   1.1.1
croniter                    1.4.1
ctranslate2                 3.20.0
cycler                      0.11.0
Cython                      3.0.2
dateutils                   0.6.12
decorator                   5.1.1
deepdiff                    6.5.0
deepmultilingualpunctuation 1.0.1
demucs                      4.1.0a1
Distance                    0.1.3
docker-pycreds              0.4.0
docopt                      0.6.2
dora-search                 0.1.12
editdistance                0.6.2
einops                      0.6.1
exceptiongroup              1.1.3
executing                   1.2.0
fastapi                     0.103.1
faster-whisper              0.7.1
ffmpeg-python               0.2.0
filelock                    3.12.4
flatbuffers                 23.5.26
fonttools                   4.42.1
frozenlist                  1.4.0
fsspec                      2023.9.2
future                      0.18.3
g2p-en                      2.1.0
gitdb                       4.0.10
GitPython                   3.1.37
google-auth                 2.23.0
google-auth-oauthlib        1.0.0
greenlet                    2.0.2
grpcio                      1.58.0
h11                         0.14.0
huggingface-hub             0.17.3
humanfriendly               10.0
hydra-core                  1.2.0
HyperPyYAML                 1.2.2
idna                        3.4
inflect                     7.0.0
inquirer                    3.1.3
ipython                     8.15.0
ipywidgets                  8.1.1
itsdangerous                2.1.2
jedi                        0.19.0
Jinja2                      3.1.2
jinxed                      1.2.0
jiwer                       3.0.3
joblib                      1.3.2
julius                      0.2.7
jupyterlab-widgets          3.0.9
kaldi-python-io             1.2.2
kaldiio                     2.18.0
kiwisolver                  1.4.5
lameenc                     1.6.1
lazy_loader                 0.3
Levenshtein                 0.22.0
librosa                     0.10.1
lightning                   2.0.9
lightning-cloud             0.5.38
lightning-utilities         0.9.0
llvmlite                    0.41.0
loguru                      0.7.2
Mako                        1.2.4
Markdown                    3.4.4
markdown-it-py              3.0.0
MarkupSafe                  2.1.3
marshmallow                 3.20.1
matplotlib                  3.8.0
matplotlib-inline           0.1.6
mdurl                       0.1.2
more-itertools              10.1.0
mpmath                      1.3.0
msgpack                     1.0.6
multidict                   6.0.4
nemo-toolkit                1.20.0
networkx                    3.1
nltk                        3.8.1
numba                       0.58.0
numpy                       1.23.5
oauthlib                    3.2.2
omegaconf                   2.2.3
onnx                        1.14.1
onnxruntime                 1.16.0
openunmix                   1.2.1
optuna                      3.3.0
ordered-set                 4.1.0
packaging                   23.1
pandas                      2.1.1
parso                       0.8.3
pathtools                   0.1.2
pickleshare                 0.7.5
Pillow                      10.0.1
pip                         23.2.1
plac                        1.4.0
platformdirs                3.10.0
pooch                       1.7.0
primePy                     1.3
prompt-toolkit              3.0.39
protobuf                    4.24.3
psutil                      5.9.5
pure-eval                   0.2.2
pyannote.audio              3.0.0
pyannote.core               5.0.0
pyannote.database           5.0.1
pyannote.metrics            3.2.1
pyannote.pipeline           3.0.1
pyasn1                      0.5.0
pyasn1-modules              0.3.0
pybind11                    2.11.1
pycparser                   2.21
pydantic                    1.10.12
pydub                       0.25.1
Pygments                    2.16.1
PyJWT                       2.8.0
pyparsing                   3.1.1
pyreadline3                 3.4.1
python-dateutil             2.8.2
python-editor               1.0.4
python-multipart            0.0.6
pytorch-lightning           1.9.4
pytorch-metric-learning     2.3.0
pytz                        2023.3.post1
PyYAML                      6.0.1
rapidfuzz                   3.3.1
readchar                    4.0.5
regex                       2023.8.8
requests                    2.31.0
requests-oauthlib           1.3.1
retrying                    1.3.4
rich                        13.5.3
rsa                         4.9
ruamel.yaml                 0.17.32
ruamel.yaml.clib            0.2.7
sacremoses                  0.0.53
safetensors                 0.3.3
scikit-learn                1.3.1
scipy                       1.11.2
semver                      3.0.1
sentencepiece               0.1.99
sentry-sdk                  1.31.0
setproctitle                1.3.2
setuptools                  65.5.1
shellingham                 1.5.3
six                         1.16.0
smmap                       5.0.1
sniffio                     1.3.0
sortedcontainers            2.4.0
soundfile                   0.12.1
soupsieve                   2.5
sox                         1.4.1
soxr                        0.3.6
speechbrain                 0.5.15
SQLAlchemy                  2.0.21
stack-data                  0.6.2
starlette                   0.27.0
starsessions                1.3.0
submitit                    1.4.6
sympy                       1.12
tabulate                    0.9.0
tensorboard                 2.14.0
tensorboard-data-server     0.7.1
tensorboardX                2.6.2.2
termcolor                   2.3.0
text-unidecode              1.3
texterrors                  0.4.4
threadpoolctl               3.2.0
tokenizers                  0.13.3
torch                       2.0.1+cu118
torch-audiomentations       0.11.0
torch-pitch-shift           1.2.4
torchaudio                  2.0.2+cu118
torchmetrics                1.2.0
torchvision                 0.15.2+cu118
tqdm                        4.66.1
traitlets                   5.10.1
transformers                4.33.2
treetable                   0.2.5
typer                       0.9.0
typing_extensions           4.8.0
tzdata                      2023.3
urllib3                     1.26.16
uvicorn                     0.23.2
wandb                       0.15.11
wcwidth                     0.2.6
webdataset                  0.1.62
websocket-client            1.6.3
websockets                  11.0.3
Werkzeug                    2.3.7
wget                        3.2
wheel                       0.41.2
whisperx                    1.0
widgetsnbextension          4.0.9
win32-setctime              1.1.0
wrapt                       1.15.0
yarl                        1.9.2
youtokentome                1.0.6

As far as I know, none of audio files work. I am not sure if there is any incompatibility with the set up. Hopefully someone can help in troubleshooting this.

asdfmonster261 commented 10 months ago

I am also getting this pickling error.

Environment

``` Windows 11 Miniconda, Python 3.9.18 Cuda 11.7 # Also tested with 11.8 ```

Pip list
``` Package Version --------------------------- ------------ absl-py 2.0.0 aiohttp 3.8.5 aiosignal 1.3.1 alembic 1.12.0 ansicon 1.89.0 antlr4-python3-runtime 4.9.3 anyio 3.7.1 appdirs 1.4.4 arrow 1.3.0 asteroid-filterbanks 0.4.0 asttokens 2.4.0 async-timeout 4.0.3 attrs 23.1.0 audioread 3.0.1 av 10.0.0 backcall 0.2.0 backoff 2.2.1 beautifulsoup4 4.12.2 blessed 1.20.0 braceexpand 0.1.7 cachetools 5.3.1 certifi 2022.12.7 cffi 1.16.0 charset-normalizer 2.1.1 click 8.1.7 cloudpickle 2.2.1 cmaes 0.10.0 colorama 0.4.6 coloredlogs 15.0.1 colorlog 6.7.0 comm 0.1.4 contourpy 1.1.1 croniter 1.4.1 ctranslate2 3.20.0 cycler 0.12.0 Cython 3.0.2 dateutils 0.6.12 decorator 5.1.1 deepdiff 6.5.0 deepmultilingualpunctuation 1.0.1 demucs 4.1.0a1 Distance 0.1.3 docker-pycreds 0.4.0 docopt 0.6.2 dora-search 0.1.12 editdistance 0.6.2 einops 0.7.0 exceptiongroup 1.1.3 executing 2.0.0 fastapi 0.103.2 faster-whisper 0.7.1 ffmpeg-python 0.2.0 filelock 3.9.0 flatbuffers 23.5.26 fonttools 4.43.0 frozenlist 1.4.0 fsspec 2023.9.2 future 0.18.3 g2p-en 2.1.0 gitdb 4.0.10 GitPython 3.1.37 google-auth 2.23.2 google-auth-oauthlib 1.0.0 greenlet 2.0.2 grpcio 1.59.0 h11 0.14.0 huggingface-hub 0.17.3 humanfriendly 10.0 hydra-core 1.2.0 HyperPyYAML 1.2.2 idna 3.4 importlib-metadata 6.8.0 importlib-resources 6.1.0 inflect 7.0.0 inquirer 3.1.3 ipython 8.16.0 ipywidgets 8.1.1 itsdangerous 2.1.2 jedi 0.19.0 Jinja2 3.1.2 jinxed 1.2.0 jiwer 3.0.3 joblib 1.3.2 julius 0.2.7 jupyterlab-widgets 3.0.9 kaldi-python-io 1.2.2 kaldiio 2.18.0 kiwisolver 1.4.5 lameenc 1.6.1 lazy_loader 0.3 Levenshtein 0.22.0 librosa 0.10.1 lightning 2.0.9.post0 lightning-cloud 0.5.39 lightning-utilities 0.9.0 llvmlite 0.41.0 loguru 0.7.2 Mako 1.2.4 Markdown 3.4.4 markdown-it-py 3.0.0 MarkupSafe 2.1.2 marshmallow 3.20.1 matplotlib 3.8.0 matplotlib-inline 0.1.6 mdurl 0.1.2 more-itertools 10.1.0 mpmath 1.3.0 msgpack 1.0.7 multidict 6.0.4 nemo-toolkit 1.20.0 networkx 3.0 nltk 3.8.1 numba 0.58.0 numpy 1.23.5 oauthlib 3.2.2 omegaconf 2.2.3 onnx 1.14.1 onnxruntime 1.16.0 onnxruntime-gpu 1.16.0 openunmix 1.2.1 optuna 3.3.0 ordered-set 4.1.0 packaging 23.2 pandas 2.1.1 parso 0.8.3 pathtools 0.1.2 pickleshare 0.7.5 Pillow 9.3.0 pip 23.2.1 plac 1.4.0 platformdirs 3.10.0 pooch 1.7.0 primePy 1.3 prompt-toolkit 3.0.39 protobuf 4.24.3 psutil 5.9.5 pure-eval 0.2.2 pyannote.audio 3.0.1 pyannote.core 5.0.0 pyannote.database 5.0.1 pyannote.metrics 3.2.1 pyannote.pipeline 3.0.1 pyasn1 0.5.0 pyasn1-modules 0.3.0 pybind11 2.11.1 pycparser 2.21 pydantic 1.10.13 pydub 0.25.1 Pygments 2.16.1 PyJWT 2.8.0 pyparsing 3.1.1 pyreadline3 3.4.1 python-dateutil 2.8.2 python-editor 1.0.4 python-multipart 0.0.6 pytorch-lightning 1.9.4 pytorch-metric-learning 2.3.0 pytz 2023.3.post1 PyYAML 6.0.1 rapidfuzz 3.3.1 readchar 4.0.5 regex 2023.8.8 requests 2.28.1 requests-oauthlib 1.3.1 retrying 1.3.4 rich 13.6.0 rsa 4.9 ruamel.yaml 0.17.33 ruamel.yaml.clib 0.2.7 sacremoses 0.0.53 safetensors 0.3.3 scikit-learn 1.3.1 scipy 1.11.3 semver 3.0.1 sentencepiece 0.1.99 sentry-sdk 1.31.0 setproctitle 1.3.2 setuptools 65.5.1 shellingham 1.5.3 six 1.16.0 smmap 5.0.1 sniffio 1.3.0 sortedcontainers 2.4.0 soundfile 0.12.1 soupsieve 2.5 sox 1.4.1 soxr 0.3.6 speechbrain 0.5.15 SQLAlchemy 2.0.21 stack-data 0.6.3 starlette 0.27.0 starsessions 1.3.0 submitit 1.4.6 sympy 1.12 tabulate 0.9.0 tensorboard 2.14.1 tensorboard-data-server 0.7.1 tensorboardX 2.6.2.2 termcolor 2.3.0 text-unidecode 1.3 texterrors 0.4.4 threadpoolctl 3.2.0 tokenizers 0.13.3 torch 2.0.1+cu117 torch-audiomentations 0.11.0 torch-pitch-shift 1.2.4 torchaudio 2.0.2+cu117 torchmetrics 1.2.0 torchvision 0.15.2+cu117 tqdm 4.66.1 traitlets 5.10.1 transformers 4.33.3 treetable 0.2.5 typer 0.9.0 types-python-dateutil 2.8.19.14 typing_extensions 4.8.0 tzdata 2023.3 urllib3 1.26.13 uvicorn 0.23.2 wandb 0.15.11 wcwidth 0.2.8 webdataset 0.1.62 websocket-client 1.6.3 websockets 11.0.3 Werkzeug 3.0.0 wget 3.2 wheel 0.41.2 whisperx 1.0 widgetsnbextension 4.0.9 win32-setctime 1.1.0 wrapt 1.15.0 yarl 1.9.2 youtokentome 1.0.6 zipp 3.17.0 ```

Steps to reproduce

``` git clone https://github.com/MahmoudAshraf97/whisper-diarization.git cd .\whisper-diarization\ conda create -n whisper_diarization python=3.9 # Also tested with 3.10 conda activate whisper_diarization pip install cython pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu117 pip install -r requirements.txt python .\diarize.py --whisper-model large-v2 --device cuda -a .\test\1315.mp3 ```

The error

``` Traceback (most recent call last): File "C:\Users\username\Desktop\whisper-diarization\diarize.py", line 112, in msdd_model.diarize() File "C:\Users\username\miniconda3\envs\whisper_diarization\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "C:\Users\username\miniconda3\envs\whisper_diarization\lib\site-packages\nemo\collections\asr\models\msdd_models.py", line 1180, in diarize self.clustering_embedding.prepare_cluster_embs_infer() File "C:\Users\username\miniconda3\envs\whisper_diarization\lib\site-packages\nemo\collections\asr\models\msdd_models.py", line 699, in prepare_cluster_embs_infer self.emb_sess_test_dict, self.emb_seq_test, self.clus_test_label_dict, _ = self.run_clustering_diarizer( File "C:\Users\username\miniconda3\envs\whisper_diarization\lib\site-packages\nemo\collections\asr\models\msdd_models.py", line 866, in run_clustering_diarizer scores = self.clus_diar_model.diarize(batch_size=self.cfg_diar_infer.batch_size) File "C:\Users\username\miniconda3\envs\whisper_diarization\lib\site-packages\nemo\collections\asr\models\clustering_diarizer.py", line 437, in diarize self._perform_speech_activity_detection() File "C:\Users\username\miniconda3\envs\whisper_diarization\lib\site-packages\nemo\collections\asr\models\clustering_diarizer.py", line 325, in _perform_speech_activity_detection self._run_vad(manifest_vad_input) File "C:\Users\username\miniconda3\envs\whisper_diarization\lib\site-packages\nemo\collections\asr\models\clustering_diarizer.py", line 218, in _run_vad for i, test_batch in enumerate( File "C:\Users\username\miniconda3\envs\whisper_diarization\lib\site-packages\tqdm\std.py", line 1182, in __iter__ for obj in iterable: File "C:\Users\username\miniconda3\envs\whisper_diarization\lib\site-packages\torch\utils\data\dataloader.py", line 441, in __iter__ return self._get_iterator() File "C:\Users\username\miniconda3\envs\whisper_diarization\lib\site-packages\torch\utils\data\dataloader.py", line 388, in _get_iterator return _MultiProcessingDataLoaderIter(self) File "C:\Users\username\miniconda3\envs\whisper_diarization\lib\site-packages\torch\utils\data\dataloader.py", line 1042, in __init__ w.start() File "C:\Users\username\miniconda3\envs\whisper_diarization\lib\multiprocessing\process.py", line 121, in start self._popen = self._Popen(self) File "C:\Users\username\miniconda3\envs\whisper_diarization\lib\multiprocessing\context.py", line 224, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "C:\Users\username\miniconda3\envs\whisper_diarization\lib\multiprocessing\context.py", line 327, in _Popen return Popen(process_obj) File "C:\Users\username\miniconda3\envs\whisper_diarization\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__ reduction.dump(process_obj, to_child) File "C:\Users\username\miniconda3\envs\whisper_diarization\lib\multiprocessing\reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) _pickle.PicklingError: Can't pickle : attribute lookup SpeechLabelEntity on nemo.collections.common.parts.preprocessing.collections failed ```

MahmoudAshraf97 commented 9 months ago

Hi @asdfmonster261 , can you upload .\test\1315.mp3 for me to reproduce?

asdfmonster261 commented 9 months ago

Sure

https://pixeldrain.com/u/3XS9c9Ej

mayur-neuralgarage commented 9 months ago

`# Initialize NeMo MSDD diarization model
msdd_model = NeuralDiarizer(cfg=create_config(temp_path)).to("cuda")
msdd_model.diarize()

del msdd_model
torch.cuda.empty_cache()`

`[NeMo I 2023-10-05 16:30:20 msdd_models:1092] Loading pretrained diar_msdd_telephonic model from NGC
[NeMo I 2023-10-05 16:30:20 cloud:58] Found existing object C:\Users\mayur\.cache\torch\NeMo\NeMo_1.20.0\diar_msdd_telephonic\3c3697a0a46f945574fa407149975a13\diar_msdd_telephonic.nemo.
[NeMo I 2023-10-05 16:30:20 cloud:64] Re-using file from: C:\Users\mayur\.cache\torch\NeMo\NeMo_1.20.0\diar_msdd_telephonic\3c3697a0a46f945574fa407149975a13\diar_msdd_telephonic.nemo
[NeMo I 2023-10-05 16:30:20 common:913] Instantiating model from pre-trained checkpoint
[NeMo W 2023-10-05 16:30:21 modelPT:161] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
    Train config : 
    manifest_filepath: null
    emb_dir: null
    sample_rate: 16000
    num_spks: 2
    soft_label_thres: 0.5
    labels: null
    batch_size: 15
    emb_batch_size: 0
    shuffle: true

[NeMo W 2023-10-05 16:30:21 modelPT:168] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s). 
    Validation config : 
    manifest_filepath: null
    emb_dir: null
    sample_rate: 16000
    num_spks: 2
    soft_label_thres: 0.5
    labels: null
    batch_size: 15
    emb_batch_size: 0
    shuffle: false

[NeMo W 2023-10-05 16:30:21 modelPT:174] Please call the ModelPT.setup_test_data() or ModelPT.setup_multiple_test_data() method and provide a valid configuration file to setup the test data loader(s).
    Test config : 
    manifest_filepath: null
    emb_dir: null
    sample_rate: 16000
    num_spks: 2
    soft_label_thres: 0.5
    labels: null
    batch_size: 15
    emb_batch_size: 0
    shuffle: false
    seq_eval_mode: false

[NeMo I 2023-10-05 16:30:21 features:289] PADDING: 16
[NeMo I 2023-10-05 16:30:21 features:289] PADDING: 16
[NeMo I 2023-10-05 16:30:22 audio_preprocessing:517] Numba CUDA SpecAugment kernel is being used
[NeMo I 2023-10-05 16:30:22 save_restore_connector:249] Model EncDecDiarLabelModel was successfully restored from C:\Users\mayur\.cache\torch\NeMo\NeMo_1.20.0\diar_msdd_telephonic\3c3697a0a46f945574fa407149975a13\diar_msdd_telephonic.nemo.
[NeMo I 2023-10-05 16:30:22 features:289] PADDING: 16
[NeMo I 2023-10-05 16:30:22 audio_preprocessing:517] Numba CUDA SpecAugment kernel is being used
[NeMo I 2023-10-05 16:30:22 clustering_diarizer:127] Loading pretrained vad_multilingual_marblenet model from NGC
[NeMo I 2023-10-05 16:30:22 cloud:58] Found existing object C:\Users\mayur\.cache\torch\NeMo\NeMo_1.20.0\vad_multilingual_marblenet\670f425c7f186060b7a7268ba6dfacb2\vad_multilingual_marblenet.nemo.
[NeMo I 2023-10-05 16:30:22 cloud:64] Re-using file from: C:\Users\mayur\.cache\torch\NeMo\NeMo_1.20.0\vad_multilingual_marblenet\670f425c7f186060b7a7268ba6dfacb2\vad_multilingual_marblenet.nemo
[NeMo I 2023-10-05 16:30:22 common:913] Instantiating model from pre-trained checkpoint
[NeMo W 2023-10-05 16:30:22 modelPT:161] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
    Train config : 
    manifest_filepath: /manifests/ami_train_0.63.json,/manifests/freesound_background_train.json,/manifests/freesound_laughter_train.json,/manifests/fisher_2004_background.json,/manifests/fisher_2004_speech_sampled.json,/manifests/google_train_manifest.json,/manifests/icsi_all_0.63.json,/manifests/musan_freesound_train.json,/manifests/musan_music_train.json,/manifests/musan_soundbible_train.json,/manifests/mandarin_train_sample.json,/manifests/german_train_sample.json,/manifests/spanish_train_sample.json,/manifests/french_train_sample.json,/manifests/russian_train_sample.json
    sample_rate: 16000
    labels:
    - background
    - speech
    batch_size: 256
    shuffle: true
    is_tarred: false
    tarred_audio_filepaths: null
    tarred_shard_strategy: scatter
    augmentor:
      shift:
        prob: 0.5
        min_shift_ms: -10.0
        max_shift_ms: 10.0
      white_noise:
        prob: 0.5
        min_level: -90
        max_level: -46
        norm: true
      noise:
        prob: 0.5
        manifest_path: /manifests/noise_0_1_musan_fs.json
        min_snr_db: 0
        max_snr_db: 30
        max_gain_db: 300.0
        norm: true
      gain:
        prob: 0.5
        min_gain_dbfs: -10.0
        max_gain_dbfs: 10.0
        norm: true
    num_workers: 16
    pin_memory: true

[NeMo W 2023-10-05 16:30:22 modelPT:168] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s). 
    Validation config : 
    manifest_filepath: /manifests/ami_dev_0.63.json,/manifests/freesound_background_dev.json,/manifests/freesound_laughter_dev.json,/manifests/ch120_moved_0.63.json,/manifests/fisher_2005_500_speech_sampled.json,/manifests/google_dev_manifest.json,/manifests/musan_music_dev.json,/manifests/mandarin_dev.json,/manifests/german_dev.json,/manifests/spanish_dev.json,/manifests/french_dev.json,/manifests/russian_dev.json
    sample_rate: 16000
    labels:
    - background
    - speech
    batch_size: 256
    shuffle: false
    val_loss_idx: 0
    num_workers: 16
    pin_memory: true

[NeMo W 2023-10-05 16:30:22 modelPT:174] Please call the ModelPT.setup_test_data() or ModelPT.setup_multiple_test_data() method and provide a valid configuration file to setup the test data loader(s).
    Test config : 
    manifest_filepath: null
    sample_rate: 16000
    labels:
    - background
    - speech
    batch_size: 128
    shuffle: false
    test_loss_idx: 0

[NeMo I 2023-10-05 16:30:22 features:289] PADDING: 16
[NeMo I 2023-10-05 16:30:23 save_restore_connector:249] Model EncDecClassificationModel was successfully restored from C:\Users\mayur\.cache\torch\NeMo\NeMo_1.20.0\vad_multilingual_marblenet\670f425c7f186060b7a7268ba6dfacb2\vad_multilingual_marblenet.nemo.
[NeMo I 2023-10-05 16:30:23 msdd_models:864] Multiscale Weights: [1, 1, 1, 1, 1]
[NeMo I 2023-10-05 16:30:23 msdd_models:865] Clustering Parameters: {
        "oracle_num_speakers": false,
        "max_num_speakers": 8,
        "enhanced_count_thres": 80,
        "max_rp_threshold": 0.25,
        "sparse_search_volume": 30,
        "maj_vote_spk_count": false
    }
[NeMo I 2023-10-05 16:30:23 speaker_utils:93] Number of files to diarize: 1
[NeMo I 2023-10-05 16:30:23 clustering_diarizer:309] Split long audio file to avoid CUDA memory issue
splitting manifest: 100%|██████████| 1/1 [00:02<00:00,  2.18s/it]
[NeMo I 2023-10-05 16:30:25 classification_models:272] Perform streaming frame-level VAD
[NeMo I 2023-10-05 16:30:25 collections:301] Filtered duration for loading collection is  0.00 hours.
[NeMo I 2023-10-05 16:30:25 collections:302] Dataset loaded with 1 items, total duration of  0.01 hours.
[NeMo I 2023-10-05 16:30:25 collections:304] # 1 files loaded accounting to # 1 labels

vad:   0%|          | 0/1 [00:00<?, ?it/s]
---------------------------------------------------------------------------
PicklingError                             Traceback (most recent call last)
Cell In[15], line 3
      1 # Initialize NeMo MSDD diarization model
      2 msdd_model = NeuralDiarizer(cfg=create_config(temp_path)).to("cuda")
----> 3 msdd_model.diarize()
      5 del msdd_model
      6 torch.cuda.empty_cache()

File ~\AppData\Local\anaconda3\envs\ASR\lib\site-packages\torch\utils\_contextlib.py:115, in context_decorator.<locals>.decorate_context(*args, **kwargs)
    112 @functools.wraps(func)
    113 def decorate_context(*args, **kwargs):
    114     with ctx_factory():
--> 115         return func(*args, **kwargs)

File ~\AppData\Local\anaconda3\envs\ASR\lib\site-packages\nemo\collections\asr\models\msdd_models.py:1180, in NeuralDiarizer.diarize(self)
   1173 @torch.no_grad()
   1174 def diarize(self) -> Optional[List[Optional[List[Tuple[DiarizationErrorRate, Dict]]]]]:
   1175     """
   1176     Launch diarization pipeline which starts from VAD (or a oracle VAD stamp generation), initialization clustering and multiscale diarization decoder (MSDD).
   1177     Note that the result of MSDD can include multiple speakers at the same time. Therefore, RTTM output of MSDD needs to be based on `make_rttm_with_overlap()`
   1178     function that can generate overlapping timestamps. `self.run_overlap_aware_eval()` function performs DER evaluation.
   1179     """
-> 1180     self.clustering_embedding.prepare_cluster_embs_infer()
   1181     self.msdd_model.pairwise_infer = True
   1182     self.get_emb_clus_infer(self.clustering_embedding)

File ~\AppData\Local\anaconda3\envs\ASR\lib\site-packages\nemo\collections\asr\models\msdd_models.py:699, in ClusterEmbedding.prepare_cluster_embs_infer(self)
    695 """
    696 Launch clustering diarizer to prepare embedding vectors and clustering results.
    697 """
    698 self.max_num_speakers = self.cfg_diar_infer.diarizer.clustering.parameters.max_num_speakers
--> 699 self.emb_sess_test_dict, self.emb_seq_test, self.clus_test_label_dict, _ = self.run_clustering_diarizer(
    700     self._cfg_msdd.test_ds.manifest_filepath, self._cfg_msdd.test_ds.emb_dir
    701 )

File ~\AppData\Local\anaconda3\envs\ASR\lib\site-packages\nemo\collections\asr\models\msdd_models.py:866, in ClusterEmbedding.run_clustering_diarizer(self, manifest_filepath, emb_dir)
    864 logging.info(f"Multiscale Weights: {self.clus_diar_model.multiscale_args_dict['multiscale_weights']}")
    865 logging.info(f"Clustering Parameters: {clustering_params_str}")
--> 866 scores = self.clus_diar_model.diarize(batch_size=self.cfg_diar_infer.batch_size)
    868 # If RTTM (ground-truth diarization annotation) files do not exist, scores is None.
    869 if scores is not None:

File ~\AppData\Local\anaconda3\envs\ASR\lib\site-packages\nemo\collections\asr\models\clustering_diarizer.py:437, in ClusteringDiarizer.diarize(self, paths2audio_files, batch_size)
    434 os.makedirs(out_rttm_dir, exist_ok=True)
    436 # Speech Activity Detection
--> 437 self._perform_speech_activity_detection()
    439 # Segmentation
    440 scales = self.multiscale_args_dict['scale_dict'].items()

File ~\AppData\Local\anaconda3\envs\ASR\lib\site-packages\nemo\collections\asr\models\clustering_diarizer.py:325, in ClusteringDiarizer._perform_speech_activity_detection(self)
    320         logging.warning(
    321             "If you encounter CUDA memory issue, try splitting manifest entry by split_duration to avoid it."
    322         )
    324     self._setup_vad_test_data(manifest_vad_input)
--> 325     self._run_vad(manifest_vad_input)
    327 elif self._diarizer_params.vad.external_vad_manifest is not None:
    328     self._speaker_manifest_path = self._diarizer_params.vad.external_vad_manifest

File ~\AppData\Local\anaconda3\envs\ASR\lib\site-packages\nemo\collections\asr\models\clustering_diarizer.py:218, in ClusteringDiarizer._run_vad(self, manifest_file)
    215     data.append(get_uniqname_from_filepath(file))
    217 status = get_vad_stream_status(data)
--> 218 for i, test_batch in enumerate(
    219     tqdm(self._vad_model.test_dataloader(), desc='vad', leave=True, disable=not self.verbose)
    220 ):
    221     test_batch = [x.to(self._vad_model.device) for x in test_batch]
    222     with autocast():

File ~\AppData\Local\anaconda3\envs\ASR\lib\site-packages\tqdm\std.py:1182, in tqdm.__iter__(self)
   1179 time = self._time
   1181 try:
-> 1182     for obj in iterable:
   1183         yield obj
   1184         # Update and possibly print the progressbar.
   1185         # Note: does not call self.update(1) for speed optimisation.

File ~\AppData\Local\anaconda3\envs\ASR\lib\site-packages\torch\utils\data\dataloader.py:442, in DataLoader.__iter__(self)
    440     return self._iterator
    441 else:
--> 442     return self._get_iterator()

File ~\AppData\Local\anaconda3\envs\ASR\lib\site-packages\torch\utils\data\dataloader.py:388, in DataLoader._get_iterator(self)
    386 else:
    387     self.check_worker_number_rationality()
--> 388     return _MultiProcessingDataLoaderIter(self)

File ~\AppData\Local\anaconda3\envs\ASR\lib\site-packages\torch\utils\data\dataloader.py:1043, in _MultiProcessingDataLoaderIter.__init__(self, loader)
   1036 w.daemon = True
   1037 # NB: Process.start() actually take some time as it needs to
   1038 #     start a process and pass the arguments over via a pipe.
   1039 #     Therefore, we only add a worker to self._workers list after
   1040 #     it started, so that we do not call .join() if program dies
   1041 #     before it starts, and __del__ tries to join but will get:
   1042 #     AssertionError: can only join a started process.
-> 1043 w.start()
   1044 self._index_queues.append(index_queue)
   1045 self._workers.append(w)

File ~\AppData\Local\anaconda3\envs\ASR\lib\multiprocessing\process.py:121, in BaseProcess.start(self)
    118 assert not _current_process._config.get('daemon'), \
    119        'daemonic processes are not allowed to have children'
    120 _cleanup()
--> 121 self._popen = self._Popen(self)
    122 self._sentinel = self._popen.sentinel
    123 # Avoid a refcycle if the target function holds an indirect
    124 # reference to the process object (see bpo-30775)

File ~\AppData\Local\anaconda3\envs\ASR\lib\multiprocessing\context.py:224, in Process._Popen(process_obj)
    222 @staticmethod
    223 def _Popen(process_obj):
--> 224     return _default_context.get_context().Process._Popen(process_obj)

File ~\AppData\Local\anaconda3\envs\ASR\lib\multiprocessing\context.py:327, in SpawnProcess._Popen(process_obj)
    324 @staticmethod
    325 def _Popen(process_obj):
    326     from .popen_spawn_win32 import Popen
--> 327     return Popen(process_obj)

File ~\AppData\Local\anaconda3\envs\ASR\lib\multiprocessing\popen_spawn_win32.py:93, in Popen.__init__(self, process_obj)
     91 try:
     92     reduction.dump(prep_data, to_child)
---> 93     reduction.dump(process_obj, to_child)
     94 finally:
     95     set_spawning_popen(None)

File ~\AppData\Local\anaconda3\envs\ASR\lib\multiprocessing\reduction.py:60, in dump(obj, file, protocol)
     58 def dump(obj, file, protocol=None):
     59     '''Replacement for pickle.dump() using ForkingPickler.'''
---> 60     ForkingPickler(file, protocol).dump(obj)

PicklingError: Can't pickle <class 'nemo.collections.common.parts.preprocessing.collections.SpeechLabelEntity'>: attribute lookup SpeechLabelEntity on nemo.collections.common.parts.preprocessing.collections failed
`

MahmoudAshraf97 commented 9 months ago

@asdfmonster261

I couldn't reproduce it unfortunately

asdfmonster261 commented 9 months ago

If it's working fine on your end, could you post what your environment is for using it

ie. pip list, os, python version and possibly a sample audio to confirm with

mayur-neuralgarage commented 9 months ago

@MahmoudAshraf97 i am getting that error when i am runing it in my local PC on colab it is working superfine ....

can you please share your local env file all requirements and everything ie. pip list, os, python version

tomonarifeehan commented 9 months ago

HI @MahmoudAshraf97 any updates on this error? I'm having the same issue.

I'm running Python 3.10.12

Package Version

absl-py 2.0.0 aiohttp 3.8.6 aiosignal 1.3.1 alembic 1.12.0 ansicon 1.89.0 antlr4-python3-runtime 4.9.3 anyio 3.7.1 appdirs 1.4.4 arrow 1.3.0 asteroid-filterbanks 0.4.0 asttokens 2.4.0 async-timeout 4.0.3 attrs 23.1.0 audioread 3.0.1 av 10.0.0 backcall 0.2.0 backoff 2.2.1 beautifulsoup4 4.12.2 blessed 1.20.0 braceexpand 0.1.7 cachetools 5.3.1 certifi 2023.7.22 cffi 1.16.0 charset-normalizer 3.3.0 click 8.1.7 cloudpickle 2.2.1 cmaes 0.10.0 colorama 0.4.6 coloredlogs 15.0.1 colorlog 6.7.0 comm 0.1.4 contourpy 1.1.1 croniter 1.4.1 ctranslate2 3.20.0 cycler 0.12.1 dateutils 0.6.12 decorator 5.1.1 deepdiff 6.6.0 deepmultilingualpunctuation 1.0.1 demucs 4.1.0a1 Distance 0.1.3 docker-pycreds 0.4.0 docopt 0.6.2 dora-search 0.1.12 editdistance 0.6.2 einops 0.7.0 exceptiongroup 1.1.3 executing 2.0.0 fastapi 0.103.2 faster-whisper 0.9.0 ffmpeg-python 0.2.0 filelock 3.12.4 flatbuffers 23.5.26 fonttools 4.43.1 frozenlist 1.4.0 fsspec 2023.9.2 future 0.18.3 g2p-en 2.1.0 gitdb 4.0.10 GitPython 3.1.37 google-auth 2.23.3 google-auth-oauthlib 1.0.0 greenlet 3.0.0 grpcio 1.59.0 h11 0.14.0 huggingface-hub 0.17.3 humanfriendly 10.0 hydra-core 1.2.0 HyperPyYAML 1.2.2 idna 3.4 inflect 7.0.0 inquirer 3.1.3 ipython 8.16.1 ipywidgets 8.1.1 itsdangerous 2.1.2 jedi 0.19.1 Jinja2 3.1.2 jinxed 1.2.0 jiwer 3.0.3 joblib 1.3.2 julius 0.2.7 jupyterlab-widgets 3.0.9 kaldi-python-io 1.2.2 kaldiio 2.18.0 kiwisolver 1.4.5 lameenc 1.6.1 lazy_loader 0.3 Levenshtein 0.23.0 librosa 0.10.1 lightning 2.0.9.post0 lightning-cloud 0.5.39 lightning-utilities 0.9.0 llvmlite 0.41.0 loguru 0.7.2 Mako 1.2.4 Markdown 3.5 markdown-it-py 3.0.0 MarkupSafe 2.1.3 marshmallow 3.20.1 matplotlib 3.8.0 matplotlib-inline 0.1.6 mdurl 0.1.2 mpmath 1.3.0 msgpack 1.0.7 multidict 6.0.4 nemo-toolkit 1.20.0 networkx 3.1 nltk 3.8.1 numba 0.58.0 numpy 1.23.5 oauthlib 3.2.2 omegaconf 2.2.3 onnx 1.14.1 onnxruntime 1.16.0 openunmix 1.2.1 optuna 3.3.0 ordered-set 4.1.0 packaging 23.2 pandas 2.1.1 parso 0.8.3 pathtools 0.1.2 pickleshare 0.7.5 Pillow 10.0.1 pip 23.2.1 plac 1.4.0 platformdirs 3.11.0 pooch 1.7.0 primePy 1.3 prompt-toolkit 3.0.39 protobuf 4.24.4 psutil 5.9.5 pure-eval 0.2.2 pyannote.audio 3.0.0 pyannote.core 5.0.0 pyannote.database 5.0.1 pyannote.metrics 3.2.1 pyannote.pipeline 3.0.1 pyasn1 0.5.0 pyasn1-modules 0.3.0 pybind11 2.11.1 pycparser 2.21 pydantic 1.10.13 pydub 0.25.1 Pygments 2.16.1 PyJWT 2.8.0 pyparsing 3.1.1 pyreadline3 3.4.1 python-dateutil 2.8.2 python-editor 1.0.4 python-multipart 0.0.6 pytorch-lightning 1.9.4 pytorch-metric-learning 2.3.0 pytz 2023.3.post1 PyYAML 5.4.1 rapidfuzz 3.4.0 readchar 4.0.5 regex 2023.10.3 requests 2.31.0 requests-oauthlib 1.3.1 retrying 1.3.4 rich 13.6.0 rsa 4.9 ruamel.yaml 0.17.35 ruamel.yaml.clib 0.2.8 sacremoses 0.0.53 safetensors 0.4.0 scikit-learn 1.3.1 scipy 1.11.3 semver 3.0.2 sentencepiece 0.1.99 sentry-sdk 1.31.0 setproctitle 1.3.3 setuptools 65.5.1 shellingham 1.5.3 six 1.16.0 smmap 5.0.1 sniffio 1.3.0 sortedcontainers 2.4.0 soundfile 0.12.1 soupsieve 2.5 sox 1.4.1 soxr 0.3.7 speechbrain 0.5.15 SQLAlchemy 2.0.21 stack-data 0.6.3 starlette 0.27.0 starsessions 1.3.0 submitit 1.5.0 sympy 1.12 tabulate 0.9.0 tensorboard 2.14.1 tensorboard-data-server 0.7.1 tensorboardX 2.6.2.2 termcolor 2.3.0 text-unidecode 1.3 texterrors 0.4.4 threadpoolctl 3.2.0 tokenizers 0.14.1 torch 2.1.0 torch-audiomentations 0.11.0 torch-pitch-shift 1.2.4 torchaudio 2.1.0 torchmetrics 1.2.0 torchvision 0.16.0 tqdm 4.66.1 traitlets 5.11.2 transformers 4.34.0 treetable 0.2.5 typer 0.9.0 types-python-dateutil 2.8.19.14 typing_extensions 4.8.0 tzdata 2023.3 urllib3 2.0.6 uvicorn 0.23.2 wandb 0.15.12 wcwidth 0.2.8 webdataset 0.1.62 websocket-client 1.6.4 websockets 11.0.3 Werkzeug 3.0.0 wget 3.2 wheel 0.41.2 whisperx 3.1.1 widgetsnbextension 4.0.9 win32-setctime 1.1.0 wrapt 1.15.0 yarl 1.9.2 youtokentome 1.0.6

manjunath7472 commented 9 months ago

I found a fix by setting num_worker to 0 in dataloader.py. "venv\Lib\site-packages\torch\utils\data\dataloader.py" self.num_workers = 0

MahmoudAshraf97 commented 9 months ago

I implemented @manjunath7472 solution in the latest commit here f740cd1acd80e8e4172348377a1f69903edb5f59

MahmoudAshraf97 / whisper-diarization

PicklingError: Can't pickle <class 'nemo.collections.common.parts.preprocessing.collections.SpeechLabelEntity'>: attribute lookup SpeechLabelEntity on nemo.collections.common.parts.preprocessing.collections failed #89

Initialize NeMo MSDD diarization model