search

HHousen / speaker-change-detection

Speaker change detection using SincNet and an LSTM/Transformer
GNU General Public License v3.0
39 stars 6 forks source link

TypeError: cannot pickle '_thread.RLock' object #1

Closed viterbi546 closed 1 year ago

viterbi546 commented 1 year ago

Hi, thanks for your code. But I got error when run the code. I setup the AMI dataset by AMI-diarization-setup and didn't change any code. Run python train.py and got following error messages:

GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Traceback (most recent call last):
  File "train.py", line 41, in <module>
    trainer.fit(model, datamodule=dm)
  File "/home/trainer/miniconda3/envs/ulypy38/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 770, in fit
    self._call_and_handle_interrupt(
  File "/home/trainer/miniconda3/envs/ulypy38/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 721, in _call_and_handle_interrupt
    return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, **kwargs)
  File "/home/trainer/miniconda3/envs/ulypy38/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/spawn.py", line 78, in launch
    mp.spawn(
  File "/home/trainer/miniconda3/envs/ulypy38/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 240, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/trainer/miniconda3/envs/ulypy38/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 189, in start_processes
    process.start()
  File "/home/trainer/miniconda3/envs/ulypy38/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/home/trainer/miniconda3/envs/ulypy38/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/home/trainer/miniconda3/envs/ulypy38/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/home/trainer/miniconda3/envs/ulypy38/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/home/trainer/miniconda3/envs/ulypy38/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/home/trainer/miniconda3/envs/ulypy38/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread.RLock' object

Please help me to solve this problem. Thank you.

HHousen commented 1 year ago

I suspect the issue is because you are using multiple GPUs. Multiple GPUs should be supported by lightning so it's strange that this is failing. There are a few troubleshooting steps on the PyTorch lightning documentation. So, you could try running the below code to see if that highlights the error:

import pickle
model = SSCDModel()
pickle.dumps(model)

Alternatively, you could try passing the strategy="ddp" argument to the Trainer object.

viterbi546 commented 1 year ago

Thank you, HHousen. Solve this issue. My system has two GPU cards. I set export CUDA_VISIBLE_DEVICES=0 to use first card only. No more error message. Thanks again.

HHousen commented 1 year ago

Glad you fixed it 👍