search

pyannote / pyannote-audio

Neural building blocks for speaker diarization: speech activity detection, speaker change detection, overlapped speech detection, speaker embedding
http://pyannote.github.io
MIT License
6.31k stars 780 forks source link

Training process is being always killed at the second epoch #419

Closed ooza closed 3 years ago

ooza commented 4 years ago

The bug When I tried to fine-tune pretrained models / train the SAD pipeline from scratch on my own data, the training process is always killed at the second epoch: To Reproduce

$ pyannote-audio sad train --subset=train --to=200 --parallel=4 experiments/2107/models/speech_activity_detection ADVANCE.SpeakerDiarization.advComp01

Content of config.yml

task:
   name: SpeechActivityDetection
   params:
      duration: 2.0
      batch_size: 64
      per_epoch: 1

feature_extraction:
   name: RawAudio
   params:
      sample_rate: 16000

architecture:
   name: pyannote.audio.models.PyanNet
   params:
      rnn:
         unit: LSTM
         hidden_size: 128
         num_layers: 2
         bidirectional: True
      ff:
         hidden_size: [128, 128]

scheduler:
   name: ConstantScheduler
   params:
      learning_rate: 0.01

pyannote environment

$ pip freeze | grep pyannote
pyannote.audio==2.0.a1
pyannote.core==4.1
pyannote.database==4.0.1
pyannote.db.voxceleb==1.0.2
pyannote.metrics==3.0.1
pyannote.pipeline==1.5.2

Additional context Input data's sample rate= 44khz

I'm wondering if it is linked to the memory, batch-size, or something else; knowing that I've in vain set the --parallel option to 1,2..

hbredin commented 4 years ago

Thanks @ooza for your interest in pyannote.audio.

Can you please provide more details on the error message you get when your job is killed?

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.