Bug when training with DDP

facebookresearch / denoiser

Real Time Speech Enhancement in the Waveform Domain (Interspeech 2020)We provide a PyTorch implementation of the paper Real Time Speech Enhancement in the Waveform Domain. In which, we present a causal speech enhancement model working on the raw waveform that runs in real-time on a laptop CPU. The proposed model is based on an encoder-decoder architecture with skip-connections. It is optimized on both time and frequency domains, using multiple loss functions. Empirical evidence shows that it is capable of removing various kinds of background noise including stationary and non-stationary noises, as well as room reverb. Additionally, we suggest a set of data augmentation techniques applied directly on the raw waveform which further improve model performance and its generalization abilities.

Other

1.69k stars 303 forks source link

Bug when training with DDP #79

Open sagyHarpaz opened 3 years ago

sagyHarpaz commented 3 years ago

When training with DDP it stuck on validation. Any suggestions?

adefossez commented 3 years ago

Do you have the logs ? that would be very useful to help you debug the issue.

adiyoss commented 3 years ago

Hi @sagyHarpaz, It might be related to the python version, are you using python 3.6 or 3.7?