search

SeanNaren / deepspeech.pytorch

Speech Recognition using DeepSpeech2.
MIT License
2.1k stars 620 forks source link

Training is interrupted without error with 3 gpus #657

Closed DatNgoBK closed 3 years ago

DatNgoBK commented 3 years ago

The training is interrupted when reaches 33% of the first epoch. I tried many times and always stopped at 33%. The graphic cards are still used 100% utils by python processes. My config: trainer.accelerator=ddp traner.plugins=null trainer.gradient_clip_val=400 trainer.gpus=3 trainer.amp_level=01

I trained on 3 V100 GPUs

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.