Closed piraka9011 closed 2 years ago
Btw, current workaround is to set loss_list.contrastive.loss.sample_from_same_utterance_only=False
of course, but ideally this works with it set to true.
This issue is stale because it has been open for 60 days with no activity.
This issue was closed because it has been inactive for 7 days since being marked as stale.
Describe the bug
After a few steps when pretraining a
SpeechEncDecSelfSupervisedModel
, training fails with the following errorSteps/Code to reproduce bug
I used the speech_pre_training.py script with the default configuration for the conformer here.
I only modified
train_ds.max_duration: 25.0
. I also tried the config in the Self_Supervised_Pre_Training.ipynb notebook and I got the same error.Expected behavior
Training should process normally and/or a more detailed explanation why the reshape failed and what parameters to change.
Environment overview (please complete the following information)
nvcr.io/nvidia/pytorch:22.05-py3
)pip install 'nemo_toolkit[all]==1.10.0'
docker pull
&docker run
commands used