strange unstable wav2vec2 pretraining

[Mohammadtvk]

❓ Questions and Help

I am trying to pretrain wav2vec2 on persian language using common voice dataset. I did not modify anything but the dataset path in configs. here is plots of training metrics

as can be seen in above figures, after few epochs grad norm and accuracy drops, code perpelexity spike, and loss increases and then stays almost flat. why is that ?

What's your environment?

• fairseq Version (e.g., 1.0 or main): 0.12.2
• PyTorch Version (e.g., 1.0): 1.13.1
• OS (e.g., Linux): ubuntu 20.04
• How you installed fairseq (pip, source): pip
• Build command you used (if compiling from source):
• Python version: 3.8.10
• CUDA/cuDNN version: 12.0
• GPU models and configuration: Tesle T4 x2
• Any other relevant information:

[tarudesu]

@Mohammadtvk Hi there, I have the same problem with you. Could I know that how can you fix that one, please?

[StudentofSong]

Hi there, I have the same problem with you. Have you solved your problem.Could I know that how can you fix that one, please?