training loss always `nan`

seongminp / transformers-into-vaes

Code for "Finetuning Pretrained Transformers into Variational Autoencoders"

https://aclanthology.org/2021.insights-1.5

36 stars 11 forks source link

training loss always `nan` #4

Closed fhrzn closed 2 years ago

fhrzn commented 2 years ago

Hello, first of all thanks for your amazing work! I am trying to barely run your code in Google Colab but got the training loss always nan when starting pretraining encoder. Among 15 epochs, it just stop after 1st epoch as the EarlyStopCallback noticed that the loss keep infinite since the beginning. If it possible, could you please check to reproduce my errors?

My environment:

Transformers == 4.23.1
Pytorch Lightning == 1.7.7

Looking forward for your response, thank you)

fhrzn commented 2 years ago

Found the issue, there were a mismatch on assigning variables at denoising input function.

Need to modify run_batch() in the file model_t5.py. Go to line 100 and 101, then change these lines

encoder_inputs[i] = msk
encoder_masks[i] = inp

into

encoder_inputs[i] = inp
encoder_masks[i] = msk

seongminp commented 2 years ago

Oh my.. thank you for finding this bug. Really appreciate it!