Loss is nan

[YoungSeng]

Hello, thanks for the great work and code, but I seem to be having some problems.

While I was training, the printout didn't seem right: 2022-05-17 16:12:20,277: (loss terms) l1 nan, cont nan, var nan 2022-05-17 16:12:24,159: EP 42 ( 75) | 32m 55s, 1139 samples/s | loss: nan, 2022-05-17 16:12:33,106: (loss terms) l1 nan, cont nan, var nan 2022-05-17 16:12:34,003: EP 42 (150) | 33m 4s, 841 samples/s | loss: nan, And I checked three loss: tensor(nan, device='cuda:1', grad_fn=<MulBackward0>) tensor(nan, device='cuda:1', grad_fn=<MulBackward0>) tensor(nan, device='cuda:1', grad_fn=<MulBackward0>) loss

And the output of the net: tensor([[[-1.1089e-02, 2.0337e+00, 1.5299e+00, ..., -7.2902e-01, -9.6424e-04, 8.3192e-02], [ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan],... [ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan]], [[ 3.3295e-02, 3.7979e-01, 4.9924e+00, ..., -3.4652e+00, -1.3517e-03, -6.6605e-02], [ nan, nan, nan, ..., nan, nan, nan],...

Is there something wrong with the code?

[youngwoo-yoon]

Hello, Could you tell me more details like your modifications and dataset used?

[YoungSeng]

Thanks for your reply, I followed the steps exactly, and I seem to have found the problem, the zip file I downloaded from fasttext has a corrupted .bin in it, downloading it several times didn't work, using your cache file it works fine. Btw, can you also provide cache in trimodal? Or is the cache the same?

Thanks！

[youngwoo-yoon]

Good to hear that the cache works for you. Trimodal repo uses a different dataset, so the vocab cache should be different. Regarding the request of the vocab cache for trimodal, could you make a new issue on trimodal repo?

[YoungSeng]

Of course, I just addeda a new issue on trimodal repo, thanks a lot!