Closed Snir-Dekel closed 2 years ago
Hey hey, thank you for the issue.
Looks like it may have been solved though?
@beneyal found a fix and setup this notebook: https://colab.research.google.com/drive/1NQBvn3QcQtcBm4xW00zjG66Zs1cMXYpE?usp=sharing
Turns out it's because of the TextVectorizer
turning the sentences with @
and .
into nothing... causing NaN's in the loss.
Some questions from yourself and explanations from @beneyal:
- Why does the loss function return nan? And why only at custom embeddings?
- Why does the loss function keep returning nan for everything after the first nan?
There's one way in which categorical cross entropy will return nan
and that's when we reach a point where we do log(0.0)
, which is part of the categorical cross entropy loss calculation.
If you have nan
in the loss, the optimizer will take the loss and derive it to get the errors so that it can update the weights of the layers.
Since nan
is what is sometimes called "an absorbing element", everything that touches it will become nan
, and so every weight that was affected by this loss is now nan
itself, which means that now every calculation will include that nan
, which finally leads to nan
everywhere.
EDIT: Everything I said above is true, but it is not the root cause of the problem in our case.
I traced the problem to TextVectorizer
that has a "nice" default: standardize='lower_and_strip_punctuation'
, which, as the name suggests, gets rid of punctuation.
Text like "@ ) ." is all punctuation, so it's stripped completely.
The vectorizer gives us a vector of only zeros.
The embeddings are masking zeros (mask_zero=True)
, so the GlobalAveragePooling1D
doesn't "see" any elements, and try to average zero elements of zero value, giving nan
.
So it appears the problem could have been also solved by 1) not stripping punctuation in the TextVectorizer
, or 2) Not masking the embeddings (not recommended).
Anyway, the nan
coming out from the pooling layer, naturally came out as nan
from the loss function, creating the mess mentioned above.
From: video 285, Comparing the performance of all of our modelling experiments Time in video: 3:53 / 9:36 Demo: https://colab.research.google.com/drive/1qrg2y7bTCVZwWS_xxpBQ8i-Y3EMu1wij?usp=sharing#scrollTo=b5147e40-b745-4e1d-9c9c-730e42ee4501
My implementation code:
When fitting the model with the code above, I am getting a
nan
loss.Original (Your code):