Open fairy-of-9 opened 4 years ago
more error case:
emb[[-inf -inf -inf...]...]
emb_shape:[480 480]
offset[[0][480][960]...]
offset_shape:[480 1]
indices[[316 0 1...]...]
indices_shape:[480 50]
indices+offset:[[316 0 1...]...]
indices+offset_shape:[480 50]
flattened_emb[[-inf][-inf][-inf]...]
flattened_emb_shape:[230400 1]
gathered[[[nan][-inf][-inf]]...]
gathered_shape:[480 50 1]
IIRC, I don't think I changed that part of the code from the original e2e-coref. Are you seeing this on English OntoNotes? Possibly related to https://github.com/mandarjoshi90/coref/issues/6 ?
in independent.py
top_fast_antecedent_scores = util.batch_gather(fast_antecedent_scores, top_antecedents) # [k, c]
sometimes return[NaN, NaN ...]
I tried to print the value of tensors using
tf.Print()
batch_gather in util.py
The results are as follows.
Sometimes it doesn't happen(well done), but it happens frequently(training stops because Loss is NaN).
Can you help me..?