jvbsoares
commented
1 year ago Hi Leo, Thanks for trying this out. Perhaps this issue could be related to your TF installation? What TF and CUDA versions are you using? I'm able to successfully train with the above command, which is why I'm thinking that it might be worth checking the details of your setup. Does your loss usually go up before you get NaN? Here's what the initial losses look like for me:
Epoch 1/1000 32/32 [==============================] - 31s 967ms/step - loss: 0.5310 Epoch 2/1000 32/32 [==============================] - 25s 776ms/step - loss: 0.2052 Epoch 3/1000 32/32 [==============================] - 25s 769ms/step - loss: 0.1407 Epoch 4/1000 32/32 [==============================] - 24s 757ms/step - loss: 0.1249 Epoch 5/1000 32/32 [==============================] - 24s 764ms/step - loss: 0.1138 Epoch 6/1000 32/32 [==============================] - 26s 797ms/step - loss: 0.1034 Epoch 7/1000 32/32 [==============================] - 24s 757ms/step - loss: 0.0967 Epoch 8/1000 32/32 [==============================] - 24s 765ms/step - loss: 0.0885 Epoch 9/1000 32/32 [==============================] - 24s 752ms/step - loss: 0.0814 Epoch 10/1000 32/32 [==============================] - 178s 6s/step - loss: 0.0776 - val_loss: 0.0773
I'm also not totally sure what's the best way to debug this. As a sanity check, you could verify your TF installation by training some simple model from one of the TF tutorials. You could also try to make the learning rate very low (like 1e-6 or 1e-7), just to see what happens.
lpc-eol
Could you please guide me to debug what is causing the loss Nan? I only modify the
MEMORY_LIMIT_IN_MB = 13 * 1024to leave some memory space for running the validation set and the-lranddwdfor debugging purposes. Before running the commands below, I have followed theFeature pre-processingand run the commands for generating baidu2.0 features.I am trying to reproducing results from your experiments:
Challenge Validated protocolinExperiments using Combination×2 featuresin SoccerNet-action-spotting-challenge-2022.md and facing loss = Nan after 1-4 epochs training after running the command :By running the command on the above, I got loss=Nan in the
second epochs,first epochloss=0.685, and I found after setting thelrto1e-5, the loss=Nan occurs after4-5epochs.(after multiple attemps)Sample output:
Model summary(skip)
I am more familiar with Pytorch and trying my best to understand the code. Thank you for your assistance!