search

pierluigiferrari / ssd_keras

A Keras port of Single Shot MultiBox Detector
Apache License 2.0
1.86k stars 938 forks source link

Urgent!! Invalid Loss #384

Open jc-barreto opened 3 years ago

jc-barreto commented 3 years ago

Tensorflow ==1.15 Keras = 2.2.4

Running on Google collab. I don't have GPU.

I'm trying to run the notebook SSD300_training but I'm getting an error of invalid loss. The error is:

Epoch 1/10

Epoch 00001: LearningRateScheduler setting learning rate to 0.001.
   6/1000 [..............................] - ETA: 9:26:20 - loss: nan    Batch 5: Invalid loss, terminating training

I'm running with adam optimizer. The parameters are:

adam = Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0) batch_size = 8 initial_epoch = 0 final_epoch = 10 steps_per_epoch = 1000

Could someone help me? I really need to run this code I have tried to chance to sgd optimizer but gives the same error...

ManishKakarla commented 2 years ago

did you solved the problem?

jc-barreto commented 2 years ago

No... I give up :/

ManishKakarla commented 2 years ago

I got a fix for it .I increased the batch size and tried to run with few other changes in the dependency now I am able to train .

iagupov commented 1 year ago

@ManishKakarla could you please share waht did change apart from a batch size? thanks