search

shinjayne / shinTB

Textboxes : Image Text Detection Model : python package (tensorflow)
https://shinjayne.github.io/deeplearning/2017/07/21/text-boxes-paper-review-1.html
91 stars 35 forks source link

Loss goes to NaN during training #14

Open marshallllllll opened 6 years ago

marshallllllll commented 6 years ago

After a few hundred global steps, the loss goes to NaN. Any ideas as to why?

parksunwoo commented 5 years ago

Hi, I have a same problem...

8 random batch data selected by DataLoader... Done! from graph, get pred_conf, pred_loc ... Done! calculating (positive, negative, true_location) with jaccard overlap by DefaultBoxControl ... calculate_pos_neg_trueloc START matching_dbboxes_gtboxes_in_batch_i START get_top_confidences START get_top_confidences END matching_dbboxes_gtboxes_in_batch_i END prepare_pos_neg_trueloc_in_matches START prepare_pos_neg_trueloc_in_matches END calculate_pos_neg_trueloc END Done! ... GLOBAL STEP : 29003 / LEARNING RATE : 0.001 / LOSS : nan ( 1.9297988414764404 secs)