Model loss/total loss of 0.01

[seovchinnikov]

Hey! I'm trying to fugure out why model with pretrained vgg weights always demonstrates starting training model loss of 0.01 and total loss (as I understood its just an addition of regularization loss) of 0.04 on my own dataset. Further, model loss remains the same and total loss converges to model loss of 0.01. But after the training the model cant predict anything as I can see on TBoard. Also I've tried to train only on 2 images from samples folder. Losses are close to those values but model can predict with good accuracy. So why model can't cross 0.01 point? As I can see from code, it seem like actual loss is 1 and that means that intersection over union is 0.

And why score_map_pred sometimes looks like ? Is it because of nans?

[argman]

i dont know, maybe your input and output cannot make the network converge

[LouisRuan]

Your initial loss is close to 0.01? I think the loss function has any questions.

[IngleJaya95]

@seovchinnikov Facing Same issues, any solution/suggestion?

[JackeyDeng]

I met the same issue too, does anyone knows the reason?

[PGrothaus]

This seems to be caused when no valid training examples are in the dataset. It occured to me when using '###' as text label for all training examples. Those examples are ignored in the loss. (Same for '*'):

In icdar.py:

if label == '*' or label == '###':
                text_tags.append(True)
            else:
                text_tags.append(False)

and later on those examples are masked out:

if tag:
     cv2.fillPoly(training_mask, poly.astype(np.int32)[np.newaxis, :, :], 0)

I'd suggest to check the training data set.

[jiangmiaotong]

@

This seems to be caused when no valid training examples are in the dataset. It occured to me when using '###' as text label for all training examples. Those examples are ignored in the loss. (Same for '*'):

In icdar.py:

if label == '*' or label == '###':
                text_tags.append(True)
            else:
                text_tags.append(False)

and later on those examples are masked out:

if tag:
     cv2.fillPoly(training_mask, poly.astype(np.int32)[np.newaxis, :, :], 0)

I'd suggest to check the training data set.

But when I trained on one image and the same at each time , sometimes it appears to be 0.01,sometimes it works.

[peter-peng-w]

Any update or solution of this problem? I am also encountering with the problem of model loss fixing on 0.01.

[sdzbft]

Any update or solution of this problem? I am also encountering with the problem of model loss fixing on 0.01.

text_tags.append(False) set False