I got a loss of nan when training the model

[Neerajj9]

I'll need some more info. Are you using the exact same code, data and loss function ?

[Tyre20]

Everything is the same except for different dataset

[Tyre20]

My accuracy shoot up to 100% after just one epoch

[jeewenjie]

Check your X.npy and Y.npy files. Make sure the data inside are clean (No '\n' or extra characters etc.) .

[Tyre20]

I have the same error with the same dataset and code with loss of nan

[Randheer91]

I have also face the same ERROR i.e., "val_loss did not improve from inf" and find accuracy and validation loss as "nan". Kindly suggest the proper solution for it. I have used the same code with the same dataset provided in your repository.

[benjastudio]

I think that the code has been written for another version of python that you use (python 3)? I got the same problem. I took a look at the code, and a bit of work have to be done to make it compatible with python 2/3 (some division operators need to be fixed). Can you confirm @Neerajj9 ?

[goldengrisha]

I've got the same issue, any updates?

[goldengrisha]

The problem was in inactivation function 'relu' . It can be fix as: 'activation=tf.nn.leaky_relu' or 'activation=tf.nn.elu'. I've achived the same result after 100 epochs.

[mahirahzainipderas]

can you explain more @goldengrisha ? I go both loss and val_loss nan and val_acc at 1. Seems incorrect for training data.

[goldengrisha]

@mahirahzainipderas it looks like you're facing the problem of daying gradients with ReLu activation function (that what NaN means -- very small activations). If you use model.compile(loss=yolo_loss_func , optimizer=opt , metrics = ['accuracy']) the metric isn't correct in this context instead just orient your focus for decreasing loss.

    inp = Input(input_shape)

    model = MobileNetV2(
        input_tensor=inp, include_top=False, weights='imagenet')
    last_layer = model.output
    conv = Conv2D(512, (3, 3), activation=tf.nn.leaky_relu,
                  padding='same')(last_layer)
    conv = Dropout(0.4)(conv)
    bn = BatchNormalization()(conv)
    lr = LeakyReLU(alpha=0.1)(bn)
    conv = Conv2D(128, (3, 3), activation=tf.nn.leaky_relu, padding='same')(lr)
    conv = Dropout(0.4)(conv)
    bn = BatchNormalization()(conv)
    lr = LeakyReLU(alpha=0.1)(bn)
    conv = Conv2D(5, (3, 3), activation=tf.nn.leaky_relu, padding='same')(lr)
    final = Reshape((grid_h, grid_w, classes, info))(conv)

    model = Model(inp, final)

[noarotman]

Hi, I am facing this problem now. I tried @goldengrisha tip but still got nan after the first step. Any updates?

[goldengrisha]

Hello, @noarotman what version of TF do you use?

[noarotman]

@goldengrisha I use the version that was written in "readme" file: Tensorflow : 1.9.0 and I run this on CPU..

[goldengrisha]

@noarotman try switching to TF 2.1 and check the code above, it should work.

[noarotman]

@goldengrisha thanks I will try. another issue is that when I try to run the yolo model I get this error:

[goldengrisha]

@noarotman, you're welcome, please check the last URL, it can be wrong.

[Neerajj9]

Hello everyone, the code was written in python3 and the exact versions mentioned in the README.md file. There might be some compatibility issues as @goldengrisha pointed out which might be because of a different version of tensorflow or keras. Also yes concentrate on the decreasing loss as against the 'accuracy' metric.

[HOD101s]

@mahirahzainipderas it looks like you're facing the problem of daying gradients with ReLu activation function (that what NaN means -- very small activations). If you use model.compile(loss=yolo_loss_func , optimizer=opt , metrics = ['accuracy']) the metric isn't correct in this context instead just orient your focus for decreasing loss.

    inp = Input(input_shape)

    model = MobileNetV2(
        input_tensor=inp, include_top=False, weights='imagenet')
    last_layer = model.output
    conv = Conv2D(512, (3, 3), activation=tf.nn.leaky_relu,
                  padding='same')(last_layer)
    conv = Dropout(0.4)(conv)
    bn = BatchNormalization()(conv)
    lr = LeakyReLU(alpha=0.1)(bn)
    conv = Conv2D(128, (3, 3), activation=tf.nn.leaky_relu, padding='same')(lr)
    conv = Dropout(0.4)(conv)
    bn = BatchNormalization()(conv)
    lr = LeakyReLU(alpha=0.1)(bn)
    conv = Conv2D(5, (3, 3), activation=tf.nn.leaky_relu, padding='same')(lr)
    final = Reshape((grid_h, grid_w, classes, info))(conv)

    model = Model(inp, final)

Updating the activation functions worked for me. Thanks!