ValueError: Input contains NaN, infinity or a value too large for dtype('float32')

[fourk-ty]

During training the model using custom dataset, loss will occur NaN. I tried decrease the lr but the error still occurs.

The logging info of error is as follows.

Epoch 50/500
400/400 [==============================] - 0s 272us/step - loss: nan

Epoch 51/500
256/400 [==================>...........] - 

ETA: 0s - loss: nanTraceback (most recent call last):

File ".\demo.py", line 148, in <module>
    train(args)

File ".\demo.py", line 69, in train
    save_dir=args.save_dir, verbose=args.verbose, aug_pretrain=args.aug_pretrain)

File "E:\Comparison_Experiments\DEC-DA\FcDEC.py", line 171, in pretrain
    self.autoencoder.fit(x, x, batch_size=batch_size, epochs=epochs, callbacks=cb, verbose=verbose)

File "C:\Software\Anaconda3\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1639, in fit
    validation_steps=validation_steps)

File "C:\Software\Anaconda3\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py", line 239, in fit_loop
    callbacks.on_epoch_end(epoch, epoch_logs)

 File "C:\Software\Anaconda3\lib\site-packages\tensorflow\python\keras\callbacks.py", line 214, in on_epoch_end
    callback.on_epoch_end(epoch, logs)

  File "E:\Comparison_Experiments\DEC-DA\FcDEC.py", line 162, in on_epoch_end
    y_pred = km.fit_predict(features)

 File "C:\Software\Anaconda3\lib\site-packages\sklearn\cluster\k_means_.py", line 917, in fit_predict
    return self.fit(X).labels_

 File "C:\Software\Anaconda3\lib\site-packages\sklearn\cluster\k_means_.py", line 887, in fit
    X = self._check_fit_data(X)

 File "C:\Software\Anaconda3\lib\site-packages\sklearn\cluster\k_means_.py", line 858, in _check_fit_data
    X = check_array(X, accept_sparse='csr', dtype=[np.float64, np.float32])

 File "C:\Software\Anaconda3\lib\site-packages\sklearn\utils\validation.py", line 453, in check_array
    _assert_all_finite(array)

  File "C:\Software\Anaconda3\lib\site-packages\sklearn\utils\validation.py", line 44, in _assert_all_finite
    " or a value too large for %r." % X.dtype)

ValueError: Input contains NaN, infinity or a value too large for dtype('float32').
Epoch 50/500
400/400 [==============================] - 0s 272us/step - loss: nan

Epoch 51/500
256/400 [==================>...........] - ETA: 0s - loss: nanTraceback (most recent call last):

  File ".\demo.py", line 148, in <module>
    train(args)

  File ".\demo.py", line 69, in train
    save_dir=args.save_dir, verbose=args.verbose, aug_pretrain=args.aug_pretrain)

File "E:\Comparison_Experiments\DEC-DA\FcDEC.py", line 171, in pretrain
    self.autoencoder.fit(x, x, batch_size=batch_size, epochs=epochs, callbacks=cb, verbose=verbose)

File "C:\Software\Anaconda3\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1639, in fit
    validation_steps=validation_steps)

File "C:\Software\Anaconda3\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py", line 239, in fit_loop
    callbacks.on_epoch_end(epoch, epoch_logs)

File "C:\Software\Anaconda3\lib\site-packages\tensorflow\python\keras\callbacks.py", line 214, in on_epoch_end
    callback.on_epoch_end(epoch, logs)

File "E:\Comparison_Experiments\DEC-DA\FcDEC.py", line 162, in on_epoch_end
    y_pred = km.fit_predict(features)

File "C:\Software\Anaconda3\lib\site-packages\sklearn\cluster\k_means_.py", line 917, in fit_predict
    return self.fit(X).labels_

File "C:\Software\Anaconda3\lib\site-packages\sklearn\cluster\k_means_.py", line 887, in fit
    X = self._check_fit_data(X)

File "C:\Software\Anaconda3\lib\site-packages\sklearn\cluster\k_means_.py", line 858, in _check_fit_data
    X = check_array(X, accept_sparse='csr', dtype=[np.float64, np.float32])

File "C:\Software\Anaconda3\lib\site-packages\sklearn\utils\validation.py", line 453, in check_array
    _assert_all_finite(array)

File "C:\Software\Anaconda3\lib\site-packages\sklearn\utils\validation.py", line 44, in _assert_all_finite
    " or a value too large for %r." % X.dtype)

ValueError: Input contains NaN, infinity or a value too large for dtype('float32').

Can anybody know what's wrong of the setting in this code?

[XifengGuo]

@fourk-ty It seems your dataset contains NaN.

[AVCarreiro]

I've also come across this issue, and confirmed my dataset (tested different ones) doesn't have NaN values. Lowering the dimensions of the AE helps but then I lose performance.