NaN or Inf found in input tensor.
NaN or Inf found in input tensor.
NaN or Inf found in input tensor.
NaN or Inf found in input tensor.
Epoch 0/16, Iter 380/6774, lr 0.000841, train loss = 329.206, depth loss = 7.302, kl loss = 1578.294, approx_kl = 13.548, time = 4.007
Epoch 0/16, Iter 390/6774, lr 0.000855, train loss = 4807.599, depth loss = 22.143, kl loss = 23964.273, approx_kl = 14.745, time = 4.021
Epoch 0/16, Iter 400/6774, lr 0.000868, train loss = 265.932, depth loss = 6.916, kl loss = 1266.214, approx_kl = 12.690, time = 4.017
Epoch 0/16, Iter 410/6774, lr 0.000881, train loss = 699.619, depth loss = 8.129, kl loss = 3435.879, approx_kl = 12.443, time = 3.994
Epoch 0/16, Iter 420/6774, lr 0.000895, train loss = 1849.536, depth loss = 11.716, kl loss = 9175.318, approx_kl = 14.472, time = 3.993
Epoch 0/16, Iter 430/6774, lr 0.000908, train loss = 283.722, depth loss = 8.125, kl loss = 1353.582, approx_kl = 13.006, time = 4.012
Epoch 0/16, Iter 440/6774, lr 0.000921, train loss = 396.045, depth loss = 5.413, kl loss = 1915.851, approx_kl = 12.875, time = 4.017
Epoch 0/16, Iter 450/6774, lr 0.000935, train loss = 1381.491, depth loss = 12.813, kl loss = 6833.853, approx_kl = 14.720, time = 4.025
Epoch 0/16, Iter 460/6774, lr 0.000948, train loss = 899.482, depth loss = 14.196, kl loss = 4421.479, approx_kl = 15.187, time = 4.023
Epoch 0/16, Iter 470/6774, lr 0.000961, train loss = 1201.927, depth loss = 14.667, kl loss = 5935.046, approx_kl = 14.918, time = 4.002
Epoch 0/16, Iter 480/6774, lr 0.000975, train loss = 666.221, depth loss = 7.502, kl loss = 3266.265, approx_kl = 12.968, time = 4.020
Epoch 0/16, Iter 490/6774, lr 0.000988, train loss = 526.644, depth loss = 8.771, kl loss = 2569.961, approx_kl = 12.651, time = 4.046
Epoch 0/16, Iter 500/6774, lr 0.001000, train loss = 1123.106, depth loss = 13.588, kl loss = 5545.175, approx_kl = 14.071, time = 4.021
nan error occur!!
nan error occur!!
nan error occur!!
nan error occur!!
when I train train_kd.py，I occured this :
NaN or Inf found in input tensor. NaN or Inf found in input tensor. NaN or Inf found in input tensor. NaN or Inf found in input tensor. Epoch 0/16, Iter 380/6774, lr 0.000841, train loss = 329.206, depth loss = 7.302, kl loss = 1578.294, approx_kl = 13.548, time = 4.007 Epoch 0/16, Iter 390/6774, lr 0.000855, train loss = 4807.599, depth loss = 22.143, kl loss = 23964.273, approx_kl = 14.745, time = 4.021 Epoch 0/16, Iter 400/6774, lr 0.000868, train loss = 265.932, depth loss = 6.916, kl loss = 1266.214, approx_kl = 12.690, time = 4.017 Epoch 0/16, Iter 410/6774, lr 0.000881, train loss = 699.619, depth loss = 8.129, kl loss = 3435.879, approx_kl = 12.443, time = 3.994 Epoch 0/16, Iter 420/6774, lr 0.000895, train loss = 1849.536, depth loss = 11.716, kl loss = 9175.318, approx_kl = 14.472, time = 3.993 Epoch 0/16, Iter 430/6774, lr 0.000908, train loss = 283.722, depth loss = 8.125, kl loss = 1353.582, approx_kl = 13.006, time = 4.012 Epoch 0/16, Iter 440/6774, lr 0.000921, train loss = 396.045, depth loss = 5.413, kl loss = 1915.851, approx_kl = 12.875, time = 4.017 Epoch 0/16, Iter 450/6774, lr 0.000935, train loss = 1381.491, depth loss = 12.813, kl loss = 6833.853, approx_kl = 14.720, time = 4.025 Epoch 0/16, Iter 460/6774, lr 0.000948, train loss = 899.482, depth loss = 14.196, kl loss = 4421.479, approx_kl = 15.187, time = 4.023 Epoch 0/16, Iter 470/6774, lr 0.000961, train loss = 1201.927, depth loss = 14.667, kl loss = 5935.046, approx_kl = 14.918, time = 4.002 Epoch 0/16, Iter 480/6774, lr 0.000975, train loss = 666.221, depth loss = 7.502, kl loss = 3266.265, approx_kl = 12.968, time = 4.020 Epoch 0/16, Iter 490/6774, lr 0.000988, train loss = 526.644, depth loss = 8.771, kl loss = 2569.961, approx_kl = 12.651, time = 4.046 Epoch 0/16, Iter 500/6774, lr 0.001000, train loss = 1123.106, depth loss = 13.588, kl loss = 5545.175, approx_kl = 14.071, time = 4.021 nan error occur!! nan error occur!! nan error occur!! nan error occur!!