search

melonwan / denseReg

3D hand pose estimation via dense regression
179 stars 34 forks source link

AssertionError: Model diverged with loss = NaN #21

Open konrad-marzec opened 5 years ago

konrad-marzec commented 5 years ago

python model/hourglass_um_crop_tiny.py --dataset 'msra' --pid 0 --batch_size 3 --num_stack 2 --num_fea 128 --debug_level 2 --is_train False 

finally into the long long training loop
Traceback (most recent call last):
  File "model/hourglass_um_crop_tiny.py", line 932, in <module>
    run_train(dataset, val_dataset)
  File "model/hourglass_um_crop_tiny.py", line 889, in run_train
    train(model, restore_step)
  File "/home/denseReg/model/train_single_gpu.py", line 149, in train
    assert not np.isnan(loss_value), 'Model diverged with loss = NaN'
AssertionError: Model diverged with loss = NaN
konrad-marzec commented 5 years ago

@melonwan can you help ?