Stuck at the end of the training

The configuration file is the following:

{
    "ref_mesh": "data/nerd/bansi_rescaled",
    "random_textures": true,
    "iter": 100,
    "save_interval": 100,
    "texture_res": [ 512, 512 ],
    "train_res": [512, 512],
    "batch": 4,
    "learning_rate": [0.03,0.01],
    "kd_min" : [0.03, 0.03, 0.03],
    "kd_max" : [0.8, 0.8, 0.8],
    "ks_min" : [0, 0.08, 0],
    "ks_max" : [0, 1.0, 1.0],
    "dmtet_grid" : 128,
    "mesh_scale" : 2.5,
    "background" : "white",
    "camera_space_light" : true,
    "validate" : false,
    "display" : [{"bsdf":"kd"}, {"bsdf":"ks"}, {"bsdf" : "normal"}],
    "out_dir": "bansimodelo1"
}

and the output:

Loading extension module renderutils_plugin... 
iter=    0, img_loss=0.101132, reg_loss=0.333983, lr=0.02999, time=312.7 ms, rem=31.27 s 
iter=   10, img_loss=0.069344, reg_loss=0.253270, lr=0.02985, time=282.6 ms, rem=25.44 s
iter=   20, img_loss=0.044743, reg_loss=0.125129, lr=0.02971, time=283.3 ms, rem=22.67 s
iter=   30, img_loss=0.040794, reg_loss=0.026143, lr=0.02957, time=286.2 ms, rem=20.03 s
iter=   40, img_loss=0.044204, reg_loss=0.014757, lr=0.02944, time=286.5 ms, rem=17.19 s
iter=   50, img_loss=0.040038, reg_loss=0.014670, lr=0.02930, time=284.6 ms, rem=14.23 s
iter=   60, img_loss=0.038747, reg_loss=0.014608, lr=0.02917, time=283.4 ms, rem=11.34 s
iter=   70, img_loss=0.039025, reg_loss=0.014561, lr=0.02903, time=282.4 ms, rem=8.47 s
iter=   80, img_loss=0.041319, reg_loss=0.014514, lr=0.02890, time=284.0 ms, rem=5.68 s
iter=   90, img_loss=0.039661, reg_loss=0.014486, lr=0.02877, time=284.0 ms, rem=2.84 s
iter=  100, img_loss=0.040664, reg_loss=0.014457, lr=0.02864, time=283.9 ms, rem=0.00 s
Running validation
MSE,      PSNR
0.01257213, 19.220

I performed other test and works without problem, but from one day to the next it stopped working. I tried to set --validate to false but didn't work.

NVlabs / nvdiffrec

Stuck at the end of the training #123