losses stuck after 200-300 iterations and PSNR stays the same

[iraj465]

Hi. I have noticed that the img_losses and the reg_losses doesn't change much after 200-300 iterations and the PSNR values remain the same if we train for 5k iters for 500 iters. What could be the issue here? How to check if it is stuck at a local minima? Can you provide some insight into this?

Image set res is 720x1280

display_res [2048, 2048]                                                                                                                    [1047/1906]
texture_res [2048, 2048]                                                                                                                               
display_interval 0                                                                                                                                     
save_interval 100                                                                                                                                      
learning_rate [0.03, 0.01]                                                                                                                             
min_roughness 0.08                                                                                                                                     
custom_mip False                                                                                                                                       
random_textures True                                                                                                                                   
background white                                                                                                                                       
loss logl2                                                                                                                                             
out_dir out/                                                                                                                                           
ref_mesh /SHOG43V4BNZ4BGV2/                                                                                                                            
base_mesh None                                                                                                                                         
validate True                                                                                                                                          
save_custom 3D/vertical/footwear                                                                                                                       
vertical Footwear                                                                                                                                      
mtl_override None                                                                                                                                      
dmtet_grid 64                                                                                                                                          
mesh_scale 2.5                                                                                                                                         
env_scale 1.0                                                                                                                                          
envmap None                                                                                                                                            
display [{'bsdf': 'kd'}, {'bsdf': 'ks'}, {'bsdf': 'normal'}]                                                                                           
camera_space_light True                                                                                                                                
lock_light False                                                                                                                                       
lock_pos False                                                                                                                                         
sdf_regularizer 0.2                                                                                                                                    
laplace relative                                                                                                                                       
laplace_scale 15000.0                                                                                                                                  
pre_load False                                                                                                                                         
kd_min [0.03, 0.03, 0.03]                                                                                                                              
kd_max [0.8, 0.8, 0.8]                                                                                                                                 
ks_min [0, 0.08, 0]                                                                                                                                    
ks_max [0, 1, 0.6]                                                                                                                                     
nrm_min [-1.0, -1.0, 0.0]
nrm_max [1.0, 1.0, 1.0]
cam_near_far [0.1, 1000.0]
learn_light True
local_rank 0
multi_gpu True

iter=    0, img_loss=0.520525, reg_loss=0.333997, lr=0.02999, time=1110.3 ms, rem=37.01 m                                                              
iter=   10, img_loss=0.115935, reg_loss=0.322536, lr=0.02985, time=2102.7 ms, rem=1.16 h                                                               
iter=   20, img_loss=0.040579, reg_loss=0.304973, lr=0.02971, time=1210.2 ms, rem=39.94 m                                                              
iter=   30, img_loss=0.024776, reg_loss=0.286246, lr=0.02957, time=1162.7 ms, rem=38.17 m                                                              
iter=   40, img_loss=0.014629, reg_loss=0.276243, lr=0.02944, time=2284.2 ms, rem=1.24 h                                                               
iter=   50, img_loss=0.011005, reg_loss=0.267143, lr=0.02930, time=1520.7 ms, rem=49.42 m                                                              
iter=   60, img_loss=0.007770, reg_loss=0.259443, lr=0.02917, time=1691.3 ms, rem=54.68 m                                                              
iter=   70, img_loss=0.006342, reg_loss=0.252605, lr=0.02903, time=1013.6 ms, rem=32.60 m                                                              
iter=   80, img_loss=0.006045, reg_loss=0.246271, lr=0.02890, time=1866.2 ms, rem=59.72 m                                                              
iter=   90, img_loss=0.006008, reg_loss=0.240382, lr=0.02877, time=1274.1 ms, rem=40.56 m                                                              
iter=  100, img_loss=0.005665, reg_loss=0.234412, lr=0.02864, time=1540.6 ms, rem=48.79 m
iter=  110, img_loss=0.005504, reg_loss=0.228952, lr=0.02851, time=1943.8 ms, rem=1.02 h
iter=  120, img_loss=0.005388, reg_loss=0.223307, lr=0.02837, time=1444.1 ms, rem=45.25 m
iter=  130, img_loss=0.005353, reg_loss=0.217590, lr=0.02824, time=2060.9 ms, rem=1.07 h
iter=  140, img_loss=0.005315, reg_loss=0.212131, lr=0.02811, time=1754.6 ms, rem=54.39 m
iter=  150, img_loss=0.005379, reg_loss=0.206376, lr=0.02798, time=1959.8 ms, rem=1.01 h
iter=  160, img_loss=0.004989, reg_loss=0.201013, lr=0.02786, time=2995.5 ms, rem=1.53 h
iter=  170, img_loss=0.005171, reg_loss=0.195463, lr=0.02773, time=2100.5 ms, rem=1.07 h
iter=  180, img_loss=0.004836, reg_loss=0.189697, lr=0.02760, time=2153.7 ms, rem=1.09 h
iter=  190, img_loss=0.004745, reg_loss=0.184562, lr=0.02747, time=4406.9 ms, rem=2.22 h
iter=  200, img_loss=0.004990, reg_loss=0.178782, lr=0.02735, time=5243.9 ms, rem=2.62 h
iter=  210, img_loss=0.004602, reg_loss=0.173362, lr=0.02722, time=4133.3 ms, rem=2.06 h
iter=  220, img_loss=0.004554, reg_loss=0.167777, lr=0.02710, time=4683.3 ms, rem=2.32 h
iter=  230, img_loss=0.004518, reg_loss=0.162225, lr=0.02697, time=6840.4 ms, rem=3.36 h
iter=  240, img_loss=0.004821, reg_loss=0.156867, lr=0.02685, time=3525.0 ms, rem=1.72 h
iter=  250, img_loss=0.004489, reg_loss=0.151416, lr=0.02673, time=6360.3 ms, rem=3.09 h
iter=  260, img_loss=0.004426, reg_loss=0.146205, lr=0.02660, time=7235.2 ms, rem=3.50 h
iter=  270, img_loss=0.004239, reg_loss=0.140684, lr=0.02648, time=7351.8 ms, rem=3.53 h
iter=  280, img_loss=0.004277, reg_loss=0.135180, lr=0.02636, time=5051.7 ms, rem=2.41 h
iter=  290, img_loss=0.004253, reg_loss=0.129836, lr=0.02624, time=4219.4 ms, rem=2.00 h
iter=  300, img_loss=0.004522, reg_loss=0.124219, lr=0.02612, time=3221.8 ms, rem=1.52 h
iter=  310, img_loss=0.004291, reg_loss=0.119062, lr=0.02600, time=6673.6 ms, rem=3.13 h
iter=  320, img_loss=0.004226, reg_loss=0.113692, lr=0.02588, time=3849.4 ms, rem=1.80 h
iter=  330, img_loss=0.004151, reg_loss=0.108179, lr=0.02576, time=7991.0 ms, rem=3.71 h
iter=  340, img_loss=0.004067, reg_loss=0.102920, lr=0.02564, time=3777.0 ms, rem=1.74 h
iter=  350, img_loss=0.003939, reg_loss=0.097538, lr=0.02552, time=1979.4 ms, rem=54.43 m
iter=  360, img_loss=0.004042, reg_loss=0.092079, lr=0.02541, time=5179.4 ms, rem=2.36 h
iter=  370, img_loss=0.004167, reg_loss=0.086676, lr=0.02529, time=6477.5 ms, rem=2.93 h
iter=  380, img_loss=0.003982, reg_loss=0.081335, lr=0.02517, time=9030.5 ms, rem=4.06 h
iter=  390, img_loss=0.004081, reg_loss=0.075952, lr=0.02506, time=5271.0 ms, rem=2.36 h
iter=  400, img_loss=0.004170, reg_loss=0.070723, lr=0.02494, time=4942.2 ms, rem=2.20 h
iter=  410, img_loss=0.003959, reg_loss=0.065365, lr=0.02483, time=6747.7 ms, rem=2.98 h
iter=  420, img_loss=0.003835, reg_loss=0.059982, lr=0.02471, time=8019.5 ms, rem=3.52 h
iter=  430, img_loss=0.003728, reg_loss=0.054589, lr=0.02460, time=6394.1 ms, rem=2.79 h
iter=  440, img_loss=0.004048, reg_loss=0.049244, lr=0.02449, time=3360.8 ms, rem=1.46 h
iter=  450, img_loss=0.004118, reg_loss=0.043899, lr=0.02437, time=2594.9 ms, rem=1.12 h
iter=  460, img_loss=0.003833, reg_loss=0.038528, lr=0.02426, time=2175.0 ms, rem=55.82 m
iter=  470, img_loss=0.003697, reg_loss=0.033119, lr=0.02415, time=1183.8 ms, rem=30.19 m
iter=  480, img_loss=0.003709, reg_loss=0.027707, lr=0.02404, time=1120.1 ms, rem=28.38 m
iter=  490, img_loss=0.003672, reg_loss=0.022335, lr=0.02393, time=1093.1 ms, rem=27.51 m
iter=  500, img_loss=0.003597, reg_loss=0.016925, lr=0.02382, time=1242.2 ms, rem=31.05 m
iter=  510, img_loss=0.003345, reg_loss=0.014493, lr=0.02371, time=1013.3 ms, rem=25.16 m
iter=  520, img_loss=0.003478, reg_loss=0.014497, lr=0.02360, time=1037.8 ms, rem=25.60 m
iter=  530, img_loss=0.003329, reg_loss=0.014490, lr=0.02349, time=1190.3 ms, rem=29.16 m
iter=  540, img_loss=0.003333, reg_loss=0.014485, lr=0.02338, time=1202.7 ms, rem=29.27 m
iter=  550, img_loss=0.003341, reg_loss=0.014471, lr=0.02328, time=1343.1 ms, rem=32.46 m
iter=  560, img_loss=0.003375, reg_loss=0.014479, lr=0.02317, time=1116.2 ms, rem=26.79 m
iter=  570, img_loss=0.003361, reg_loss=0.014479, lr=0.02306, time=1111.8 ms, rem=26.50 m
iter=  580, img_loss=0.003329, reg_loss=0.014472, lr=0.02296, time=1087.1 ms, rem=25.73 m
iter=  590, img_loss=0.003318, reg_loss=0.014479, lr=0.02285, time=1070.4 ms, rem=25.15 m
iter=  600, img_loss=0.003304, reg_loss=0.014472, lr=0.02275, time=1290.7 ms, rem=30.12 m
iter=  610, img_loss=0.003155, reg_loss=0.014469, lr=0.02264, time=1128.6 ms, rem=26.15 m
iter=  620, img_loss=0.003173, reg_loss=0.014461, lr=0.02254, time=1165.9 ms, rem=26.82 m
iter=  630, img_loss=0.003190, reg_loss=0.014454, lr=0.02243, time=1141.5 ms, rem=26.06 m
iter=  640, img_loss=0.003252, reg_loss=0.014436, lr=0.02233, time=1179.9 ms, rem=26.74 m
iter=  650, img_loss=0.003369, reg_loss=0.014435, lr=0.02223, time=1161.1 ms, rem=26.12 m
iter=  660, img_loss=0.003468, reg_loss=0.014447, lr=0.02213, time=1203.8 ms, rem=26.89 m
iter=  670, img_loss=0.003353, reg_loss=0.014451, lr=0.02203, time=1275.3 ms, rem=28.27 m
iter=  680, img_loss=0.003268, reg_loss=0.014462, lr=0.02192, time=1252.1 ms, rem=27.55 m
iter=  690, img_loss=0.003235, reg_loss=0.014464, lr=0.02182, time=1135.2 ms, rem=24.79 m
iter=  700, img_loss=0.003287, reg_loss=0.014460, lr=0.02172, time=1016.6 ms, rem=22.03 m
iter=  710, img_loss=0.003323, reg_loss=0.014468, lr=0.02162, time=1036.8 ms, rem=22.29 m
iter=  720, img_loss=0.003333, reg_loss=0.014476, lr=0.02152, time=997.5 ms, rem=21.28 m
......

[jmunkberg]

Hello,

I would tweak the hyperparameters, like increase the learning rate, change the learning rate schedule and/or the regularizer lambdas. Given that this is a high dimensional optimization tasks, it is hard to give a succinct recommendation what would work best for a particular case.