nan and inf

[1991yuyang]

Hello, I would like to ask if during the training process, there are prompts for the presence of nan and inf in the input, and the relative depth of each pixel in the output of the model after training is basically the same. Here are my training commands.

python execute.py --exe train --model_name distdepth-distilled --frame_ids 0 -1 1 --log_dir='./tmp/model_from_server_34_256_10_frame_seq' --data_path SimSIN-simple --dataset SimSIN --batch_size 16 --width 256 --height 256 --max_depth 10.0 --num_epochs 500 --scheduler_step_size 200 --learning_rate 0.0001 --thre 0.95 --num_layers 34 --log_frequency 100 --use_stereo --load_weights_folder weights --save_frequency 20

output of model as bellow:

[choyingw]

Hi, thank you for reporting this. Does the instability happen at the start of the training, or it was triggered at some point during training? Thanks

[choyingw]

I have identified the root cause (too large loss weights), and the current head should fix the issue.