can't get the desired training result (weird issue)

[soheilAppear]

ISo last time I was trying to train the program with the tiny version of the KITTI dataset (only 5 images which you provide) and after saving the checkpoints everything looks ok. Since I did not use a lot of images for training I got this result as the output which is very low quality but its ok since it was with a small set of training:

Without any changes, I tried to do this test again and followed the exact same thing that I did and this time I got this weird output which is not similar to the above:

Also when I used your models for my own dataset (the same dataset as above) I got something like this which I believe is similar to the first output:

I'm wondered what is happening exactly and cause this weird issue for me. Looks like in the second image filtering does not do its job correctly. (Maybe there is something wrong with my graphic cards?I even tried to reset everything and again install the docker but it did not work either)

Also this is the .yaml filer that I used for tiny_kitty training:

model: name: 'SelfSupModel' optimizer: name: 'Adam' depth: lr: 0.0002 pose: lr: 0.0002 scheduler: name: 'StepLR' step_size: 30 gamma: 0.5 depth_net: name: 'PackNet01' version: '1A' pose_net: name: 'PoseNet' version: '' params: crop: 'garg' min_depth: 0.0 max_depth: 80.0 datasets: augmentation: image_shape: (192, 640) train: batch_size: 1 dataset: ['KITTI'] path: ['./data/datasets/KITTI_tiny'] split: ['kitti_tiny.txt'] depth_type: ['velodyne'] repeat: [1] validation: dataset: ['KITTI'] path: ['./data/datasets/KITTI_tiny'] split: ['kitti_tiny.txt', 'kitti_tiny.txt'] depth_type: ['velodyne'] test: dataset: ['KITTI'] path: ['./data/datasets/KITTI_tiny'] split: ['kitti_tiny.txt'] depth_type: ['velodyne'] checkpoint: filepath: './data/experiments_Final' monitor: 'abs_rel_pp_gt' monitor_index: 0 mode: 'min'

[tonytu16]

I tried training with my own custom dataset and has encountered similar issue, here is what the inference output looks like: