Detailed training parameters

[zixulll]

Dear author, thank you for your excellent work, when I used the decoder of EMSANet for the semantic segmentation task on NYUv2 and SUNRGB-D datasets, the metrics of the experiment were far from the original paper. Could you please provide the detailed multi-task training code? The code I used is as follows: CUDA_VISIBLE_DEVICES=1 python main.py \ --results-basepath ./results \ --dataset nyuv2 \ --dataset-path ./datasets/nyuv2 \ --input-modalities rgbd \ --tasks semantic scene instance orientation \ --enable-panoptic \ --tasks-weighting 1.0 0.25 2.0 0.5 \ --instance-weighting 2 1 \ --rgbd-encoder-backbone swin-multi-t-v2-128 \ --encoder-normalization layernorm \ --rgbd-encoder-backbone-pretrained-weights-filepath ./trained_models/imagenet/swin_multi_t_v2_128.pth \ --validation-batch-size 8 \ --validation-skip 1.0 \ --checkpointing-skip 0.8 \ --checkpointing-best-only \ --checkpointing-metrics valid_semantic_miou bacc mae_gt_deg panoptic_deeplab_semantic_miou panoptic_all_with_gt_deeplab_pq \ --batch-size 8 \ --learning-rate 0.03 \ --no-pretrained-backbone \ --semantic-decoder emsanet \ --semantic-encoder-decoder-fusion swin-ln-add \ --semantic-decoder-n-channels 512 256 128 \ --semantic-decoder-upsampling learned-3x3-zeropad \ --wandb-mode disabled

[Tripton]

The command looks right in general, however, I noticed the --no-pretrained-backbone argument. Even if you've specified --rgbd-encoder-backbone-pretrained-weights-filepath, this argument prevents the use of the pretrained ImageNet weights. This leads to a performance gap and might be the root cause of your issue.

To confirm that the weights are being loaded correctly, you should see this print just before training begins:

Get model and dataset
Loading pretrained weights from: './trained_models/imagenet/swin_multi_t_v2_128.pth'

Please try running the training without the --no-pretrained-backbone argument. If this still doesn't resolve your issue, please let me know.

[zixulll]

The above problems have been solved according to your guidance, thank you. Sorry I have another question, when the model is trained on the sunrgbd dataset, do I need to add --sunrgbd-depth-do-not-force-mm to the training script? I found this line in the inference script. My detailed training script on the sunrgbd dataset is as follows:

CUDA_VISIBLE_DEVICES=1 python main.py \ --results-basepath ./results \ --dataset sunrgbd \ --dataset-path ./datasets/sunrgbd \ --input-modalities rgbd \ --tasks semantic scene instance orientation \ --sunrgbd-depth-do-not-force-mm\ --enable-panoptic \ --tasks-weighting 1.0 0.25 2.0 0.5 \ --instance-weighting 2 1 \ --rgbd-encoder-backbone swin-multi-t-v2-128 \ --encoder-normalization layernorm \ --rgbd-encoder-backbone-pretrained-weights-filepath ./trained_models/imagenet/swin_multi_t_v2_128.pth \ --validation-batch-size 8 \ --validation-skip 1.0 \ --checkpointing-skip 0.8 \ --checkpointing-best-only \ --checkpointing-metrics valid_semantic_miou bacc mae_gt_deg panoptic_deeplab_semantic_miou panoptic_all_with_gt_deeplab_pq \ --batch-size 8 \ --learning-rate 0.03 \ --wandb-mode disabled

[Tripton]

Great to hear that the previous issue is resolved. Regarding SUNRGB-D training: Our original training used depth images in a way, which now can only be replicated using the --sunrgbd-depth-do-not-force-mm flag. This flag was introduced due to a change in the Dataset Repo (in Version 0.5.4), which adjusted the scaling of SUNRGB-D images to millimeters, to maintain backward compatibility. The updated scaling is particularly advantageous for deploying the model in real-world applications.

So setting the flag or not depends on your scenario:

• If your goal is to replicate our trainings as close as possible, you should set the flag.
• If your focus is towards real-world application, you should omit the flag. This will use depth images as outputted by most depth sensors, and the training will still archive comparable results to the original trainings.

[zixulll]

Thanks for your patient guidance, I will follow your method to experiment. Thank you very much.