LEN, ODN, and Joint training result looks odd

[chengzhag]

Hi Yinyu:

I tried LEN, ODN training code with batch size 32, but get the following loss curve: It looks like the test loss stops converging soon after a few epochs. All the test loss shows below:

I then tried joint training with the best LEN and ODN I have and the downloaded pretrained MGNet (Total3D_downloaded_mgnet): The test results of IoU compared to Total3D without joint training (Total3D_mgnet_beforejoint) and the downloaded pretrained Total3D (Total3D downloaded):

The results shows that joint training with my best LEN and ODN makes 3D Layout IoU worsen. However it's slightly better than the downloaded Total3D which is close to the results of the paper.

Also, It seems that only IoU is provided in the testing results. How can I get the mean absolute error of cam pose, mAP of 3D detection, object translation, rotation and scale errors same as the paper?

[yinyunie]

Hi,

For the ODN and LEN training, SUN-RGBD is of a small-scale dataset (~5000 images for training), it indeed overfits with training too many epochs, but we did not see such a serious overfitting problem. We adopt similar learning targets and evaluation metrics as https://github.com/thusiyuan/cooperative_scene_parsing. There would not be too much difference in the test curves, I think you could also have a check.

[chengzhag]

I'll check it out. Thanks a lot!

[chengzhag]

I tried the layout_evaluate code from cooperative_scene_parsing on the test results of the downloaded checkpoint with essential changes to path settings. However, I got 0 IoU from every sample. Are there any details that I missed? Is it possible to opensource the full evaluation code of the paper?

[Wi-sc]

Hi,

I also have some problems about the training. I tried to use downloaded pretrained models and my pretrained models to initialize the framework before joint training, but the results decrease both. Can you help me check the config file?

Downloaded pretrained models to initialize joint training: layout_iou: 0.440098 iou_3d: 0.109684 iou_2d: 0.594273 pitch_mae: 6.258617 roll_mae: 6.970254

My pretrained models to initialize joint training: layout_iou: 0.456693 iou_3d: 0.118330 iou_2d: 0.599362

This is my config setting:

method: TOTAL3D resume: False finetune: True weight: ['out/pretrained_models/pretrained_model.pth', 'out/pretrained_models/meshnet_model.pth'] seed: 123 device: use_gpu: True num_workers: 2 data: dataset: sunrgbd split: data/sunrgbd/splits tmn_subnetworks: 2 face_samples: 1 with_edge_classifier: True model: layout_estimation: method: PoseNet loss: PoseLoss object_detection: method: Bdb3DNet loss: DetLoss mesh_reconstruction: method: DensTMNet loss: ReconLoss optimizer: method: Adam lr: 1e-4 betas: [0.9, 0.999] eps: 1e-08 weight_decay: 1e-04 scheduler: patience: 5 factor: 0.5 threshold: 0.01 train: epochs: 200 phase: 'joint' # 'layout_estimation' or 'object_detection' or 'joint'. freeze: ['mesh_reconstruction'] batch_size: 3 test: phase: 'joint' batch_size: 3 demo: phase: 'joint' log: vis_path: visualization save_results: True vis_step: 100 print_step: 50 path: out/total3d

[longbowzhang]

Hi @chengzhag,

Sorry to bother you. I am also trying to reproduce the results presented in the paper. Now I am working on the first stage, which is training LEN and ODN individually. But I fail to reproduce the results. For example, '3D Layout' in table 1 says 57.6 w/o joint training. But the highest value I can get is 48.9. Therefore, I am wondering whether you could kindly share your config. file?

Best, Longbow