The problems about testing on DeMoN test set

[XiWJ]

Hi, thank you for sharing the code. Then I have 2 questions about testing on DeMoN.

1. The python command line of testing on DeMoN without consistency module whether should be

   python train.py ./dataset/test --ttype2 test.txt -e -np --pretrained-mvdn pretrained/mvdnet_demon.pth.tar --print-freq 1

   it just data path difference, from ./dataset/train to ./dataset/test, is that true?

2. I used the above test command to test the NAS performance on DeMoN with the pretrained model without consistency module, then I got the logs saved in progress_log_summary.csv,

   
   MVS    A.Rel  A.diff Sq.Rel  RMSE  R. log   a=1    a=2    a=3
   Paper  0.0679 0.1677 0.0555 0.3752 0.1419 0.9054 0.9644 0.9879
   Tested 0.3625 1.0126 0.6441 1.2951 0.4524 0.4049 0.7062 0.8436

SUN3D A.Rel A.diff Sq.Rel RMSE R. log a=1 a=2 a=3 Paper 0.1332 0.3038 0.0910 0.3994 0.1820 0.8168 0.9421 0.9789 Tested 0.2267 0.5327 0.1904 0.6412 0.3027 0.5448 0.8342 0.9465

RGBD A.Rel A.diff Sq.Rel RMSE R. log a=1 a=2 a=3 Paper 0.1314 0.4737 0.2126 0.6190 0.2091 0.8565 0.9289 0.9450 Tested 0.2798 0.6658 0.3694 0.8199 0.3409 0.6345 0.8208 0.8954

Scenes A.Rel A.diff Sq.Rel RMSE R. log a=1 a=2 a=3 Paper 0.0380 0.1130 0.0666 0.3710 0.0946 0.9754 0.9900 0.9947 Tested 0.3366 1.5897 0.8402 2.2504 0.4843 0.3826 0.6690 0.8363

Besides, I tested the NAS performance on SUN3D with the pretrained model with consistency module, then I got the logs saved in progress_log_summary.csv,

SUN3D A.Rel A.diff Sq.Rel RMSE R. log a=1 a=2 a=3 Paper 0.1332 0.3038 0.0910 0.3994 0.1820 0.8168 0.9421 0.9789 Tested 0.2207 0.5100 0.1850 0.6115 0.2922 0.5693 0.8455 0.9443

The performance of the above two log records is quite different from the paper, maybe there is something wrong with the test command (some parameters need to be adjusted)? Or the pretrained models are not the final result model of the paper?

Could you check these testing experiments again? 

Looking forward to your reply!

[udaykusupati]

Yes, it must just be a path difference. Just make sure the SUN3D test has 160 instances and the whole DeMoN set has 708. I just tested the commands and the models and could replicate the results( Just made a change in train.py which results in a minor change, but not as much as your values suggest, something else might be going wrong). So I might need more information from you about anything you might have done to help you better!

[XiWJ]

Thank you for your quick reply!

1. The environment in my PC is

   Ubuntu 16.04
   python 3.6
   pytorch 1.5.0
   CUDA 10.1
   Cudnn 7.5.0
   pip install pillow scipy==1.2.1 argparse tensorboardX progressbar2 path.py h5py blessings scikit-image

2. Firstly, I used this command to test NAS on SUN3D with consistency module.

   python train.py ./dataset/test --ttype2 sun_test.txt -e --pretrained-mvdn pretrained/mvdnet_sun3d.pth.tar --pretrained-cons pretrained/cons_sun3d.pth.tar -tc --print-freq 1

   then I got the error log as shown in the picture

3. So I modify some code in train.py, from line 119 to 152, here is

   print("=> fetching scenes in '{}'".format(args.data))
   # train_set = SequenceFolder(
   #     args.data,
   #     transform=train_transform,
   #     seed=args.seed,
   #     ttype=args.ttype,
   #     dataset = args.dataset
   # )
   val_set = SequenceFolder(
       args.data,
       transform=valid_transform,
       seed=args.seed,
       ttype=args.ttype2,
       dataset = args.dataset
   )
   
   # train_set.samples = train_set.samples[:len(train_set) - len(train_set)%args.batch_size]
   
   # print('{} samples found in {} train scenes'.format(len(train_set), len(train_set.scenes)))
   print('{} samples found in {} valid scenes'.format(len(val_set), len(val_set.scenes)))
   # train_loader = torch.utils.data.DataLoader(
   #     train_set, batch_size=args.batch_size, shuffle=True,
   #     num_workers=args.workers, pin_memory=True)
   val_loader = torch.utils.data.DataLoader(
       val_set, batch_size=1, shuffle=False,
       num_workers=args.workers, pin_memory=True)
   
   if args.epoch_size == 0:
       args.epoch_size = len(val_loader)   # modify from train_loader to val_loader
   
   # create model
   print("=> creating model")

   Then it can run successfully as shown in picture

The number of SUN3D samples is 160. But the progress_log_summary.csv in the checkpoint is

SUN3D  A.Rel  A.diff Sq.Rel  RMSE  R. log   a=1    a=2    a=3
Paper  0.1332 0.3038 0.0910 0.3994 0.1820 0.8168 0.9421 0.9789
Tested 0.2207 0.5100 0.1850 0.6115 0.2922 0.5693 0.8455 0.9443

It is quite different from the paper, something might be wrong. Could you give me some suggestions?

Looking forward to your reply again!

[udaykusupati]

Thanks for the screenshot. I see the values for SUN3D test instances match with mine (the expected ones), so you might be referring to a wrong progress log. I checked the if there is a bug in logging but there isn't. You can set an experiment name for the checkpoint folder using the --exp argument which makes it easier to locate the specific log.

[XiWJ]

Thank you for your suggestion, I tried it. But the problem still exists.

Finally, I just modify pytorch version from 1.5.0 down to 1.1.0, something amazing happened, it works, replicate the results of the paper.

Thank you!

[freshjh]

@XiWJ Hello, I am searching for the Sun 3D depth dataset, could you please share the download link of the dataset. Thanks!