DeMoN dataset evaluation Problem

[KyuminHwang]

@udaykusupati Hi, First of all thanks for your great efforts on this research. I met same issue about evaluation of DeMoN results but still suffering the problem. Also corrected pytorch v1.1.0 and v1.0.1 and CUDA 10.1 but results like below :

MVS    A.Rel  A.diff Sq.Rel   a=1    a=2    a=3
Paper  0.0679 0.1677 0.0555 0.9054 0.9644 0.9879
Tested 0.0855 0.2743 0.1089 0.9020 0.9612 0.9789

SUN3D  A.Rel  A.diff Sq.Rel   a=1    a=2    a=3
Paper  0.1332 0.3038 0.0910 0.8168 0.9421 0.9789
Tested 0.1333 0.3039 0.0910 0.8169 0.9421 0.9789

I changed train.py some codes for evaluation

# at line 35
parser.add_argument('data', metavar='DIR',
                    help='path to dataset')
parser.add_argument('data2', metavar='DIR',
                    help='path to dataset')
# at line 127
    if args.evaluate :
      val_set = SequenceFolder(
          args.data2,
          transform=valid_transform,
          seed=args.seed,
          ttype=args.ttype2,
          dataset = args.dataset
      )
  else :
      val_set = SequenceFolder(
          args.data,
          transform=valid_transform,
          seed=args.seed,
          ttype=args.ttype2,
          dataset = args.dataset
      )

And i runned using command like below :

python train.py ./dataset/train ./dataset/test --ttype2 test.txt -e -np --exp result --pretrained-mvdn pretrained/mvdnet_demon.pth.tar --print-freq 1

Is there any problem in my modified code ? I guessed that modified code isn't relevant to evaluation results.

@udaykusupati , Are you union the train and test into train folder ?

Originally posted by @makeastir in https://github.com/udaykusupati/Normal-Assisted-Stereo/issues/1#issuecomment-694671653

[udaykusupati]

Hi,

There is no problem with your modifications. The reported error (eg. A.Rel: 0.0855) is average across the DeMoN datasets where as MVS is a subset of the testset (containing MVS, RGBD, Scenes11, SUN3D). So you might need to change the test.txt so that it only loads the relevant testset for MVS (I vaguely remember the first 130 samples of the 708 sample testset are from MVS). Also make sure you uncomment https://github.com/udaykusupati/Normal-Assisted-Stereo/blob/491c0d3d2efd0e6c9f97cf84b1688266b9c201d6/train.py#L389 for a fair comparison.

[KyuminHwang]

@udaykusupati

Thank you for reply. I still has one questions about Table 1.

MVS    A.Rel  A.diff Sq.Rel   a=1    a=2    a=3
Paper  0.0679 0.1677 0.0555 0.9054 0.9644 0.9879

When testing this quantitative measure, Did you trained MVDNet using DeMoN dataset and Consistency module trained using MVS Datasets only?

Similarly, others dataset also trained MVDNet using DeMoN dataset and Consistency module trained specific datasets only ? Or, trained MVDNet and Consistency module through DeMoN and evaluated each datasets ?