Can't reproduce the result for MT-PNet

[AlbertHuyb]

I ran the original code in this repo with the data downloaded from the according url. According to the config.json file, I trained for 40 epochs and the loss value reached 1.0892. But when I test on the Area 5 data, it reported as follows:

> Overall accuracy: 0.812
> Mean accuracy: 0.490
> Mean IoU: 0.397
> Mean precision: 0.281
> Mean recall: 0.238
> Writing report to logs/s3dis/eval.json...

I ran the prediction code without --mvcrf and I don't think the performance reaches the result reported in the paper, which has 86.7% oAcc for MT-PNet. Could you please help me about this issue?

[LiDaiY]

Same question. I train for 200 epochs and the loss reached 0.4957. The test result(without CRF) on the Area 5 is:

Overall accuracy: 0.833 Mean accuracy: 0.523 Mean IoU: 0.432

And I train for 40 epochs and the loss reached 1.0127. The test result(without CRF) on the Area 5 is:

Overall accuracy: 0.807 Mean accuracy: 0.476 Mean IoU: 0.394

[pqhieu]

Hi all,

Thank you for your interests. I found out that all of my CVPR test results are on Area 6 instead of Area 5. Sorry for the trouble. Your results are correct on Area 5.

[sidml]

@pqhieu Is the comparison with other methods still valid (since the test sets are different) ?