search

waychin-weiqin / ITSA

ITSA: An Information-Theoretic Approach to Automatic Shortcut Avoidance and Domain Generalization in Stereo Matching Networks (CVPR 2022)
MIT License
15 stars 1 forks source link

Issues about the performace of model trained with ITSA #5

Closed leonidas-timefly closed 1 year ago

leonidas-timefly commented 1 year ago

Thanks for your great work!

While I was reproducing your work, I encountered such a problem: I could reproduce the experimental results using the pre-trained weights you provided but could not train such a model from scratch on my own. These are the D1 results I obtained on the ETH3D dataset using your provided pre-trained model, which is in line with the result of Table 4 (9.8 / 7.1 / 5.1 ) in the paper:

| PSMNet | GwcNet | CFNet | | 9.768 | 7.122 | 5.152 |

While I trained the PSMNet from scratch with ITSA on the SceneFlow dataset, I obtained quite strange results. This is the training log:

[INFO] Epoch: 1/20 Total Training Loss: 0.041 Time/Epoch: 0.00 hrs [INFO] Epoch: 2/20 Total Training Loss: 0.033 Time/Epoch: 0.00 hrs [INFO] Epoch: 3/20 Total Training Loss: 0.025 Time/Epoch: 0.00 hrs [INFO] Epoch: 4/20 Total Training Loss: 0.018 Time/Epoch: 0.00 hrs [INFO] Epoch: 5/20 Total Training Loss: 0.032 Time/Epoch: 0.00 hrs [INFO] Epoch: 6/20 Total Training Loss: 0.028 Time/Epoch: 0.00 hrs [INFO] Epoch: 7/20 Total Training Loss: 0.023 Time/Epoch: 0.00 hrs [INFO] Epoch: 8/20 Total Training Loss: 0.025 Time/Epoch: 0.00 hrs [INFO] Epoch: 9/20 Total Training Loss: 0.026 Time/Epoch: 0.00 hrs [INFO] Epoch: 10/20 Total Training Loss: 0.024 Time/Epoch: 0.00 hrs [INFO] Epoch: 11/20 Total Training Loss: 0.019 Time/Epoch: 0.00 hrs [INFO] Epoch: 12/20 Total Training Loss: 0.023 Time/Epoch: 0.00 hrs [INFO] Epoch: 13/20 Total Training Loss: 0.014 Time/Epoch: 0.00 hrs [INFO] Epoch: 14/20 Total Training Loss: 0.020 Time/Epoch: 0.00 hrs [INFO] Epoch: 15/20 Total Training Loss: 0.013 Time/Epoch: 0.00 hrs [INFO] Epoch: 16/20 Total Training Loss: 0.015 Time/Epoch: 0.00 hrs [INFO] Epoch: 17/20 Total Training Loss: 0.019 Time/Epoch: 0.00 hrs [INFO] Epoch: 18/20 Total Training Loss: 0.013 Time/Epoch: 0.00 hrs [INFO] Epoch: 19/20 Total Training Loss: 0.020 Time/Epoch: 0.00 hrs [INFO] Epoch: 20/20 Total Training Loss: 0.016 Time/Epoch: 0.00 hrs

It can be seen that the training loss is decreasing stably, so the model converged. But when I evaluate the domain generalization ability of this model on ETH3D, the D1 is very large:

[INFO] Processing: 1/27 im0.png D1: 99.256 EPE: 41.973 Time: 3.289 [INFO] Processing: 2/27 im0.png D1: 99.999 EPE: 31.850 Time: 0.415 [INFO] Processing: 3/27 im0.png D1: 99.056 EPE: 37.971 Time: 0.391 [INFO] Processing: 4/27 im0.png D1: 97.579 EPE: 19.702 Time: 0.436 [INFO] Processing: 5/27 im0.png D1: 99.257 EPE: 61.363 Time: 0.305 [INFO] Processing: 6/27 im0.png D1: 94.594 EPE: 20.138 Time: 0.300 [INFO] Processing: 7/27 im0.png D1: 99.987 EPE: 44.592 Time: 0.329 [INFO] Processing: 8/27 im0.png D1: 93.245 EPE: 14.864 Time: 0.313 [INFO] Processing: 9/27 im0.png D1: 89.783 EPE: 16.917 Time: 0.421 [INFO] Processing: 10/27 im0.png D1: 99.878 EPE: 46.286 Time: 0.305 [INFO] Processing: 11/27 im0.png D1: 99.985 EPE: 29.110 Time: 0.409 [INFO] Processing: 12/27 im0.png D1: 99.200 EPE: 17.821 Time: 0.306 [INFO] Processing: 13/27 im0.png D1: 99.203 EPE: 32.307 Time: 0.415 [INFO] Processing: 14/27 im0.png D1: 98.734 EPE: 14.128 Time: 0.288 [INFO] Processing: 15/27 im0.png D1: 99.457 EPE: 51.270 Time: 0.411 [INFO] Processing: 16/27 im0.png D1: 99.874 EPE: 29.473 Time: 0.310 [INFO] Processing: 17/27 im0.png D1: 98.966 EPE: 18.426 Time: 0.301 [INFO] Processing: 18/27 im0.png D1: 99.997 EPE: 30.620 Time: 0.307 [INFO] Processing: 19/27 im0.png D1: 83.594 EPE: 23.378 Time: 0.413 [INFO] Processing: 20/27 im0.png D1: 99.382 EPE: 21.329 Time: 0.307 [INFO] Processing: 21/27 im0.png D1: 99.999 EPE: 19.246 Time: 0.419 [INFO] Processing: 22/27 im0.png D1: 100.000 EPE: 23.887 Time: 0.297 [INFO] Processing: 23/27 im0.png D1: 95.117 EPE: 30.394 Time: 0.274 [INFO] Processing: 24/27 im0.png D1: 98.748 EPE: 18.404 Time: 0.292 [INFO] Processing: 25/27 im0.png D1: 93.469 EPE: 16.792 Time: 0.304 [INFO] Processing: 26/27 im0.png D1: 99.664 EPE: 27.692 Time: 0.298 [INFO] Processing: 27/27 im0.png D1: 99.998 EPE: 40.488 Time: 0.316 [INFO] Mean D1: 97.704 Mean EPE: 28.904 [INFO] Average Time: 0.451

I also obtained similar results with GwcNet:

[INFO] Processing: 1/27 im0.png D1: 95.305 EPE: 39.213 Time: 2.175 [INFO] Processing: 2/27 im0.png D1: 100.000 EPE: 17.730 Time: 0.154 [INFO] Processing: 3/27 im0.png D1: 89.984 EPE: 31.256 Time: 0.152 [INFO] Processing: 4/27 im0.png D1: 93.118 EPE: 6.798 Time: 0.117 [INFO] Processing: 5/27 im0.png D1: 90.497 EPE: 18.765 Time: 0.111 [INFO] Processing: 6/27 im0.png D1: 97.333 EPE: 12.341 Time: 0.111 [INFO] Processing: 7/27 im0.png D1: 99.972 EPE: 13.697 Time: 0.111 [INFO] Processing: 8/27 im0.png D1: 96.325 EPE: 4.712 Time: 0.114 [INFO] Processing: 9/27 im0.png D1: 90.128 EPE: 7.376 Time: 0.154 [INFO] Processing: 10/27 im0.png D1: 99.955 EPE: 22.883 Time: 0.118 [INFO] Processing: 11/27 im0.png D1: 99.999 EPE: 24.707 Time: 0.170 [INFO] Processing: 12/27 im0.png D1: 95.077 EPE: 13.278 Time: 0.114 [INFO] Processing: 13/27 im0.png D1: 97.103 EPE: 24.140 Time: 0.155 [INFO] Processing: 14/27 im0.png D1: 93.056 EPE: 12.162 Time: 0.111 [INFO] Processing: 15/27 im0.png D1: 99.005 EPE: 33.062 Time: 0.153 [INFO] Processing: 16/27 im0.png D1: 99.594 EPE: 31.603 Time: 0.154 [INFO] Processing: 17/27 im0.png D1: 92.647 EPE: 10.884 Time: 0.112 [INFO] Processing: 18/27 im0.png D1: 99.948 EPE: 26.999 Time: 0.115 [INFO] Processing: 19/27 im0.png D1: 81.703 EPE: 9.385 Time: 0.154 [INFO] Processing: 20/27 im0.png D1: 98.502 EPE: 15.410 Time: 0.118 [INFO] Processing: 21/27 im0.png D1: 100.000 EPE: 55.834 Time: 0.153 [INFO] Processing: 22/27 im0.png D1: 99.996 EPE: 17.310 Time: 0.114 [INFO] Processing: 23/27 im0.png D1: 99.820 EPE: 22.402 Time: 0.113 [INFO] Processing: 24/27 im0.png D1: 94.862 EPE: 6.031 Time: 0.113 [INFO] Processing: 25/27 im0.png D1: 82.943 EPE: 5.530 Time: 0.152 [INFO] Processing: 26/27 im0.png D1: 99.853 EPE: 29.638 Time: 0.111 [INFO] Processing: 27/27 im0.png D1: 99.997 EPE: 20.117 Time: 0.152 [INFO] Mean D1: 95.805 Mean EPE: 19.750 [INFO] Average Time: 0.207

and CFNet:

[INFO] Processing: 1/27 im0.png D1: 99.751 EPE: 43.274 Time: 2.083 [INFO] Processing: 2/27 im0.png D1: 98.707 EPE: 29.183 Time: 0.150 [INFO] Processing: 3/27 im0.png D1: 98.158 EPE: 25.470 Time: 0.147 [INFO] Processing: 4/27 im0.png D1: 95.308 EPE: 10.168 Time: 0.108 [INFO] Processing: 5/27 im0.png D1: 99.149 EPE: 33.761 Time: 0.108 [INFO] Processing: 6/27 im0.png D1: 95.462 EPE: 22.213 Time: 0.108 [INFO] Processing: 7/27 im0.png D1: 98.955 EPE: 44.339 Time: 0.108 [INFO] Processing: 8/27 im0.png D1: 95.411 EPE: 11.887 Time: 0.108 [INFO] Processing: 9/27 im0.png D1: 91.759 EPE: 7.506 Time: 0.147 [INFO] Processing: 10/27 im0.png D1: 99.817 EPE: 40.963 Time: 0.108 [INFO] Processing: 11/27 im0.png D1: 99.522 EPE: 20.332 Time: 0.147 [INFO] Processing: 12/27 im0.png D1: 98.390 EPE: 14.653 Time: 0.108 [INFO] Processing: 13/27 im0.png D1: 98.504 EPE: 24.600 Time: 0.147 [INFO] Processing: 14/27 im0.png D1: 95.592 EPE: 8.742 Time: 0.107 [INFO] Processing: 15/27 im0.png D1: 99.131 EPE: 26.545 Time: 0.146 [INFO] Processing: 16/27 im0.png D1: 99.570 EPE: 32.308 Time: 0.146 [INFO] Processing: 17/27 im0.png D1: 97.567 EPE: 18.711 Time: 0.107 [INFO] Processing: 18/27 im0.png D1: 99.168 EPE: 26.704 Time: 0.108 [INFO] Processing: 19/27 im0.png D1: 89.824 EPE: 6.451 Time: 0.148 [INFO] Processing: 20/27 im0.png D1: 98.728 EPE: 23.367 Time: 0.108 [INFO] Processing: 21/27 im0.png D1: 99.758 EPE: 38.880 Time: 0.146 [INFO] Processing: 22/27 im0.png D1: 99.502 EPE: 29.948 Time: 0.107 [INFO] Processing: 23/27 im0.png D1: 97.808 EPE: 22.382 Time: 0.106 [INFO] Processing: 24/27 im0.png D1: 93.502 EPE: 12.633 Time: 0.107 [INFO] Processing: 25/27 im0.png D1: 96.486 EPE: 25.823 Time: 0.148 [INFO] Processing: 26/27 im0.png D1: 99.306 EPE: 28.860 Time: 0.110 [INFO] Processing: 27/27 im0.png D1: 99.443 EPE: 35.199 Time: 0.152 [INFO] Mean D1: 97.566 Mean EPE: 24.626 [INFO] Average Time: 0.197

What I have done is just download this repository and do not modify any code. The dataset I download is: image After I downloaded these sets, I organized them in this hierarchy: SceneFlow   |-- Driving     |-- disparity     |-- frames_cleanpass   |-- Flyingthings3D     |-- disparity     |-- frames_cleanpass   |-- Monkaa     |-- disparity     |-- frames_cleanpass I don't know what's wrong with my operation. As I could reproduce the results claimed in paper with the provided pre-trained model, the problem should exist in the training process or the training dataset. But I could not figure it out. Could you please figure it out? Thanks.

waychin-weiqin commented 1 year ago

Hi @leonidas-timefly, thank you for your interest in our work.

May I know if this issue only occurs with the ETH dataset, or if it also exists in other benchmarks (e.g. KITTI and Middlebury)?

I would suggest checking if you have loaded your pre-trained weights correctly. Also, check the data preprocessing pipeline and make sure it is consistent across datasets.

leonidas-timefly commented 1 year ago

waychin-weiqin

Hi @waychin-weiqin, thanks for your fast reply.

I've checked the performance with the KITTI 2015 benchmark. The performance obtained with your provided pre-trained weights on KITTI 2015 is consistent with the results in your paper: I reproduced: PSMNet: 5.793; GwcNet: 5.384; CFNet: 4.733 In your paper:PSMNet: 5.8; GwcNet: 5.4; CFNet: 4.7 While I still obtain strange results with the models trained by myself:

for PSMNet:

[INFO] Mean D1: 87.601 Mean EPE: 23.236

for GwcNet:

[INFO] Mean D1: 74.531 Mean EPE: 9.046

for CFNet:

[INFO] Mean D1: 87.859 Mean EPE: 17.956

So I guess there is no problem existing in the evaluation stage. I'll check the training process focusing on the data preprocessing pipeline. Another issue is that could you please check the MiddleBury2014 dataset on its official website? I find that there is no Full / Half / Quarter size subset. I also find that the size of MiddleBury2014 is 10 for the test set, 10 for the training set, and 13 for an additional set while not 15 claimed in your paper. Thanks for your help. That's very kind of you.

waychin-weiqin commented 1 year ago

Hi again,

in regards to the Middlebury dataset, you can find the Full, Half and Quarter resolution datasets using this URL.

leonidas-timefly commented 1 year ago

Hi again,

in regards to the Middlebury dataset, you can find the Full, Half and Quarter resolution datasets using this URL.

Hi @waychin-weiqin ,

Thanks for your information. The valid URL is https://vision.middlebury.edu/stereo/submit3/.

leonidas-timefly commented 1 year ago

Hi again,

in regards to the Middlebury dataset, you can find the Full, Half and Quarter resolution datasets using this URL.

Hi @waychin-weiqin ,

Sorry to trouble you. I checked the training process and data processing, and found no problem. Still, I obtain strange results. I want to make sure if the training loss is normal? This is the training log of PSMNet on SceneFlow:

[INFO] BEST epoch: 1 Total Test Error = 56.606 Time/Epoch: 0.07 hrs [INFO] Epoch: 1/20 Total Training Loss: 0.046 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 1 Total Test Error = 56.644 Time/Epoch: 0.08 hrs [INFO] Epoch: 2/20 Total Training Loss: 0.035 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 2 Total Test Error = 46.876 Time/Epoch: 0.08 hrs [INFO] Epoch: 3/20 Total Training Loss: 0.023 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 3 Total Test Error = 33.355 Time/Epoch: 0.08 hrs [INFO] Epoch: 4/20 Total Training Loss: 0.021 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 4 Total Test Error = 30.693 Time/Epoch: 0.07 hrs [INFO] Epoch: 5/20 Total Training Loss: 0.018 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 4 Total Test Error = 30.693 Time/Epoch: 0.08 hrs [INFO] Epoch: 6/20 Total Training Loss: 0.015 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 6 Total Test Error = 29.578 Time/Epoch: 0.08 hrs [INFO] Epoch: 7/20 Total Training Loss: 0.025 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 6 Total Test Error = 29.578 Time/Epoch: 0.08 hrs [INFO] Epoch: 8/20 Total Training Loss: 0.016 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 8 Total Test Error = 28.171 Time/Epoch: 0.07 hrs [INFO] Epoch: 9/20 Total Training Loss: 0.013 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 9 Total Test Error = 25.306 Time/Epoch: 0.08 hrs [INFO] Epoch: 10/20 Total Training Loss: 0.012 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 10 Total Test Error = 25.045 Time/Epoch: 0.08 hrs [INFO] Epoch: 11/20 Total Training Loss: 0.014 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 11 Total Test Error = 22.326 Time/Epoch: 0.08 hrs [INFO] Epoch: 12/20 Total Training Loss: 0.016 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 12 Total Test Error = 20.686 Time/Epoch: 0.08 hrs [INFO] Epoch: 13/20 Total Training Loss: 0.018 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 13 Total Test Error = 18.278 Time/Epoch: 0.08 hrs [INFO] Epoch: 14/20 Total Training Loss: 0.013 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 14 Total Test Error = 16.742 Time/Epoch: 0.07 hrs [INFO] Epoch: 15/20 Total Training Loss: 0.012 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 15 Total Test Error = 15.593 Time/Epoch: 0.08 hrs [INFO] Epoch: 16/20 Total Training Loss: 0.016 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 16 Total Test Error = 14.866 Time/Epoch: 0.07 hrs [INFO] Epoch: 17/20 Total Training Loss: 0.012 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 17 Total Test Error = 14.420 Time/Epoch: 0.08 hrs [INFO] Epoch: 18/20 Total Training Loss: 0.011 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 17 Total Test Error = 14.420 Time/Epoch: 0.08 hrs [INFO] Epoch: 19/20 Total Training Loss: 0.011 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 17 Total Test Error = 14.420 Time/Epoch: 0.07 hrs [INFO] Epoch: 20/20 Total Training Loss: 0.014 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 17 Total Test Error = 14.420 Time/Epoch: 0.07 hrs

The training arguments are: python -u main.py --epochs 20 --batch 12 --model PSMNet --lambd 0.1 --eps 0.5 --lr 0.001 --maxdisp 192 --savemodel ./checkpoints/PSMNet/ --cuda_group 2 --verbose

Thank you for your help.

leonidas-timefly commented 1 year ago

Hi @leonidas-timefly, thank you for your interest in our work.

May I know if this issue only occurs with the ETH dataset, or if it also exists in other benchmarks (e.g. KITTI and Middlebury)?

I would suggest checking if you have loaded your pre-trained weights correctly. Also, check the data preprocessing pipeline and make sure it is consistent across datasets.

Hi @waychin-weiqin ，

I would like to confirm what the pre-trained weights mean here. Does it mean that the training of ITSA on SceneFlow is based on a pre-trained model (trained on SceneFlow or other datasets?)?

Thanks for your clarification.

waychin-weiqin commented 1 year ago

Hi again, in regards to the Middlebury dataset, you can find the Full, Half and Quarter resolution datasets using this URL.

Hi @waychin-weiqin ,

Sorry to trouble you. I checked the training process and data processing, and found no problem. Still, I obtain strange results. I want to make sure if the training loss is normal? This is the training log of PSMNet on SceneFlow:

[INFO] BEST epoch: 1 Total Test Error = 56.606 Time/Epoch: 0.07 hrs [INFO] Epoch: 1/20 Total Training Loss: 0.046 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 1 Total Test Error = 56.644 Time/Epoch: 0.08 hrs [INFO] Epoch: 2/20 Total Training Loss: 0.035 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 2 Total Test Error = 46.876 Time/Epoch: 0.08 hrs [INFO] Epoch: 3/20 Total Training Loss: 0.023 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 3 Total Test Error = 33.355 Time/Epoch: 0.08 hrs [INFO] Epoch: 4/20 Total Training Loss: 0.021 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 4 Total Test Error = 30.693 Time/Epoch: 0.07 hrs [INFO] Epoch: 5/20 Total Training Loss: 0.018 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 4 Total Test Error = 30.693 Time/Epoch: 0.08 hrs [INFO] Epoch: 6/20 Total Training Loss: 0.015 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 6 Total Test Error = 29.578 Time/Epoch: 0.08 hrs [INFO] Epoch: 7/20 Total Training Loss: 0.025 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 6 Total Test Error = 29.578 Time/Epoch: 0.08 hrs [INFO] Epoch: 8/20 Total Training Loss: 0.016 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 8 Total Test Error = 28.171 Time/Epoch: 0.07 hrs [INFO] Epoch: 9/20 Total Training Loss: 0.013 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 9 Total Test Error = 25.306 Time/Epoch: 0.08 hrs [INFO] Epoch: 10/20 Total Training Loss: 0.012 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 10 Total Test Error = 25.045 Time/Epoch: 0.08 hrs [INFO] Epoch: 11/20 Total Training Loss: 0.014 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 11 Total Test Error = 22.326 Time/Epoch: 0.08 hrs [INFO] Epoch: 12/20 Total Training Loss: 0.016 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 12 Total Test Error = 20.686 Time/Epoch: 0.08 hrs [INFO] Epoch: 13/20 Total Training Loss: 0.018 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 13 Total Test Error = 18.278 Time/Epoch: 0.08 hrs [INFO] Epoch: 14/20 Total Training Loss: 0.013 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 14 Total Test Error = 16.742 Time/Epoch: 0.07 hrs [INFO] Epoch: 15/20 Total Training Loss: 0.012 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 15 Total Test Error = 15.593 Time/Epoch: 0.08 hrs [INFO] Epoch: 16/20 Total Training Loss: 0.016 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 16 Total Test Error = 14.866 Time/Epoch: 0.07 hrs [INFO] Epoch: 17/20 Total Training Loss: 0.012 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 17 Total Test Error = 14.420 Time/Epoch: 0.08 hrs [INFO] Epoch: 18/20 Total Training Loss: 0.011 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 17 Total Test Error = 14.420 Time/Epoch: 0.08 hrs [INFO] Epoch: 19/20 Total Training Loss: 0.011 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 17 Total Test Error = 14.420 Time/Epoch: 0.07 hrs [INFO] Epoch: 20/20 Total Training Loss: 0.014 Time/Epoch: 0.00 hrs [INFO] BEST epoch: 17 Total Test Error = 14.420 Time/Epoch: 0.07 hrs

The training arguments are: python -u main.py --epochs 20 --batch 12 --model PSMNet --lambd 0.1 --eps 0.5 --lr 0.001 --maxdisp 192 --savemodel ./checkpoints/PSMNet/ --cuda_group 2 --verbose

Thank you for your help.

Hi, I have rerun the experiment on my machine but I am not facing the issue you are having. Could you check if your training dataset includes all SceneFlow data (from FlyingThings3D, Driving and Monkaa scenes)?

Sorry I wish I could be more helpful on this

waychin-weiqin commented 1 year ago

Hi @leonidas-timefly, thank you for your interest in our work. May I know if this issue only occurs with the ETH dataset, or if it also exists in other benchmarks (e.g. KITTI and Middlebury)? I would suggest checking if you have loaded your pre-trained weights correctly. Also, check the data preprocessing pipeline and make sure it is consistent across datasets.

Hi @waychin-weiqin ，

I would like to confirm what the pre-trained weights mean here. Does it mean that the training of ITSA on SceneFlow is based on a pre-trained model (trained on SceneFlow or other datasets?)?

Thanks for your clarification.

Sorry, that was a typo. What I meant was your trained model (model trained on SceneFlow).

leonidas-timefly commented 1 year ago

Hi @leonidas-timefly, thank you for your interest in our work. May I know if this issue only occurs with the ETH dataset, or if it also exists in other benchmarks (e.g. KITTI and Middlebury)? I would suggest checking if you have loaded your pre-trained weights correctly. Also, check the data preprocessing pipeline and make sure it is consistent across datasets.

Hi @waychin-weiqin ， I would like to confirm what the pre-trained weights mean here. Does it mean that the training of ITSA on SceneFlow is based on a pre-trained model (trained on SceneFlow or other datasets?)? Thanks for your clarification.

Sorry, that was a typo. What I meant was your trained model (model trained on SceneFlow).

Hi @waychin-weiqin ,

Thanks for your clarification. I'm quite confused because I tested two pipelines while obtaining some conflicting results: The first pipeline is that I downloaded your code, organized the SceneFlow dataset, and then directly trained the model with ITSA; The second pipeline is that I downloaded your code, organized the SceneFlow dataset, downloaded the pre-training weights from https://github.com/JiaRenChang/PSMNet, https://github.com/xy-guo/GwcNet, and https://github.com/gallenszl/CFNet: Then I loaded these weights into the model and trained these models with ITSA.

In fact, the second pipeline ensured that the training data and data processing were correct. So I want to confirm which pipeline you adopted. Thanks for your information.

leonidas-timefly commented 1 year ago

Hi @waychin-weiqin ,

I think I've found the core of this issue:

You remained a "break" in line 271 of main.py, which means that ITSA only trains one iteration for each epoch. In this way, ITSA is under-fitted. I believe that this is caused by carelessness. By the way, there are some other little bugs while I'm running ITSA. For example, there should be a "featEsts = None" in the "else" (line 200) condition. Hope you can examine the uploaded code twice. Thank you.

Thanks for your contribution and help again.