reproduce test result using checkpoint

[gyjung975]

Hello. I downloaded checkpoint file for S3DIS (best.pth) from #27 and ran test.sh. I did not change any config or code but I got result like this..

Val result: mIoU/mAcc/allAcc 0.4647/0.5019/0.7736

Isn't it the best model for 72.6 IoU?

Here is my log.. test-best.log

[Gofinge]

Hi, I checked and compared the released test log and your log. It seems that I trained the model with a voxel size of 0.05m. I think you can reproduce the result after aligning the two configs. Thanks for pointing out the miss.

[gyjung975]

Thank you for your answer!

I have one more thing. I'm trying to reproduce result of PTv2 on S3DIS. Dataset loop is 30, but there is no difference in test mIoU between 10 and 30, even though 30 takes 3 times longer.. Is there any reason for 30 loops?

[Gofinge]

For the testing process, the data loop is 1. Consequently, there is no difference during the test. Loop 30 and eval epoch 100 mean actual epoch 3000.

[gyjung975]

Oh my question wan't clear. It means difference of test mIoU between model trained on 1000 epochs and 3000 epochs! i.e., loop 10 vs 30

My result of 3 trials each

10 loop : 70.4 ~ 72.0 30 loop : 70.3 ~ 71.5

[Gofinge]

Okay, there might not be much difference because I did not carefully tune the parameters.

Btw, recently I officially released our codebase Pointcept. For S3DIS, I try to render a normal vector as additional input, and an example config trained with SpUNet seem has a better training curve. Maybe you could try this.

[gyjung975]

Thank you. I will try those.

[Carl-Yang98]

Hi, I downloaded the checkpoint and got the same result. Did you reproduce the new result? If you do, could you give me some suggestions about how to modify the configs.

[gyjung975]

@Carl-Yang98 No, I didn't. So I trained from scratch with the same configs. The best was 72.0, but oftem not even 70 after several attempt...

[Carl-Yang98]

Thank you. I will try to train it.

[Carl-Yang98]

Sorry to bother you again! I met the new questions. I trained from scrath with the same configs and got the best miou is:0.69. But when I ran the test.py, I got the crazy result. The miou was close to zero. Did you met this?

[gyjung975]

No I've never met. Even an untrained model does not yield zero.. I think it might be problem of IoU calculation.

[Carl-Yang98]

Thanks for your reply. I will check it. But I didn't change the code about IoU calculation. Is it possible that it's an overfit? I just changed the size of batchsize and numwork, but I think it should not have such a big impact on the test result, so it bothers me