Cannot reproduce the self-train results

[liwell24]

Thanks you for the great work of open source！

I tried to reproduce the results of self-train on 3dfront to s3dis with 8 Tesla V100 s , but the mIoU is only 47.52 % , nearly 8 % behind the results of the paper. In order to ensure no errors ,the environment of experiment is same as yours, the pre-trained model used is what you provide，and I didn' t make any modifictaions. Could you tell me the skills of training the model ？

Then，I wonder the meaning of DODA(w/o TACM) , DODA(w/o TACM) should be a pre-trained model, generate pseudo labels on the target domain, and then self-supervised directly to optimize the parameters and test the results. Am I understanding this right ？

[Dingry]

Hi, we self-train the model exactly follow the config. Can you provide your enviroment and training logs so that we can figure out what is the problem?

I think you are right. DODA (W/o TACM) means NO TACM module in the self-training process compared to the full method.

[Dingry]

In addition, if you use spconv1.2, make sure that you install it following PointGroup modifications. It modifies spconv\spconv\functional.py to make grad_output contiguous. We are sorry that we miss this in the installment guidance and we will clarify it soon.

[liwell24]

Thank you very much for your quick reply!

My environment is as follows： python 3.7.13 pytorch1.5.1 spconv1.2.1 My training logs is https://github.com/liwell24/testRepository/blob/main/DODA_selftrain_log_train_20221014-142800, I am sorry that previous log forgot to save, this one is running now. I seem to have forgotten to modify spconv\spconv\functional.py, I'll try again. Thanks again for the reminder ！

[liwell24]

Hi~

I have tried self-train on 3dfront to s3dis with modifying spconv\spconv\functional.py. The mIoU is 51.22%, which is better than the previous 47.52%， but still 4% worse than the result of DODA(55.54%). The link of my training logs is https://github.com/liwell24/testRepository/blob/main/DODA_selftrain_log_train_20221015-102626.

Could you help me figure out the problem if it's convenient for you ？

[Dingry]

Hi, I have checked your log, and it is weird that the evaluation performance after the first 2 epochs was 6% lower than the pretrained model. Nomally, the performance keeps rising in the first several training epochs. I am re-trying the self-training following the default config. The performance reaches about ~55% in first 30 epochs. Here is my log:https://github.com/CVMI-Lab/DODA/blob/dev/docs/log_train_20221015-171509.txt. May I ask if you use the preprocessed S3DIS or process it on your own? Besides, is the gradient contiguous issue completely addressed (modify 4 times in the functional.py) in your spconv?

[liwell24]

Because you have provided the preprocessed S3DIS, so I downloaded and used it directly. Yes , I have modified 4 times in the functional.py in spconv . I found that your log doesn't generate pseudo-labels, but my log does. Is that the reason for the difference?

[liwell24]

Hi, I took a closer look at your training log, there is no step of generating pseudo-labels in your log, but my training log has a process for generating pseudo-labels. Could I ask how you set it up ？This may have something to do with my poor results.

[Dingry]

Hi, we check the log and find we load the previously generated pseudo labels since we have already generated it previously. And we try to re-generate pseudo labels during self-training today, the performance drops like yours. If the pseudo labels are already generated and we just load it in the self-training stage, the performance backs to normal. You can try this way for self-training first. It seems that some bugs existing and we will check and fix it later. Sorry for our mistakes and sincerely thanks for your careful attention!

[Dingry]

Hi, we have found the bugs. We forgot to re-set the data downsampling ratio after pseudo label generation. Now it is fixed. Thanks again for your attention.

[liwell24]

Okay, I see, thank you very much！