Instability of results when tested individually after model training.

[xiaozheng-liu]

Hi yang, thank you for your excellent work！ I have some questions when debugging the code. There may be a problem with my reproduction steps. I hope you can help me. We noticed that you integrated the training and testing code into one file and optimized the code structure. We can get better results in the test set at the end of training, but when I retest using the trained model weights, the effect is significantly reduced. As you can see from the screenshot, the best result during training is 0.596, and the result when tested alone is 0.437, and each test will produce different results. What is the reason for this? We hope to receive your reply, thank you. Thanks again for your excellent work!

[xiaozheng-liu]

Supplement: We only commented out the training part of the traning.py file

[mooncakehub]

so if we use the code ,we just can train,but we can't test?

[xiaozheng-liu]

so if we use the code ,we just can train,but we can't test?

Thanks for your reply. As I described above, testing is done at the end of the training phase and the results are good. However, when the weights are loaded separately for testing, the previous test results cannot be reproduced. And the results of each test are different. We don’t know exactly what the reason is. Have you ever encountered this problem? We hope you can answer it.

[xiaozheng-liu]

so if we use the code ,we just can train,but we can't test?

Sorry, I mistakenly thought you were the author. To answer your question, yes, I was unable to reproduce the results during individual testing.

[mooncakehub]

I am interested in this code too,but if this code is incomplete, it is difficult for us to take advantage of it

[mooncakehub]

Because I don't have enough memory on my single gpu, I can't reproduce the code right now

[mooncakehub]

Because I don't have enough memory on my single gpu, I can't reproduce the code right now

[xiaozheng-liu]

I am interested in this code too,but if this code is incomplete, it is difficult for us to take advantage of it

Yes, I can't be sure what the reason is, and I very much hope the author can help.

[xiaozheng-liu]

Because I don't have enough memory on my single gpu, I can't reproduce the code right now

The code does not require much GPU memory, I only need about 5.4 G GPU memory during training.

[xiaozheng-liu]

Because I don't have enough memory on my single gpu, I can't reproduce the code right now

The code does not require much GPU memory, I only need about 5.4 G GPU memory during training.

What is your graphics card model?

[yangyangyang127]

so if we use the code ,we just can train,but we can't test?

Thanks for your reply. As I described above, testing is done at the end of the training phase and the results are good. However, when the weights are loaded separately for testing, the previous test results cannot be reproduced. And the results of each test are different. We don’t know exactly what the reason is. Have you ever encountered this problem? We hope you can answer it.

Sorry for the inconvenience the code caused. I have encountered issues with unstable results. But I do not know the problem you described. I will solve this problem in the next version.

[xiaozheng-liu]

so if we use the code ,we just can train,but we can't test?

Thanks for your reply. As I described above, testing is done at the end of the training phase and the results are good. However, when the weights are loaded separately for testing, the previous test results cannot be reproduced. And the results of each test are different. We don’t know exactly what the reason is. Have you ever encountered this problem? We hope you can answer it.

Sorry for the inconvenience the code caused. I have encountered issues with unstable results. But I do not know the problem you described. I will solve this problem in the next version.

Thank you very much for your answer and we look forward to your improvements to the project！