search

harvardnlp / im2markup

Neural model for converting Image-to-Markup (by Yuntian Deng yuntiandeng.com)
https://im2markup.yuntiandeng.com
MIT License
1.19k stars 214 forks source link

Do nothing,but BLEU value increased by 3% #9

Closed zhangw-memo closed 5 years ago

zhangw-memo commented 6 years ago

HI! Recently I experimented with your model and found that the accuracy has improved by more than 3% compared to what you mentioned in the paper.I would like to to ask if you have modified the model, or you think there may be a problem. 2018-03-27 10:40:10,218 root INFO BLEU = 91.20, 96.9/94.0/91.3/88.6 (BP=0.984, ratio=0.985, hyp_len=537287, ref_len=545740) 2018-03-27 12:02:39,761 root INFO Accuracy (w spaces): 0.821012 2018-03-27 12:02:39,761 root INFO Accuracy (w/o spaces): 0.846854

da03 commented 6 years ago

Hmm thanks for repeating our experiments! I think that might due to random seeds etc, but anyway it's good to see our results are reproducible.

zhangw-memo commented 6 years ago

Thank you for your suggestions. In addition to the random numbers(910820) given in your code.which random numbers do you use? I would like to try to find out the real reason.

da03 commented 6 years ago

I'm not sure what's the exact reason of this increase, but there could be several sources of randomness: CNN is random since we use cudnn (https://github.com/soumith/cudnn.torch/issues/274), torch/cuda version etc.

zhangw-memo commented 6 years ago

before I was doing experiments, I discovered that cudnn 6.0 was not supported. It can only run on cudnn 5.1 as same as yours. so I think that the season is not cuda or cudnn. Now I am adjusting the random number to see if there is any effect on the experiment. Some of my classmates suggested that I should change a new computer, and try to see if there is any influence. The experiment is running. Anyway, thanks for your suggestion.

da03 commented 5 years ago

Closing, plz let us know if there are any other issues.

zhangwei2019 commented 5 years ago

I don't know. I tried to find out the problem, but I didn't find anything. while I reinstalled the system and downloaded the data from GitHub again, it was not a big deviation. It is not very clear whether this is related to my system configuration.