I wanted to reproduce the results from your paper on the sentiment analysis task. I followed all the steps you list and then ran the notebook for DBRD. I found two unclarities that I hope you could clarify:
the notebook uses v1, and it is not clear whether the same notebook should be used to reproduce results for v2;
the Config that is used in the notebook is not the same as the one in the repo: in the repo, the config has gradient_accumulation_steps = 8. However, in the notebook output cells I can see that you originally ran this with gradient_accumulation_steps = 1. I could only reproduce your results when I changed the notebook so that config.gradient_accumulation_steps = 1, while running with 4 GPUs.
Maybe these things can be clarified/made more consistent in the repo?
In the end I was able to reproduce your results. My results are a bit lower than than the ones you report in this repository, but they are within the confidence interval that you described in the paper so thanks for including that CI!
Hi there
I wanted to reproduce the results from your paper on the sentiment analysis task. I followed all the steps you list and then ran the notebook for DBRD. I found two unclarities that I hope you could clarify:
gradient_accumulation_steps = 8. However, in the notebook output cells I can see that you originally ran this withgradient_accumulation_steps = 1. I could only reproduce your results when I changed the notebook so thatconfig.gradient_accumulation_steps = 1, while running with 4 GPUs.Maybe these things can be clarified/made more consistent in the repo?
In the end I was able to reproduce your results. My results are a bit lower than than the ones you report in this repository, but they are within the confidence interval that you described in the paper so thanks for including that CI!