Closed sugiyama-hiroaki closed 6 years ago
Any updates @sugiyama-hiroaki ? I also trying to reproduce RACE results.
It works! I had a simple bug on converting problems to tensor format.
How much accuracy did you get on RACE @sugiyama-hiroaki ? I am getting 53.
I had 57.2. This is slightly lower than the paper score, but I think it's a reasonable level. In our experiment, it depends on the batch size. We got 57.2 with 32 batch size and 54.8 with 4 batch size.
I am trying but due to the low number of GPU, I cannot reproduce the results. I would be very grateful to you if you can provide me pre-trained weights. @sugiyama-hiroaki
Can you share the code for RACE ? @sugiyama-hiroaki
I'm so sorry but I cannot share my weights because I rewrote the code for adapting to RACE. Besides, I'm a company researcher so I don't have a permission to share my code.
If you don't have enough number of GPUs, you can train the model with skipping the update function until it reaches the batch size.
@sugiyama-hiroaki Thanks for the tips! I'm also trying to reimplement for adapting to RACE, but I can't get higher than 35% acc on dev & test set and 58% acc on training set. I use batch_size=4 and all the others are left as default. Could you please tell me what hyperparameters did you choose for training the model?
@sugiyama-hiroaki Can you tell whether the test set scores were fluctuating a lot or stable (like uniformly going up or a smooth pattern) with respect to the epochs. Also can you tell the number of epochs you trained the model for?
Hi,
We tried several settings based on your code and paper, but unfortunately we cannot reproduce the RACE score (the training loss decreases, but the dev accuracy only reaches 0.26). Could you tell me some tips about the parameter or code modification for achieving the performance?
Thanks