Closed li-ziang closed 2 years ago
Hi @li-ziang, you can find the exact commands that we've used for RTE and all other SuperGLUE tasks here: https://github.com/timoschick/pet/issues/19#issuecomment-747483960
One reason for your different results may be that we did not use auxiliary language modeling (i.e., we didn't set --lm_training
) because with that, we ran into memory issues for some tasks. Let me know if you're unable to reproduce our results with the commands linked above.
Hi @li-ziang, you can find the exact commands that we've used for RTE and all other SuperGLUE tasks here: #19 (comment) One reason for your different results may be that we did not use auxiliary language modeling (i.e., we didn't set
--lm_training
) because with that, we ran into memory issues for some tasks. Let me know if you're unable to reproduce our results with the commands linked above.
Many thanks! @timoschick , I will have a try.
Hi @timoschick, I have tried the command for RTE in https://github.com/timoschick/pet/issues/19#issuecomment-747483960, and I got the following output:
2021-09-12 10:28:52,495 - INFO - modeling - --- RESULT (pattern_id=0, iteration=0) ---
2021-09-12 10:28:52,495 - INFO - modeling - {'acc': 0.7075812274368231}
2021-09-12 10:28:53,692 - INFO - modeling - === OVERALL RESULTS ===
2021-09-12 10:28:53,692 - INFO - modeling - acc-p0: 0.7075812274368231 +- 0
2021-09-12 10:28:53,692 - INFO - modeling - acc-all-p: 0.7075812274368231 +- 0
It seems that this is the correct result, thanks again!
Great, thanks for letting me know :)
Hi, I am trying to reproduce the result of the RTE task. I followed the method in https://github.com/timoschick/pet/issues/4#issuecomment-707031187 and get the following output in my terminal.
Since I used the
--do_eval
argument, the result should be that on dev set, but the result shown in Table1 in your paper is 69.8 (mine is 70.8). Did I use the correct training arguments? And here is my command for trainingIf I want to reproduce the results of the other 7 taks in Table1 in your paper, can I simply change the
--task_name
argument?THANKS IN ADVANCE!