Closed ruiqi-zhong closed 2 years ago
Hey Riqui, How strongly are you tied to this dataset? We can provide the predictions for a larger dataset (https://arxiv.org/pdf/2204.07705.pdf) and a larger model, if that works better.
For v1, we have all prediction files saved in AI2 beaker, I can dig in the specific files relevant to Table 5 and share it in the next few days if that's ok.
I was mainly a bit surprised by the fact that fine-tuned BART (a smaller model) can outperform GPT-3 by a large margin, so I want to compare the model predictions head-to-head to get more intuitions.
So I guess any smaller model vs. GPT-3 would help my purpose, and the predictions from the new paper would be great.
Thanks a lot!
(I also skimmed through your new paper. It looks super coool!!! I've been waiting for this paper for a while and I finally get to see what are the results.)
Thanks! Would you love to hear your feedback on the work!
In summary, @swarooprm will share the predictions of BART in the next couple of days. @yizhongw, let's plan to share the predictions for the v2 dataset's experiments as we move toward finalizing them.
Sorry for the delay @ruiqi-zhong; here are the predictions from @yizhongw 's models. I have reminded @swarooprm several times for pulling out his model predictions, though I suspect that Yizhong's predictions are more relevant (and hence, informative) than the BART prediction.
Thanks a lot for sharing the model predictions!! @danyaljj @yizhongw It's really useful for my research.
Hey! Would it be possible to share BART's predictions as well (in particular the ones used to generate Table 5)? The repo currently only contains a small subset of it, it seems. I would love to look at the model predictions and get some intuitions about them.
Thanks!!