search

allenai / natural-instructions-v1

Benchmarking Generalization to New Tasks from Natural Language Instructions
https://arxiv.org/abs/2104.08773
Apache License 2.0
25 stars 0 forks source link

Sharing BART predictions #6

Closed ruiqi-zhong closed 2 years ago

ruiqi-zhong commented 2 years ago

Hey! Would it be possible to share BART's predictions as well (in particular the ones used to generate Table 5)? The repo currently only contains a small subset of it, it seems. I would love to look at the model predictions and get some intuitions about them.

Thanks!!

danyaljj commented 2 years ago

Hey Riqui, How strongly are you tied to this dataset? We can provide the predictions for a larger dataset (https://arxiv.org/pdf/2204.07705.pdf) and a larger model, if that works better.

swarooprm commented 2 years ago

For v1, we have all prediction files saved in AI2 beaker, I can dig in the specific files relevant to Table 5 and share it in the next few days if that's ok.

ruiqi-zhong commented 2 years ago

I was mainly a bit surprised by the fact that fine-tuned BART (a smaller model) can outperform GPT-3 by a large margin, so I want to compare the model predictions head-to-head to get more intuitions.

So I guess any smaller model vs. GPT-3 would help my purpose, and the predictions from the new paper would be great.

Thanks a lot!

(I also skimmed through your new paper. It looks super coool!!! I've been waiting for this paper for a while and I finally get to see what are the results.)

danyaljj commented 2 years ago

Thanks! Would you love to hear your feedback on the work!

In summary, @swarooprm will share the predictions of BART in the next couple of days. @yizhongw, let's plan to share the predictions for the v2 dataset's experiments as we move toward finalizing them.

danyaljj commented 2 years ago

Sorry for the delay @ruiqi-zhong; here are the predictions from @yizhongw 's models. I have reminded @swarooprm several times for pulling out his model predictions, though I suspect that Yizhong's predictions are more relevant (and hence, informative) than the BART prediction.

ruiqi-zhong commented 2 years ago

Thanks a lot for sharing the model predictions!! @danyaljj @yizhongw It's really useful for my research.