Models used to get SQuAD data mentioned in former issues

[Djokovic0311]

Hi! I am trying to apply other datasets to this py-irt model, and I noticed that your SQuAD dataset had been converted to jsonlines with fields 'subjects' and 'responses'. So could I ask which models were used to generate these responses? Or any models are okay as long as the format is the same as SQuAD you offered? Thanks in advance!

[jplalor]

Any models are ok, as long as the format is the same. For your models you'll just want to make sure that you have a unique "subject_id" that maps back to a particular model.

[Djokovic0311]

Thanks a lot for your reply!

[pk1130]

Hey @jplalor! Thanks for your response to this issue! We understand that any models can be used as long as the format is the same and there is a unique subject_id that maps to each model. For the purposes of our research, could we still get access to which particular models were used in the squad.jsonlines file that @EntilZha had linked to in an earlier issue? We are doing some IRT analysis on the SQuAD questions, and knowing which particular models were used in squad.jsonlines would be tremendously helpful to us :) Thanks a ton!

[EntilZha]

The code for the ACL paper is now open source @pk1130 https://github.com/facebookresearch/irt-leaderboard, I can take a closer look at your issue later, but is the issue that you'd like a correspondence between the model id and a readable name? If so, the data discussed on that github repo probably would help with that issue

[pk1130]

That's precisely our issue, @EntilZha. Thanks a ton for pointing me to a useful resource! I'll take a look at it now :)

[Djokovic0311]

Sure, we will have a close look at the repo you gave. Thanks for your kind help!

[jplalor]

Closing this. If you have additional questions let us know!