search

lm-sys / FastChat

An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
Apache License 2.0
36.63k stars 4.52k forks source link

Recommendation for close-book QA dataset #1442

Closed ghost closed 11 months ago

ghost commented 1 year ago

Thank you for providing the wonderful repo.

The performance of fastchat-t5-3b is surprisingly good. I am applying it for close-book QA (answering questions using the context provided). I see a huge improvement over flan-t5.

However, I am not able to use it for commercial due to the ShareGPT dataset. I'm thinking about finetuning the flan-t5 model myself.

I'm struggling to find open-sourced close-book QA dataset. Do you have any recommendations? I believe SquadV2 is part of the dataset that flan-t5 was trained on, but it's extractive QA. Generative/abstractive QA is what I need, so the model can write entire sentences, rather than just chunk of words extracted from the context.

Any help would be much appreciated. Thank you.

surak commented 11 months ago

@jasontian6666 did you manage to find one? How did you go further on the QA dataset? Mind sharing it with us?

I'm closing this one for now, but I would be happy to know about your results, so if you have something to tell, please reopen this one, ok? Thanks!