search

deepset-ai / haystack

:mag: AI orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots.
https://haystack.deepset.ai
Apache License 2.0
17.63k stars 1.91k forks source link

Yes/No question QA model #993

Closed vladgets closed 7 months ago

vladgets commented 3 years ago

Do you have example/pretrained model to support Yes/No boolean questions?

tholor commented 3 years ago

Hey @vladgets ,

Interesting question and definitely a valuable feature in QA pipelines - especially for conversational use cases!

We don't have a pre-trained model for yes/no questions yet. We added some basic functionality in FARM a while ago to train such models on the natural questions dataset. You can find an example script here. The idea there was to have two prediction heads - one for extractive QA and one for classifying yes/no. If you are interested in training such a model yourself, maybe give this a try.

We also have the support for yes/no on the roadmap of Haystack. What's missing? Well, we were not 100% happy with the inference part and answer format of the existing FARM implementation. Our plan is to refactor this in FARM before we make it available in Haystack (and publish models).

In general, we are also following the research activities for this feature quite closely. If you are considering training your own model, maybe this dataset could also be interesting as NQ doesn't contain too many yes/no samples: https://www.aclweb.org/anthology/N18-1074/

Happy to hear also your thoughts on this feature and any modeling/experiments you maybe already did yourself!

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 21 days if no further activity occurs.

shahrukhx01 commented 3 years ago

Hey @tholor, this seems like an interesting add-on to Haystack, I'd really like to play around in doing the modeling part of this. Would the end goal of this be to have a binary or three-class classifier? Could you please give me some more context for this? Before I dive into the dataset paper itself.

shahrukhx01 commented 3 years ago

There are already 5 pre-trained models by Facebook research on the model hub on this type of task. https://huggingface.co/models?pipeline_tag=text-classification&sort=downloads&search=anli

More details: https://github.com/facebookresearch/anli

tholor commented 3 years ago

@shahrukhx01 The idea here is not to have a full, stand-alone transformer that answers with yes/no given a premise + hypothesis, but rather to extend an existing reader with a small, additional prediction head that allows the model to decide whether to spit out a "span answer", "yes" or "no". This is rather a complex issue and to manage expectations: the main work here is to figure out a very clean, lean implementation and modeling approach. We won't have the time to guide you here closely in the next weeks.

A simpler alternative might be to train a generator (like RAG) on a dataset that includes more "yes/no" answers. This is probably less dependent on deep FARM/Haystack integrations but can also easily take a couple of weeks work.

Whoever is going to kick-off work here: We should also do a quick literature review about other recent approaches here. Papers addressing the binary questions in Natural Questions might be a good starting point.

shahrukhx01 commented 3 years ago

@tholor I see, thanks for your response. There is one dataset by Google that is explicitly annotated for this task. Anyone who takes this task later on, following resources could be a good starting point for binary QA: Papers: https://arxiv.org/pdf/1905.10044.pdf => BoolQ https://arxiv.org/abs/1909.06146 => PubMedQA https://arxiv.org/pdf/2010.02605.pdf => DaNetQA

Talk: BoolQ talk: https://vimeo.com/356079637

tvaisanen commented 2 years ago

Hi, has there been any progress on this?

We also have the support for yes/no on the roadmap of Haystack. What's missing? Well, we were not 100% happy with the inference part and answer format of the existing FARM implementation. Our plan is to refactor this in FARM before we make it available in Haystack (and publish models). @tholor

tholor commented 2 years ago

Hey @tvaisanen, no we haven't kicked off work on this as we saw higher priorities for other tasks and not much demand from the community. What's your use case for it?

tvaisanen commented 2 years ago

I'm researching into potential solutions on how to extract classification results from FARM reader results. These are often questions that require an answer in yes/no/na format. It seems that currently the best approach would be to use a pre trained classifier on the FARM result set and do some fine tuning?

tholor commented 2 years ago

Currently: Yes, that's probably the simplest and most pragmatic option. Also quite flexible, as you can run this in combination with any of the existing reader models out there. Biggest downside: you "waste" quite some compute when running a second transformer "just for classification".

Future: The reader could get a second, optional prediction head for classification (as stated above). We did a basic implementation a while ago. However, the implementation details are nasty and we'd need a way to make this second head really an optional one that can get easily attached to an existing reader (and trained without sacrificing performance on the QA head).

masci commented 7 months ago

Closing as stale