search

deepset-ai / haystack

AI orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots.
https://haystack.deepset.ai
Apache License 2.0
17.72k stars 1.92k forks source link

support for long answers #975

Closed kruthikakr closed 3 years ago

kruthikakr commented 3 years ago

Hi

Haystack is for granular answers. Q: Which parameter determines the length of the answer ? Q: can we extend to obtain the long answers say in case getting the definitions in general long answers.

I have a reference to FinBert where it gives out long answers - https://github.com/yuanbit/FinBERT-QA . Q: How to use this similar FinBERT model here.

Also i am trying to use this model google/bigbird-base-trivia-itc. getting error as not correct model.

Timoeller commented 3 years ago

Hi, I hope that haystack can do more than "granular answers" :smile:

Granular Answers

I think what you mean is "Extractive QA Models trained on short answers" are giving short granular answers.

So if you want to have longer answer or even answer passages please just use the retriever, either BM25 within Elasticsearch or a DPR based retriever, see Tutorial 6 Better retrieveal with DPR for this.

Longer answers within QA Model

Bigbird as model type is not supported in FARM and thereby FARMReader yet unfortunately. As far as I understand BiBird it is a model for long contexts, not explicitly for giving long answers. Also the TriviaQA dataset contains mostly short answers, so a model trained on this dataset is primed to extract short answers.

Please try out using only the DPR retriever, this might be what you want in the end.

kruthikakr commented 3 years ago

@Timoeller Thank you for quick reply. I will try this. and model preference for extractive QA trained on long answers from deepset or huggingface ?

Is the pdf to text convertor handles the page layout ? say image and tables. Any referees to look for it ?

kruthikakr commented 3 years ago

How to convert - https://github.com/yuanbit/FinBERT-QA model in Haystack ?

tholor commented 3 years ago

How to convert - https://github.com/yuanbit/FinBERT-QA model in Haystack ?

From a quick look it seems like this is a re-ranker rather than an extractive QA model. We are planning to add support for re-rankers soon (#423 ). We can ping you once it is available :)

Timoeller commented 3 years ago

we have the reranking PR coming up in #1025

aqsa27 commented 3 years ago

Does haystack now support long answers? If so, can some one help me with which model or retriever should I use?

kruthikakr commented 3 years ago

Hi, I hope that haystack can do more than "granular answers" smile

Granular Answers

I think what you mean is "Extractive QA Models trained on short answers" are giving short granular answers.

So if you want to have longer answer or even answer passages please just use the retriever, either BM25 within Elasticsearch or a DPR based retriever, see Tutorial 6 Better retrieveal with DPR for this.

Longer answers within QA Model

Bigbird as model type is not supported in FARM and thereby FARMReader yet unfortunately. As far as I understand BiBird it is a model for long contexts, not explicitly for giving long answers. Also the TriviaQA dataset contains mostly short answers, so a model trained on this dataset is primed to extract short answers.

Please try out using only the DPR retriever, this might be what you want in the end.

Here again can you please tell me how to download and load the model for this query_embedding_model="facebook/dpr-question_encoder-single-nq-base", passage_embedding_model="facebook/dpr-ctx_encoder-single-nq-base",

I am downloading the model locally using git clone https://huggingface.co/facebook/dpr-question_encoder-single-nq-base and giving the path

Timoeller commented 3 years ago

Sure, using DPR would be an option for having longer answers. You would actually just get a text passage where the answer should be located.

Do you have firewall constraints or why dont you load the model by just initializing the DPR as shown in our tutorial like:

from haystack.retriever.dense import DensePassageRetriever
retriever = DensePassageRetriever(document_store=document_store,
                                  query_embedding_model="facebook/dpr-question_encoder-single-nq-base",
                                  passage_embedding_model="facebook/dpr-ctx_encoder-single-nq-base",
                                  max_seq_len_query=64,
                                  max_seq_len_passage=256,
                                  batch_size=16,
                                  use_gpu=True,
                                  embed_title=True,
                                  use_fast_tokenizers=True)

If you should encounter bugs related to loading DPR, please open a separate issue with detailed error messages. Thanks.

Btw can we close this issue than?

kruthikakr commented 3 years ago

yes,Thank you