search

deepset-ai / haystack

AI orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots.
https://haystack.deepset.ai
Apache License 2.0
17.73k stars 1.92k forks source link

Evaluation of TransformersReader #732

Closed dg4271 closed 3 years ago

dg4271 commented 3 years ago

Question What is best way to evaluate TransformersReader (I'm working on other language, korean) I've tried referring tutorial 5. But I can't evaluate using TransformersReader cause there is no eval() function. There is any plan to implement it? or should i use FarmReader to do the evaluation?

Timoeller commented 3 years ago

Hey @dg4271 thanks for using haystack.

Indeed, please use the FARMReader for the evaluation for now. It should be able to load all QA models from the Huggingface modelhub - also a korean model like https://huggingface.co/monologg/koelectra-base-v3-finetuned-korquad. Or which model do you intend to use?

dg4271 commented 3 years ago

Thanks, @Timoeller

I am using exactly that model (https://huggingface.co/monologg/koelectra-base-v3-finetuned-korquad)

As a related question, if implemented as follows,

reader = FARMReader("monologg/koelectra-base-v3-finetuned-korquad", top_k_per_candidate=4)

I found warning message below.

01/14/2021 01:09:43 - WARNING - farm.modeling.language_model -
Could not automatically detect from language it is. We guess it's an ENGLISH model ... If not: Init the language model by supplying the 'language' param.

  1. There seems to be no parameter to specifying a language in FarmReader
  2. Do I have to specify the language? (The above warning seems to have removed in the latest Farm code.)
Timoeller commented 3 years ago

Do I have to specify the language? (The above warning seems to have removed in the latest Farm code.) Exactly, it was an artifact that is now removed from farm. Please ignore the warning.

Does haystack (and I guess an ElastisSearch retriever) work with the korean model? If you run benchmarks please report back here some results, would be highly interested in non/english QA / ODQA results.

dg4271 commented 3 years ago

Of course. I'll share the results.

Thank you for your confirmation.

Timoeller commented 3 years ago

Issue seems fixed. Please still post your Korean results once you get them here or reopen the issue if more problems occur.

LI3ARA commented 1 year ago

I want to do evaluation on TransformersReader as FARMReader gives

ImportError: Failed to import 'sentence_transformers'. Run 'pip install farm-haystack[inference]'. Original error: No module named 'sentence_transformers'

julian-risch commented 1 year ago

Hi @LI3ARA when you Run 'pip install farm-haystack[inference]' the error will go away. It will install sentence_transformers. You can find detailed installation instructions for Haystack in the documentation. If you come across any other problems, please open a new issue instead.

LI3ARA commented 1 year ago

yeah I tried 'pip install farm-haystack[inference] but no difference. ran !pip install -U sentence-transformers this also.

I am using a JupyterNotebook