search

deepset-ai / FARM

:house_with_garden: Fast & easy transfer learning for NLP. Harvesting language models for the industry. Focus on Question Answering.
https://farm.deepset.ai
Apache License 2.0
1.73k stars 247 forks source link

Retreiver Fine Tuning : Are language models like roberta, gpt2 supported to use in retreiver? #852

Closed sumanthp closed 2 years ago

sumanthp commented 2 years ago

Question Are fine tuned language models like roberta, gpt2 are supported to used in haystack retriever?

Additional context I am trying to fine tune transformer language models - roberta using masked language modelling and gpt2 using casual language modelling. When I try to integrate these in haystack embedding retriever, errors are returned. For roberta - mlm part of codebert is currently not supported in Haystack. For Gpt2 - Could not infer Language model class from config.

Could you please help me out if I am trying it in correct way?

Timoeller commented 2 years ago

Could you please open the issue in haystack next time?

Bert based retrievers should be finetuned on supervised data (e.g. containing querys and matching document pairs). Finetuning them with MLM doesnt help so much. But if you still want to try, please finetune a normal BERT model (not roberta) and try it out.

For GPT-2 this is different, this model is currently not supported as retriever. How would you extract the embeddings from the GPT-2 model?

sumanthp commented 2 years ago

Thanks Timoeller for the reply. Sorry, I have opened this issue in FARM repo by mistake.

Timoeller commented 2 years ago

No harm done, so no problem :D If you want to go deeper into finetuning retrievers please feel free to open another issue in haystack. Maybe focus on finetuning bert based retrievers (not GPT-2) as a start...