search

deepset-ai / FARM

:house_with_garden: Fast & easy transfer learning for NLP. Harvesting language models for the industry. Focus on Question Answering.
https://farm.deepset.ai
Apache License 2.0
1.73k stars 247 forks source link

Add Big Bird to FARM #790

Closed tanmaylaud closed 3 years ago

tanmaylaud commented 3 years ago

requires latest version of transformers v4.7.0.dev0

Timoeller commented 3 years ago

Nice one, thanks @tanmaylaud

We are planning a FARM release mid next week. Would be nice if we can get this in, but I would prefer not including the dev transformers version. Already testing current master in #791

tanmaylaud commented 3 years ago

Nice one, thanks @tanmaylaud

We are planning a FARM release mid next week. Would be nice if we can get this in, but I would prefer not including the dev transformers version. Already testing current master in #791

Yes, may be we can merge it in a later release.

julian-risch commented 3 years ago

Hi @tanmaylaud FARM is now upgraded to transformers=4.7.0 and we are about to merge your Big Bird PR 🎉 Did you do some testing, e.g., running the doc classification example with a Big Bird model? Edit: I ran the example. See below.

tanmaylaud commented 3 years ago

Regarding testing, We did a dry run (training) on SQUAD: https://public-mlflow.deepset.ai/#/experiments/2/runs/40a887dac3e24e75b891a0b050aba3dc

In order to fully inspect the performance of bigbird, we need larger passages, which are not available in SQUAD.