search

deepset-ai / COVID-QA

API & Webapp to answer questions about COVID-19. Using NLP (Question Answering) and trusted data sources.
Apache License 2.0
344 stars 121 forks source link

Preprocessing of context to fit max_length #176

Open Geethi2020 opened 3 years ago

Geethi2020 commented 3 years ago

Hi, would you please help me understand how the preprocessing is done for theCovidQA corpus ? Why I ask is because the context in the CovidQA dataset seems to be so much larger than the maximum length set in the code (which is 300+ and BERT max_length is 512 tokens). How is the data processed to fit into the limit ? Couldn't find the code for that in the Git. Please advice. Thank you.

Timoeller commented 3 years ago

The data is in normal SQuAD format so you can use processors like the ones in Huggingface transformers or our FARM framework, see https://github.com/deepset-ai/FARM/blob/master/farm/data_handler/processor.py#L1889 to convert the data.

For dealing with long context you have moving windows, compute possible answers per window and combine the answer afterwords. Hope that helps? If you have detailed questions about the processing, please ask in FARM or huggingface transformers directly