Timoeller
commented
3 years ago Hey @SasikiranJ that is indeed a good question (I reformulated the issue title accordingly). The ability of Question Answering Models to generalize to out of domain data (your custom documents) is an ongoing research topic. For an up to date reference please have a look at the EMNLP 20202 paper
To give you some condensed guidance:
- In general Language Models used for Question Answering are very powerful and intelligent. They "understand" the question and context documents. To give a few examples of the capabilities: The model has knowledge about similar terms (understands synonyms) and words in context (the apple that falls from the tree is different to the apple that needs a software update) and relates questions starting with "who" to answers that contain a name of a person.
- Sometimes the model just memorizes training examples and applies the connection of question and answer to "unseen" test examples.
- Out of domain questions and texts can be very similar to the data the model was trained on - so it only seems the model can generalize.
How good the model generalizes to your custom documents, you best annotate a couple of question answer pairs and see how the model performs.
Question Hi, I am little bit unclear about how reader producing answer from given documents? I am using a model which was trained on squad2 dataset. I want to know how model is producing answer from custom documents for different question without even being trained on my data? I am just passing my own documents and passing question to the model. Please let me know how transfer learning helping me here. Thank you in advance.
Additional context Add any other context or screenshots about the question (optional).