danielfleischer
commented
4 months ago If you didn't change the top-K parameter, it still retrieves the same number of documents (default is 10). The bottleneck is probably the FiD model (encoder-decoder) and it is highly sensitive to the prompt size, like all transformer models. One thing to try is having smaller text snippets as the "documents", letting the retriever + ranker find the best matches. Try splitting the texts based on sentences to reduce the length.
Although the example you've provided "simple_odqa_pipeline.ipynb' works pretty fine on the example list you've provided. I changed that list to a list of 23 elements, each with a pretty long string. It's taking over 6.5 minutes to generate the result.
It's infeasible at runtime to incorporate this overhead. Any solution for this? Or did you try that?