Adding custom files with information to the search engine

[alelasantillan]

I've tried the search engine using your colab.ipynb and It worked perfect after several trials. It seemed that colab works better at some hours, whereas it fails when resources demand is high.

I am still wondering in which way can I add to your ParlAI_SearchEngine a custom directory with text files so the relevant documents can be found also in that directory. This way one can add custom information to the web search. Any idea?

[JulesGM]

This will not work, as the GoogleSearch search module queries the Google webpage, and parses the results from the resulting webpage. I don't see how you can add documents specifically for that. Maybe you can with either Google's or Bing's cloud services.

You could publish your documents on the web, and have Google search the webpage you published them on specifically with the site: search operator.

You could combine the results of the two by deriving the handler's class, and combining the results from the search with the site: operator and without.

[JulesGM]

In the paper they have hybrid dense retrieval + search engine solutions if I remember correctly. Maybe you can use that one to use the dense retriever on your documents.

[alelasantillan]

In the paper they have hybrid dense retrieval + search engine solutions if I remember correctly. Maybe you can use that one to use the dense retriever on your documents.

Do you refer to this paper: "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks"? Or Internet-Augmented Dialogue Generation?

[JulesGM]

Internet-Augmented Dialogue Generation