Building retrievers with multiple docs (web pages or files)

[zhiheng-huang]

Description

Support scraping pages under a given domain. As a result. we now support building retrievers with multiple webpages or documents.

make formatting All unit tests pass make test Verified the following toy experiments run okay. poetry run python experiments/index_and_query_from_docs.py poetry run python experiments/index_and_query_from_webpage.py

Related Issue

No

Type of Change

• [ ] 📚 Examples / docs / tutorials / dependencies update
• [ ] 🔧 Bug fix (non-breaking change which fixes an issue)
• [x] 🥂 Improvement (non-breaking change which improves an existing feature)
• [ ] 🚀 New feature (non-breaking change which adds functionality)
• [ ] 💥 Breaking change (fix or feature that would cause existing functionality to change)
• [ ] 🔐 Security fix

Checklist

• [x] I've read the CODE_OF_CONDUCT.md document.
• [x] I've read the CONTRIBUTING.md guide.
• [x] I've updated the code style using make codestyle.
• [ ] I've written tests for all new methods and classes that I created.
• [ ] I've written the docstring in Google format for all the methods and classes that I used.