Currently, there is no loader to scrape the full site. Instead, users need to scrape all the links themselves. With this loader, users can specify an entry point and a prefix to scrape and store all texts within a site.
Type of Change
[x] New Loader/Tool
How Has This Been Tested?
tried modifying the download repo link and tried downloading it and tested basic functionality:
Suggested Checklist:
[x] I have added a library.json file if a new loader/tool was added
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I ran make format; make lint to appease the lint gods
Description
Currently, there is no loader to scrape the full site. Instead, users need to scrape all the links themselves. With this loader, users can specify an entry point and a prefix to scrape and store all texts within a site.
Type of Change
How Has This Been Tested?
tried modifying the download repo link and tried downloading it and tested basic functionality:
Suggested Checklist:
make format; make lintto appease the lint gods