How robust is the crawler in the sense of what can it crawl?
As long as there is html it should work?
And what is the "storage" limit, can I let it crawl the official python docs? Limit might not be on the crawlers side but the LLMs' I plug the .json file in?
First of all, sounds really cool!
How robust is the crawler in the sense of what can it crawl? As long as there is html it should work? And what is the "storage" limit, can I let it crawl the official python docs? Limit might not be on the crawlers side but the LLMs' I plug the .json file in?
Cheers