Open m-mohr opened 1 year ago
Maybe re-use the crawler code from stac-index, which might include a "sleep" time between requests, a queue for retries and sampling just a number of items instead of reading them all.
cc @DanielJDufour
It can at least work on full folders now. This is not really crawling though and may look into JSON files that are not STAC.
Like stac-validator, it would be great to specify a STAC entity and traverse from it through all links and validate them.