stac-utils / stac-index

A service that lists all available and registered STAC catalogs and APIs.
https://stacindex.org
Apache License 2.0
7 stars 4 forks source link

Add Crawler #1

Open m-mohr opened 3 years ago

m-mohr commented 3 years ago

STAC Index is planned to crawl all collections from STAC static catalogs and APIs.

We plan to use PySTAC for it as it allows migrating from 0.8 and 0.9 to 1.0 with ease, validates data and it's planned to give us an easy way to get all collections after https://github.com/stac-utils/pystac/issues/169 has been implemented.

This also requires us to migrate to MongoDB, which is mostly compatible to nedb, but will need some minor changes (e.g. check timestamps, check case-insensitive sort, add schema, ...)

m-mohr commented 3 years ago

Some ideas for faster crawling: