STAC Index is planned to crawl all collections from STAC static catalogs and APIs.
We plan to use PySTAC for it as it allows migrating from 0.8 and 0.9 to 1.0 with ease, validates data and it's planned to give us an easy way to get all collections after https://github.com/stac-utils/pystac/issues/169 has been implemented.
This also requires us to migrate to MongoDB, which is mostly compatible to nedb, but will need some minor changes (e.g. check timestamps, check case-insensitive sort, add schema, ...)
Don't crawl all items for APIs, instead use the API to query for specific data, but still crawl static catalogs.
Don't get an item per catalog, but an item per (1) root catalog and (2) per collection. Fewer items for a first run. All remaining items can be crawled later.
STAC Index is planned to crawl all collections from STAC static catalogs and APIs.
We plan to use PySTAC for it as it allows migrating from 0.8 and 0.9 to 1.0 with ease, validates data and it's planned to give us an easy way to get all collections after https://github.com/stac-utils/pystac/issues/169 has been implemented.
This also requires us to migrate to MongoDB, which is mostly compatible to nedb, but will need some minor changes (e.g. check timestamps, check case-insensitive sort, add schema, ...)