search

Clay-foundation / model

The Clay Foundation Model (in development)
https://clay-foundation.github.io/model/
Apache License 2.0
242 stars 25 forks source link

First try for v1 pipeline #180

Closed yellowcap closed 3 months ago

yellowcap commented 3 months ago

This is a proof of concept for #178. This is kind of a generic approach that is solely based on STAC, so we could spin this out into a separate pipeline repository and make it standallone for it to be used also in other projects.

Its functional but a bit clunky, but it does show how this could work.

To test it locally:

Create a working directory with an /items subdirectory.

  1. In there, run create_stac.sh, this should create local static STAC items
  2. Then run edit_stac.py this will change default source paths for Landsat to S3 locations.
  3. Then run indexing.py this should create an index json file that represents the items and the chips in them
  4. Then finally the chipping.py script should print chips that are dynamically created from the index.
yellowcap commented 3 months ago

I have been working more on the v1 pipeline, realizing that we will need cloud and nodata tracking at the chip level. I played around with creating classes where the platform specific cloud and nodata filters can be added through class method overrides.

the classes are at

https://github.com/Clay-foundation/model/blob/clay-v1-pipeline/scripts/pipeline_v1/chip_indexer.py

script to try them out with the data attached below is

https://github.com/Clay-foundation/model/blob/clay-v1-pipeline/scripts/pipeline_v1/run_chip_indexer.py

test_stac_items.zip

yellowcap commented 3 months ago

This has migrated to https://github.com/Clay-foundation/stacchip