GeoscienceAustralia / digitalearthau

Code and tools for Digital Earth Australia (a deployment of Open Data Cube)

https://geoscienceaustralia.github.io/digitalearthau/

31 stars 21 forks source link

Import & update the collection 3 product definitions #231

Closed jeremyh closed 5 years ago

jeremyh commented 5 years ago

Imported from @Kirill888 's originals in dea-proto with git history intact.

Made changes based on the feedback register:

Added simple aliases for red/green/blue etc for nbart.
Expanded the product descriptions (A little. Maybe people will want more?)
There's now multiple metadata types (and a minimal default 'eo3') so that products wont have a large number of confusing null fields. (original feedback was for Level1, but it will affect all non-ARD products)
Added aliases without oa_ prefixes
Fixed invalid nodata values (uint8 -> -999)
Added the new naming conventions for products *.odc-product.yaml

Misc changes

Removed unnecessary indexes (for platform/instrument/etc), which don't differ within a product.
Ran a ~ruamel whitespace~ headless pycharm autoformat on the whole history.

Still outstanding:

~nodata value review~ (✓done)
a second/fmask cloud field (needs to be pushed upstream to the datasets first if we're doing it)

codecov[bot] commented 5 years ago

Codecov Report

Merging #231 into develop will increase coverage by 0.12%. The diff coverage is 96.66%.

@@             Coverage Diff             @@
##           develop     #231      +/-   ##
===========================================
+ Coverage     66.8%   66.92%   +0.12%     
===========================================
  Files           42       42              
  Lines         3190     3202      +12     
===========================================
+ Hits          2131     2143      +12     
  Misses        1059     1059

Impacted Files	Coverage Δ
digitalearthau/system.py	`88.7% <96.66%> (+2.7%)`	:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 1860ba1...e23930d. Read the comment docs.

jeremyh commented 5 years ago

Products now validate against the sample data:

$ eo3-validate --thorough /g/data/v10/agdc/jez/digitalearthau/digitalearthau/config/eo3/products/*.yaml `fd odc-metadata.yaml`
✓ ard_ls5
✓ ard_ls7
✓ ard_ls8
✓ l1_ls5
✓ l1_ls7
✓ l1_ls8
✓ ga_ls5t_ard_3-0-0_092084_2009-12-17_final.odc-metadata
    I different_nodata oa_combined_terrain_shadow nodata: product 255 != dataset None
✓ ga_ls7e_ard_3-0-0_092084_2016-08-23_final.odc-metadata
    I different_nodata oa_combined_terrain_shadow nodata: product 255 != dataset None
✓ ga_ls8c_ard_3-0-0_092084_2016-06-28_final.odc-metadata
    I different_nodata oa_combined_terrain_shadow nodata: product 255 != dataset None

valid: 3 infos

The combined_terrain_shadow dataset nodata is None according to @sixy6e as it never has any nodata values, and 255 will be used in the product as the ODC filll value.

(Because of the fill-value/different usage of these fields, I've changed the validator to allow nodata values to differ, only printing an info)

jeremyh commented 5 years ago

(in keeping the Git history of this branch clean I've made this issue history a mess :slightly_smiling_face: )