Add more virtual product examples (or even a bunch of virtual products)

[MatthewJA]

I found it really hard to make a virtual product that did what I wanted it to do. The examples listed in the relevant notebook work, and are great, but deviating from them a little can break with very cryptic errors. I'd love either more examples, or possibly just a collection of ready-to-go virtual products (e.g. combined Sentinel 2, masked LS8...).

[robbibt]

Yes, this would be really cool. For now I think the examples should live here so we have them all in one nice central catalogue:

https://github.com/GeoscienceAustralia/dea-notebooks/blob/develop/Supplementary_data/Virtual_products/virtual-product-catalog.yaml

These ones would be particularly worth focusing on I think, as they could potentially be used for real (vs. demo) analyses:

• Combined Landsat 5, 7 and 8
• Combined Landsat 5, 7 and 8, cloud masked
• Combined Sentinel-2A and 2B
• Combined Sentinel-2A and 2B, cloud masked
• Combined Landsat 5, 7, 8, Sentinel-2A and Sentinel-2B
• Combined Landsat 5, 7, 8, Sentinel-2A and Sentinel-2B, cloud masked

[robbibt]

I'd also love to have some extra documentation/comments in that YAML that separate the examples into the "demo" style recipes and the analysis-style recipes. I guess alternatively we could have one YAML called virtual-product-examples.yaml for the ones that already exist, and virtual-product-catalog.yaml for the more complex analysis-style ones?

[MatthewJA]

I'd kinda like multiple YAML files - a monolithic YAML file is hard to navigate and understand, as I discovered when I tried to learn about virtual products the other day.

[robbibt]

Yep OK, that's fair enough. I guess the question is how granular we want to go e.g. broken down to individual products/product suites? (e.g. surface reflectance virtual products, WOFS virtual products etc)? I don't mind having a few if they're nicely named and live in an easy to navigate to location like that "Virtual_products" folder

[robbibt]

I have a Landsat 5, 7 and 8 combined virtual product available here that could be handy to add: https://gist.github.com/robbibt/5e1bf4e63d1f429ca2ee078227801dfc

And some handy extra documentation: https://datacube-core.readthedocs.io/en/latest/dev/api/virtual-products.html

Also pinging @uchchwhash who probably has some thoughts on this stuff too

[uchchwhash]

What can I do to help? :smile:

First of all, @MatthewJA I would be interested to know more about the cryptic error messages that you got. I think improving the user experience there is something I can work on.

The ready-to-go virtual products were on the agenda, then the agenda sort of moved on :disappointed:

The only reason we had a monolithic catalog was to encourage YAML re-use. If more people are interested in using virtual products, it is probably better to group relevant virtual products into separate YAMLs. The thing to keep in mind is that you cannot share a piece of YAML between separate YAML documents.

I would be glad to work with you guys to develop a suite of sensible virtual products.

[MatthewJA]

@uchchwhash I didn't save the error messages - whoops! Perhaps I can try naively making some virtual products and see what issues I hit, as a user who "gets" the general point but doesn't know the details.

[uchchwhash]

Yes, @MatthewJA please unlearn some of it for the sake of future users! :smile:

[robbibt]

Here's an updated draft YAML for a combined Landsat 5, 7 and 8 virtual product, including a raw data product (ls_nbart), a normalised product with values between 0 and 1.0 (ls_nbart_norm) and finally a cloud/contiguity masked version (ls_nbart_masked):

https://gist.github.com/robbibt/5e1bf4e63d1f429ca2ee078227801dfc

(thanks for all the help @uchchwhash!)

[MatthewJA]

https://gist.github.com/MatthewJA/e754ab7ed6c58f582e6fe9ea57e699d5

Here's the MNDWI + WOfS YAML I pulled out of #641. A lot messier but it shows how to do this kind of index.

[robbibt]

Not a priority at this stage, closing for now