Enhancement Proposal: Native support for virtual products in OWS.

[SpacemanPaul]

Allow products in product layers to be structured virtual product definitions as well as simple product names.

E.g. using the cloud-masked LS7/LS8 example in Virtual Product Documentation https://datacube-core.readthedocs.io/en/latest/dev/api/virtual-products.html

{
    "name": "vprod_example",
    "title": "Example Virtual Product Layer",
    ...
    "product": {
         "collate": [
               {
                    "transform": "apply_mask",
                    "mask_measurement_name": "pixelquality",
                    "input": {
                           "juxtapose": [
                                 {
                                       "product": "ls7_nbar_albers",
                                       "measurements": ["red", "green", "blue"]
                                 },
                                 {
                                       "transform": "make_mask",
                                       "input": {
                                              "product": "ls7_pq_albers",
                                       },
                                       "flags": {
                                            "blue_saturated": False,
                                            "cloud_acca": "no_cloud",
                                            "cloud_fmask": "no_cloud",
                                            "cloud_shadow_acca": "no_cloud_shadow",
                                            "cloud_shadow_fmask": "no_cloud_shadow",
                                            "contiguous": True,
                                            "green_saturated": False,
                                            "nir_saturated": False,
                                            "red_saturated": False,
                                            "swir1_saturated": False,
                                            "swir2_saturated": False,
                                       },
                                       "mask_measurement_name": "pixelquality
                                 }
                           ]
                    }
               },
               {
                      "transform": "apply_mask",
                      "mask_measurement_name": "pixelquality",
                      "input": {
                             "juxtapose": [
                                 {
                                       "product": "ls8_nbar_albers",
                                       "measurements": ["red", "green", "blue"]
                                 },
                                 {
                                       "transform": "make_mask",
                                       "input": {
                                              "product": "ls8_pq_albers",
                                       },
                                       "flags": {
                                            "blue_saturated": False,
                                            "cloud_acca": "no_cloud",
                                            "cloud_fmask": "no_cloud",
                                            "cloud_shadow_acca": "no_cloud_shadow",
                                            "cloud_shadow_fmask": "no_cloud_shadow",
                                            "contiguous": True,
                                            "green_saturated": False,
                                            "nir_saturated": False,
                                            "red_saturated": False,
                                            "swir1_saturated": False,
                                            "swir2_saturated": False,
                                       },
                                       "mask_measurement_name": "pixelquality"
                                 }
                           ]
                     }
               }
         ]
   },
   # ... rest of layer configuration.
}

Note that in this scenario a number of existing OWS features can be replicated directly with Virtual Products. (e.g. multi-products, masking, band aliases, some of the band math, in styles).

Ideally redundant features will be gradually either deprecated and removed from the datacube_ows codebase entirely, or reimplemented using virtual products. (unless it turns out the OWS implementation is more efficient than the Virtual Product one, in which case we'll probably try and get the OWS implementation into virtual products!)

[whatnick]

Consolidation of functionality is a good thing. We should maintain the code with feature flags to activate one or the other implementation while we are in the twilight zone. My main concern in using multi-products via virtual products is the OWS table for extents of these and how that needs to be updated given datacube-core has no DB level concepts of space-time.

[SpacemanPaul]

My main concern in using multi-products via virtual products is the OWS table for extents of these and how that needs to be updated given datacube-core has no DB level concepts of space-time.

There are a few complicating issues surrounding the MVs actually. Maybe full native support of virtual products is too ambitious until after ODC 2.0 adds proper spatio-temporal extent handling.

[uchchwhash]

@SpacemanPaul this looks awesome! Let's do this! :100:

[SpacemanPaul]

The MV's currently serve three purposes in OWS land:

1) Spatio-temporal dataset queries - similar to ODC's search_eager or query methods, but considerably more efficient due to PostGIS indexes and optimisations. 2) Generating polygons for query extents - used to draw the over-resource-limit polygons. 3) Rapid and efficient population of the range tables (update_ranges) which cache product-wide extents for generating GetCapabilities documents.

The virtual product API as it stands only supports 1) from this list (without the "faster than core" efficiency gains). Unfortunately this makes the direct approach outlined above untenable as matters stand until ODC 2.0 and proper postgis indexing of spatiotemporal extents in core. This should remain our long term plan though.

[SpacemanPaul]

User band math (and the accompanying "expression" syntax) uses the same expression parser as virtual products