Advance Data Cubes

[bermud]

Several activities exist related to Data Cubes

• Geoscience Australian Data Cubes - approach for managing and analyzing for the purpose of improving access to vast quantities of satellite imagery and other Earth observations to provide information on issues that can affect populations.
• The Open Data Cube (ODC) initiative developed by the Committee on Earth Observation Satellites (CEOS) seeks to provide a data architecture solution that provides value to global Earth observation (EO) satellite data users value to its global users while increasing the impact of EO satellite data.
• The Earth System Data Cube ESDC developed to help improve simultaneously exploring of multiple EO data streams.
• Earth Server Data Cube provides access to more than 110 TB of landsat data.

Work to be advanced

• Develop a common Data Cube Model
• Provision of Data Cube examples with both regular and irregular grids
• Advance a rich semantic Data Cube - provides best practice to combination data cube model and controlled vocabularies
• Advance Data Cube implementation in Discrete Global Grids (DGGS)
• Advance uses of of Data Cube Models for other (non raster) data (e.g. in-situ, social media, vector data)
• Advance storage on the cloud that better support interfaces for Data Cube data
• Advance standards-based access using common interface (e..g WCS 2.0 and WMS 1.3) for Data Cubes. Requests should demonstrate space/time subsetting; result encoding in some user-selected format; reproduction on the fly; ad-hoc WCPS queries facilitating data processing and fusion, with an emphasis on time series processing and extraction.
• Advance client implementations
• Advance Data Cube access and discovery on data hubs (NextGEOSS DataHub, Australia Sentinel DataHub)
• Advance federated processing of Data Cubes queries, like "A overlay B" where object A is residing in data center DA and B in data center DB.

[pebau]

Trying to provide some prose here: Datacubes form an enabling paradigm for serving massive spatio-temporal Earth data in an analysis-ready way. Extending the concept of seamless maps from 2D into nD by including height and time (and maybe more axes) presents users (and m2m clients alike) with a small number of homogenized objects, thereby easing access, extraction, analysis, and fusion substantially - "one cube says more than a million images". For server-side evaluation of datacube requests, a bundle of enabling techniques is known which can massively speed up processing, including adaptive partitioning, parallel and distributed processing, dynamic orchestration of heterogeneous hardware, and even federations of data centers. Today, known datacube services exceed 500 TB, and datacube analytics queries have been split across 1,000+ cloud nodes.

From a standards perspective, datacubes belong to the category of coverages; the coverage data model is represented by OGC Coverage Implementation Schema (CIS), the service model by OGC Web Coverage Service (WCS) together with its OGC Web Coverage Processing Service (WCPS), OGC's geo datacube query language.

In Testbed-14, 3D x/y/t and 4D x/y/z/t spatio-temporal datacubes, exceeding 1 TB to convincingly demonstrate scalability, should be served via WCS and WCPS. The following features should be available and demonstrated:

• provision of datacube examples with both regular and irregular grids
• standards-based access through WCS 2, possibly additionally WMS 1.3
• standards-based ingestion relying on OGC WCS-T
• requests should demonstrate space/time subsetting; result encoding in some user-selected format; reprojection on the fly; ad-hoc WCPS queries facilitating datacube processing and fusion, with an emphasis on timeseries processing and extraction.
• federated processing of datacube queries, like "A overlay B" where object A is residing in data center DA and B in data center DB, resp.

This service functionality should be demonstrated through a wide range of clients (via WCS as the client/server API, maybe additionally WMS) facilitating visual navigation (eg, OpenLayers), Web GIS integration (eg, QGIS, ArcGIS), virtual globe display (eg, NASA WorldWind, Cesium), and programmatic analytics (eg, python, R).

[pebau]

...plus, there should be application/sponsor specific scenarios, of course. Candidates that come to my mind:

• federated processing between archives from NASA, ESA, and NCI Australia
• Worldbank scenarios (they elegantly show the need for data fusion)
• tieing in existing large-scale datacube services, such as eodataservice.org

[tomLandry]

For application/sponsor specific scenarios, climate data would fit the bill too. NetCDF files served through OPeNDAP are effectively data cubes too. Having a uniform way to query, subset and regrid climate model outputs and EO data would simplify the infrastructure required to serve both. This is a use case that CRIM envision inside Earth System Grid Federation (ESGF). Reference: Geospatial Fusion in the Era of Big Data, George Percivall (OGC), IGARSS, July 24th 2017.

[pebau]

Actually, in an earlier testbed a WCS interface has been established for OPeNDAP.

[pebau]

FWIW, in Testbed-15 a datacube federation between EU and US has been established, with complete location transparency, based on rasdaman (also used in EarthServer where ultimately 2.5+ PB of satellite and climate data have been datacubed).