echeipesh
commented
6 years ago My guess is thats going to be incredibly difficult to get this API even close to the right shape from the get-go. Instead it makes sense to tackle this feature one use case at a time and iterate on emerging API as we add complexity. One stumbling block for this approach in the past has been the concern with API stability in the master branch.
I would advocate to contain API stability concerns in the release branches (2.0 and 2.1) and merge iterative PRs more aggressively either to master or develop branch.
metasim
In order to work with multiple raster sources to produces a single output its required that they their pixel values are co-registered, that is they share:
Typically this requirement is handled by Extract Transform Load (ETL) step, which transforms input rasters to shared layout which defines above parameters. While this approach is valid and common it makes too many arbitrary decisions before starting the reading step.
The single ETL process should be broken up into four stages, which allow incremental refinement and modification of the process based on metadata.
A better approach is to query or read only the raster metadata and associate it with the source of raster cells in a
RasterRef(working name). This would allow performing, splitting, filtering and joining on the metadata records, reading the cells only when the shape and content of the final output known.The intermediate data structure represents something like virtual mosaic, like a GDAL VRT that can be further refined in Spark memory before being reified.
The expected stages to support this feature are:
Query
Identify potential rasters and optionally read their metadata if not given in query response.
Design Question
RasterRefto have user defined metadata values. Is it type parameterMor something else?Filter
Remove or modify the raster references based on metadata.
Split
Partition references to larger rasters into sets of possibly overlapping windows. In this step we may end up with records that reference different regions of the same file or table.
Windows may be overlapping if its desirable to read a buffer of pixels around a region, allowing that region to be correctly resampled or reprojected.
Join
Once the raster refs have been split into windows according to some
LayoutDefintionthey can be keyed and joined on those keys.The intent of the join is either to assemble different rasters into
MultibandTileor to select a subset of the sources based on metadata.Design Questions
LayoutDefinitionor is there some class can enforce this per result?Read
Reading the raster cells described by each window of the
RasterRefintoMultibandTile.Reading the raster cells has room for optimization. In the common case of GeoTiff we know the file has a segment structure and reading any bounding box will ultimately be expressed as reading segments.
When reading a single GeoTif file segment it may intersect multiple target windows. Knowing all of the target windows allows us to read the segment once and partially populate each target window until its complete.
The same logic extends to GeoTrellis Avro layers, which are tiled.
Design Questions
RasterRefand a set of windows?Transform
Once the
RasterRefshave been read intoRDD[(SpatialKey, MultibandTile)]orRDD[(SpatialKey, BufferedTile[MultibandTile])]we should be able to apply standard GeoTrellis operations.Design Question
MultibandTileactually good here? What if we joined two datasets of differing resolution?