Partitioning using Healpix: the onion space

[JulienPeloton]

Here is the current idea for partitioning the space.

[JulienPeloton]

The equivalent of Envelope in JTS will be Sphere here. The difference between 2 concentric spheres will define the elements of the grid (from the partition point of view). In a sense, we will have a onion space!

[JulienPeloton]

With the current implementation, we have:

// Initialise our space:
// z_min = 0.0, z_max = 1.0, with splits every dZ = 0.1.
val partitioning = new OnionPartitioning
partitioning.LinearOnionPartitioning(0.0, 1.0, 0.1)

// Grab the grid elements
val grids = partitioning.getGrids

// Build our partitioner
val partitioner = new OnionPartitioner(GridType.LINEARONIONGRID, grids)

// Draw a point and place it on our grid
val p = new Point3D(0.0, 0.0, 0.45)
val iterator = partitioner.placeObject(p)

[JulienPeloton]

Still plenty of TODOs, but things are moving forward (see https://github.com/JulienPeloton/spark3D/pull/12). Random thoughts:

• Make the partitioning based on a number of partition (either given by the user, or guess from the underlying RDD) instead of specifying manually the bounds. That would unify the interface with GeoSpark and avoid memory issue (imagine: all points could be in one partition if the points are clustered in one place + few outliers far away...)
• Use a pattern matching to trigger the partitioning based on the GridType value given by the user.
• Add method to pixelised each shell (using Healpix for example).
• ....?

[ChrisArnault]

Z-Shells may be ordered

• linear
• log
• according to a UDF (= Density model)
• consider the "last-shell"

[JulienPeloton]

https://github.com/JulienPeloton/spark3D/issues/25