GSoC: First steps...

[JulienPeloton]

The project will be divided in 3 parts:

• Find alternative to JTS (no 3D support): new dev or existing package?
• Upgrade 2DRDD to 3DRDD (circle -> sphere, parallelogram -> parallelepiped, and so on).
• Upgrade 2D methods for partitioning/join/query/indexing to 3D.

[JulienPeloton]

Week 1:

• A few shapes (Shape3D, Point3D, Sphere) are ready (see https://github.com/JulienPeloton/spark3D/pull/8 and https://github.com/JulienPeloton/spark3D/pull/9).
• One spatial partitioning of the space is there: linear onion space (see https://github.com/JulienPeloton/spark3D/issues/11 and https://github.com/JulienPeloton/spark3D/pull/12).
• One 3D RDD is available (see https://github.com/JulienPeloton/spark3D/pull/12).

A few TODO for week 2:

• Keep developing partitioners. TREE in 3D? @mayurdb ?
• Keep developing 3D RDD. I need to re-organize and make more general Shape3DRDD. Especially spatialPartitioning in Point3DRDD must be generalized and incorporated in Shape3DRDD.
• Remove rdd as input arg of partition[T<:Shape3D] and benchmark it.
• Need to add direct loading of CSV, FITS, etc. in addition to handling direct RDD[T] as it is done now.

[JulienPeloton]

Week 2:

• Shape3DRDD has been re-organized to be more generic with respect to partitioner (see https://github.com/JulienPeloton/spark3D/pull/15). Future partitioner would just have to provide an implementation of the partitioning, and the partition method in Shape3DRDD would do the job.
• Add healpix partitioning for shells in the onion partitioning (see https://github.com/JulienPeloton/spark3D/pull/15/commits/f800e8325b9c96b50c20c475952faa48e3a72269).
• Add FITS reader (see https://github.com/JulienPeloton/spark3D/pull/15).
• Add CSV reader and a file test (see https://github.com/JulienPeloton/spark3D/pull/18). a script to convert FITS to CSV has been added, to ease the testing.
• Add example and visualisation tools (see https://github.com/JulienPeloton/spark3D/pull/19). We provide a notebook example (Jupyter notebook + Toree kernel).
• ...?

A few TODO for week 3:

• Tree partitioning
• Join/query methods for the onion partitioning.
• ...?

[JulienPeloton]

Week 3:

• Add replacement for JTS (see https://github.com/JulienPeloton/spark3D/pull/22), which includes ShellEnvelope and BoxEnvelope. This also includes (https://github.com/JulienPeloton/spark3D/pull/23):
  • Force Point3D to declare the type of data, that is spherical (r, theta, phi) or cartesian. It was optional before.
  • Integrate the development made into the Onion partitioning. Now it uses Shells instead of Sphere to partition the space.
• Add mechanisms to perform the cross-match of 2 data sets. (see https://github.com/JulienPeloton/spark3D/pull/24)
• Add HOWTO notebooks.

A few TODO for week 4:

• Octree (see https://github.com/mayurdb/spark3D/commit/cfd20b3c1c3b1dee49f79ea5285fb0f64f8f4ff)
• Treatment of objects at the border of partitions (see https://github.com/JulienPeloton/spark3D/issues/25)
• Multi-resolution algorithm for cross-match (see https://github.com/JulienPeloton/spark3D/issues/26)

[JulienPeloton]

Week 4:

• Octree implementation (see #30).
• Add very simple Kryo registration for spark3D classes. Probably need more (see #31, #28).
• Major README updates, and v0.1.0 release (see #33).
• Add scripts to generate test data (see #34).
• A few fixes: #27, #29, #35.

A few TODO for week 5:

• Treatment of objects at the border of partitions (see #25)
• Multi-resolution algorithm for cross-match (see #26)
• Create Sphere3DRDD class to simplify the use of octree.
• Tackling large shuffle? #32
• Implement RangeQuery methods (to find neighbours).