Issues/249: Testing Apache Spark on DC2

JulienPeloton commented 6 years ago

This PR adds two Notebooks:

Introduce Apache Spark and test performance of data manipulations of the static coadd catalogs
Detailed benchmarks on Parquet and FITS format.

Remarks:

Apache Spark is fast (O(1) sec for most of the operations).
Results using Parquet are much faster than FITS without cache. Once data cached, performances become more similar.
There will be a repo Spark in LSSTDESC soon to introduce Spark at NERSC, and provide environment for DESC members. For the moment, the first Notebook has some information on how to quickly set up a working Spark env for JupyterLab at NERSC. Let me know if it is unclear.

See also #249

JulienPeloton commented 6 years ago

I do not have write access to this repo - so feel free to assign reviewers and labels.

wmwv commented 6 years ago

(I didn't realize you needed write access to request a reviewer?)

JulienPeloton commented 6 years ago

Note that I changed the instructions at the beginning of the 1st notebook to use the desc-spark kernel. This is a kernel combining desc-python and Spark environment.

JulienPeloton commented 6 years ago

Any follow-up on this one?

JulienPeloton commented 5 years ago

@wmwv @plaszczy I have re-ran the Spark notebooks with the desc-pyspark kernel (see LSSTDESC/jupyter-kernels). If you have a spare minute, could you have a look at it?

plaszczy commented 5 years ago

nice

LSSTDESC / DC2-production

Issues/249: Testing Apache Spark on DC2 #288