Experiments for ESIP Presentation

[rajadain]

ESIP has a meeting in July 19-22 https://www.esipfed.org/meetings

Prepare a demo of our work for this meeting.

[lewfish]

A high-level plan for getting something presentable at ESIP. We should aim to get this all working locally first on a tiny subset of the dataset if possible.

• [x] Pick a query that Fernando suggested, and make a concrete instantiation of it. To start, we should pick a time range that either uses the reanalysis OR the prediction dataset.
• [x] Lewis / Terence: Write abstract. See #49
• [x] Vijay: Save an extract of NHD to S3 to avoid needing database. See #48
• [x] (Lewis) Rechunk Zarr-encoded data and run benchmarks. See #66
  • Time the query using the Zarr-encoded data that is on S3.
  • (Not sure if we should do this) Time the query using the naive approach, ie. download NetCDF files and do the calculation. This will be slow because we are downloading huge amounts of irrelevant data. I think we want to do this as a baseline, but not sure.
  • Rechunk the data and save it on S3. Not sure, but I think doing it for the whole dataset is going to require a huge amount of resources. We should do it for a large, but manageable subset of the dataset. It needs to be large enough to make the benchmarks "realistic." This corresponds to https://github.com/azavea/noaa-hydro-data/issues/26.
  • Time the query using the rechunked Zarr-encoded data. This corresponds to https://github.com/azavea/noaa-hydro-data/issues/27.
• [x] (Terence / Vijay): Reformat data in Parquet and benchmark. See #67
  • Re-encode the data using Parquet for point-based datasets (ie. the ones representing stream reaches). Try different ways of chunking it. This corresponds to https://github.com/azavea/noaa-hydro-data/issues/28. Code in https://github.com/azavea/noaa-hydro-data/blob/master/src/notebooks/save_nwm_sample.ipynb might be a useful starting point.
  • Time the query using the Parquet-encoded data.
• [x] Repeat the queries using different numbers of workers to test scalability.
• [x] Repeat the queries using (interesting) variations on the query or another query that Fernando suggested.
• [x] Display the timings for the different approaches in a table or graph

[lewfish]

Hey @vlulla I’m using ZenHub in GitHub, click this link to join my workspace and see other features available in GitHub or download the ZenHub extension and sign up with your GitHub account. Posted using ZenHub

[lewfish]

Note that we should keep experiments small at first and use them to estimate how much the full set of experiments will cost and pass that on Stephanie.

[lewfish]

Required reading: Why is Parquet/Pandas faster than Zarr/Xarray here?