Reformat data in Parquet and benchmark

azavea / noaa-hydro-data

NOAA Phase 2 Hydrological Data Processing

11 stars 3 forks source link

Closed lewfish closed 2 years ago

lewfish commented 2 years ago

Re-encode the data using Parquet for point-based datasets (ie. the ones representing stream reaches). Try different ways of chunking it. This corresponds to https://github.com/azavea/noaa-hydro-data/issues/28. Code in https://github.com/azavea/noaa-hydro-data/blob/master/src/notebooks/save_nwm_sample.ipynb might be a useful starting point.
Time the query using the Parquet-encoded data.

This is a piece of #45

lewfish commented 2 years ago

This should roughly follow the workflow for Zarr in rechunk_zarr_subset.ipynb and huc8_streamflow_query.ipynb introduced in #71