search

azavea / noaa-hydro-data

NOAA Phase 2 Hydrological Data Processing
11 stars 3 forks source link

Add notebooks for start of ESIP experiments #71

Closed lewfish closed 2 years ago

lewfish commented 2 years ago

This PR adds some notebooks to get started on the experiments for ESIP. This includes notebooks to: get the COMIDs for a HUC2 to use for testing (save_huc2_comids.ipynb), save a subset of NWM and rechunk it (rechunk_zarr_subset.ipynb), and time different queries and save the results to a CSV (huc8_streamflow_query.ipynb). These notebooks all run locally and generate files on S3. All of this needs to be scaled up in later PRs. Two additional related changes that came up along the way: some changes to be able to run scripts on a multi-CPU instance on Batch, and changes to the Docker image to support running natively on ARM64 architectures (ie. for those of us with Macs).

Connects #66

lewfish commented 2 years ago

I had to make one additional change to get it to make an ARM image on M1:

Thanks, good catch!