search

developmentseed / eoapi-risk

An implementation of eoAPI for Risk data and the Humanitarian sector
MIT License
2 stars 0 forks source link

Create Heat Exposure Forecast notebook #49

Closed wrynearson closed 6 months ago

wrynearson commented 6 months ago

What this does

This notebook attempts to show what an automated heat exposure analysis and visualization could look like in a Jupyter Notebook. While not using cloud-native formats, it does show the utility of clear, reproducible analysis, especially within a JupyterHub environment.

Limitations

Many. I'm not a climate scientist or a geospatial analyst. There are a few issues that should be addressed:

  1. The way I join the forecast and population data. I use a point forecast in a hex population bin. There's surely a better way...
  2. The ipywidget for forecast step in the visualization. This should be a single variable, not a range.
  3. The way I'm loading and saving data could probably be improved.

How to review

  1. See if this works locally. I have not tested it in a JupyterHub environment
  2. Improve the code if you see anything wrong 😄
review-notebook-app[bot] commented 6 months ago

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

wrynearson commented 6 months ago

Rebased to update the afg-earthquake-22-vector notebook that was merged last week with https://github.com/developmentseed/eoapi-risk/pull/48

zacdezgeo commented 6 months ago

@wrynearson, your work is truly impressive! The notebook is a concrete example of assessing 'Who is impacted where', and you've utilized some excellent tools from the open-source world. The narrative and presentation are clear and well-established, making this a great demo that we could share with others. I've just sent an invite to John on this repo so we can get his feedback.

A few edits I made that might provide useful feedback. I'd recommend just re-reading through the new notebook to see these changes @wrynearson:

One point we should discuss is how to prioritize data integrations into eoAPI. A lot of the notebook takes care of the extraction (from ETL) of diverse sources. If we decide this is a core notebook to demonstrate, we may want to see how we could integrate a STAC of the ECMWF data (or at least the used sample) and, similarly, for the population data. @geohacker @batpad; It would be great to see how this fits our current roadmap...

@wildintellect @abarciauskas-bgse @sharkinsspatial, maybe you have a more cloud-optimized and STAC cataloged data reference for 2m temperature forecasts? Maybe carbon plan's CMIP6 dataset or something on the VEDA side?

Again, great job @wrynearson, for pushing this use case forward 🥳

batpad commented 6 months ago

Hmm I'm not sure we're going to be able to solve the issues with the grib driver for xarray with just a pip install. It required me to do a system level (brew on mac) install of the eccodes library. So we might need something installed on the base image when running this in the hub.

I think this is okay for now, but I do think most likely this notebook is not going to run on a remote hub unless the base image being used has eccodes installed. Currently we are using this image: quay.io/jupyter-remote-desktop-proxy/qgis to be able to get the Desktop GIS.

Let me chat with @sunu of how best to approach this - we may want to include the Dockerfile for the base image we want in the hub in this repository so it's easier to add dependencies, etc. but that is a bit of over-head work. Let's definitely not treat this as a blocker to merging this, but something we should look at resolving.

wildintellect commented 6 months ago

@batpad @zacharyDez the grib package likely has non-python dependencies. This is often where conda excels. Note in other projects that use Xarray we recommend building off the Pangeo docker containers since most batteries are included already for this kind of work.

I've confirmed on VEDA Hub that Pangeo includes this.

(notebook) jovyan@jupyter-wildintellect:~$ conda list eccodes
# packages in environment at /srv/conda/envs/notebook:
#
# Name                    Version                   Build  Channel
eccodes                   2.34.1               he84ddb8_0    conda-forge
python-eccodes            1.7.0           py311h1f0f07a_1    conda-forge

I agree the other approach is to include the non-python dependencies in a custom docker.