enable bokeh / holoviews / geoviews in pangeo.pydata.org

[rabernat]

Next Monday we will be giving a demo to some NASA folks of pangeo.pydata.org. I would like to show some example of visualization with holoviews. But, as pointed out in #144, the bokeh jupyterlab extension evidently doesn't work. And I don't think that holoviews is installed at all.

cc @mrocklin, @jbednar

[jhamman]

@rabernat - have you tried using the legacy notebook interface? We're currently defaulting to to the /lab interface but you access the default interface just the same (/tree): http://pangeo.pydata.org/user/rabernat/tree.

This seems to work for me when using the legacy interface:

[jhamman]

It looks like we would need the jupyter lab extension for bokeh: https://github.com/bokeh/jupyterlab_bokeh

I think this would need to go in the notebook image. In the short term, I think you can run jupyter labextension install jupyterlab_bokeh in a terminal to get the equivalent functionality. This worked for me.

[philippjfr]

We only merged JupyterLab support into HoloViews last week so there is not yet a release that includes it. That said I think we are ready to have a beta release tomorrow or shortly after, so you'll be able to get it going with:

conda install -c ioam/label/dev holoviews geoviews
conda install -c bokeh/label/dev bokeh
jupyter labextension install @pyviz/jupyterlab_holoviews

I'll let you know when that release is available, otherwise you could just grab holoviews/geovies from master.

[mrocklin]

Agreed. Short term I think the solution is for someone to push this and holoviews into the notebook image. That person is probably me. I'm probably occupied this afternoon on dask things though (recent tornado release has triggered some high priority work). Someone else could also try following the instructions in update.md though if they're interested.

On Wed, Mar 7, 2018 at 12:23 PM, Joe Hamman notifications@github.com wrote:

It looks like we would need the jupyter lab extension for bokeh: https://github.com/bokeh/jupyterlab_bokeh

I think this would need to go in the notebook image. In the short term, I think you can run jupyter labextension install jupyterlab_bokeh in a terminal to get the equivalent functionality. This worked for me.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pangeo-data/pangeo/issues/145#issuecomment-371214768, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszEgIwvKRI7-6zUxxHSo6PJMYG-Y1ks5tcBedgaJpZM4Sgwa3 .

[rabernat]

I will try to update the notebook image myself. We'll see if I succeed.

[mrocklin]

You'll need a place to push the images. I recommend creating a pangeo account on docker hub

On Wed, Mar 7, 2018 at 12:54 PM, Ryan Abernathey notifications@github.com wrote:

I will try to update the notebook image myself. We'll see if I succeed.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pangeo-data/pangeo/issues/145#issuecomment-371224850, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszDEQ9wGOLAAAXGmKMuYEHAMTLQWPks5tcB7WgaJpZM4Sgwa3 .

[jhamman]

We already have one: https://hub.docker.com/u/pangeo

[jhamman]

I've added @rabernat and @mrocklin to the pangeo team on dockerhub. If there are others that need push access, please send me a dm.

[rabernat]

Actually, this is still not working 100%. I can load holoviews in jupyterlab, but certain updates with holomaps and datashader don't work properly.

[mrocklin]

Yup. I'm building new docker images now.

On Mon, Mar 12, 2018 at 11:02 AM, Ryan Abernathey notifications@github.com wrote:

Actually, this is still not working 100%. I can load holoviews in jupyterlab, but certain updates with holomaps and datashader don't work properly.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pangeo-data/pangeo/issues/145#issuecomment-372340778, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszDOzL9fMwfW_fQ41peEEKe1IZl_Kks5tdo3ugaJpZM4Sgwa3 .

[rabernat]

We can always fall back on the legacy notebook, which seems to work great.

[philippjfr]

When I tried it earlier today it wasn't yet using bokeh 0.12.15devX, so upgrading that should fix it. Separately after some profiling I opened https://github.com/ioam/holoviews/pull/2434 which should be a huge improvement in responsiveness. We'll have new HoloViews and GeoViews alpha releases later today.

[jacobtomlinson]

Just FYI I've raised bokeh/jupyterlab_bokeh#9 as I've noticed code cells disappearing when saving if they produce a bokeh plot.

[rabernat]

When I tried holoviews today in pangeo.pydata.org, using lab, it did not work. No output from holoviews. hv.__version__ == 1.9.2, bokeh.__version__ == 0.12.15dev1

[mrocklin]

It would be useful if someone could come up with a working configuration for holoviews on jupyterlab.

On Fri, Mar 16, 2018 at 4:24 PM, Ryan Abernathey notifications@github.com wrote:

When I tried holoviews today in pangeo.pydata.org, using lab, it did not work. No output from holoviews. hv.version == 1.9.2, bokeh.version == 0.12.15dev1

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pangeo-data/pangeo/issues/145#issuecomment-373834667, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszKaVdmbnarq67qjSr0YksjFGyUXFks5tfB-DgaJpZM4Sgwa3 .

[philippjfr]

I've been trying to debug the issues with holoviews in JupyterLab today, but it's been hampered by the fact that I can't reproduce any issues locally. It should at least be partially working using holoviews 1.9.5 though.

[philippjfr]

The good news is, I just updated to bleeding edge JupyterLab and can now reproduce the issues I've been seeing on pangeo on my own machine, so I should be able to get a jupyterlab_holoviews release out to fix it very soon.

[mrocklin]

Huzzah! Thank you for handling this Philipp.

On Fri, Mar 16, 2018 at 8:12 PM, Philipp Rudiger notifications@github.com wrote:

The good news is, I just updated to bleeding edge JupyterLab and can now reproduce the issues I've been seeing on pangeo on my own machine, so I should be able to get a jupyterlab_holoviews release out to fix it very soon.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pangeo-data/pangeo/issues/145#issuecomment-373877225, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszOYNQPcGMTxkEUkrXduaATFrtQnGks5tfFTrgaJpZM4Sgwa3 .

[rsignell-usgs]

@philippjfr, is this still recommended install procedure to get geoviews going?

conda install -c ioam/label/dev holoviews geoviews
conda install -c bokeh/label/dev bokeh
jupyter labextension install @pyviz/jupyterlab_holoviews

or is it just:

conda install -c pyviz pyviz
jupyter labextension ??

or something else?

[rabernat]

FWIW, whatever is in the current pangeo notebook docker images does work now on pangeo.pydata.org: https://github.com/pangeo-data/helm-chart/blob/master/docker-images/notebook/Dockerfile

[rsignell-usgs]

@rabernat, my workflow uses geoviews, which the current pangeo notebook image does not seem to have.

[philippjfr]

Currently I would recommend:

conda install -c pyviz holoviews geoviews
conda install bokeh
jupyter labextension install @pyviz/jupyterlab_holoviews

and for the latest dev releases:

conda install -c pyviz/label/dev holoviews geoviews
conda install -c bokeh/label/dev bokeh
jupyter labextension install @pyviz/jupyterlab_holoviews

[chiaral]

Geoviews is still not in pangeo.pydata.org - is there any plan to include it? or should I install it in my directory?

[mrocklin]

Does it depend on the GDAL stack? Do we plan to add this to our dependencies? If so how much larger does this make images?

On Wed, Jun 27, 2018 at 11:31 AM, chiaral notifications@github.com wrote:

Geoviews is still not in pangeo.pydata.org - is there any plan to include it? or should I install it in my directory?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pangeo-data/pangeo/issues/145#issuecomment-400718374, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszCIDrmbV8tQa4oQkRbfr3O7ePFhKks5uA6VZgaJpZM4Sgwa3 .

[rsignell-usgs]

We have EarthSim (datashader, holoviews, geoviews) working in our Pangeo instance on AWS (http://pangeo.esipfed.org).

Here's the Dockerfile we are using for both Notebook and worker: https://github.com/rsignell-usgs/helm-chart/blob/conda-forge/docker-images/notebook/Dockerfile

Just FYI, I gave a live demo of this yesterday as part of a plenary talk at the ECM15 Meeting here in Seattle, using Dask with 30 workers to compute the maximum water levels during Hurricane Ike on a 9 million node unstructured mesh, crunching 50GB in less than a minute, then using geoviews to display the mesh interactively in the notebook! The Coastal and Estuarine Modeling community was wowed by Pangeo! 😸

[mrocklin]

Whoa! Nice. Any chance this was recorded?

On Wed, Jun 27, 2018 at 5:01 PM, Rich Signell notifications@github.com wrote:

We have EarthSim https://pyviz.github.io/EarthSim/user_guide/Visualizing_Meshes.html (datashader, holoviews, geoviews) working in our Pangeo instance on AWS ( http://pangeo.esipfed.org).

Here's the Dockerfile we are using for both Notebook and worker: https://github.com/rsignell-usgs/helm-chart/blob/conda- forge/docker-images/notebook/Dockerfile

Just FYI, I gave a live demo of this yesterday as part of a plenary talk at the ECM15 Meeting https://ecm.github.io/ECM15/ here in Seattle, using Dask with 30 workers to compute the maximum water levels during Hurricane Ike on a 9 million node unstructured mesh, crunching 50GB in less than a minute, then using geoviews to display the mesh interactively in the notebook! The Coastal and Estuarine Modeling community was wowed by Pangeo! 😸

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pangeo-data/pangeo/issues/145#issuecomment-400827701, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszMwKyNFewdhHlkla0DGpt5NRz7jEks5uA_KngaJpZM4Sgwa3 .

[rabernat]

@rsignell-usgs that sounds amazing!

We would love to include this use case in the gallery! Any chance you could submit your notebook via PR?

[rsignell-usgs]

@mrocklin, unfortunately not recorded, but I can show this demo at ESIP summer meeting, which will be recorded.

@rabernat, sure, I can submit a PR, although obviously the notebook won't work on pangeo.pydata.org if geoviews is not working. I guess I should also switch the data source bucket from AWS to GCS, right?

This is the notebook I used. https://nbviewer.jupyter.org/gist/rsignell-usgs/33476f91fa181840a1326b6a6ffad7ed (I just noticed that the start time metadata was not quite correct! I can fix that for the PR)

[rabernat]

The gallery is not exclusively for pangeo.pydata.org. There is an example from cheyenne in there already: http://pangeo-data.org/use_cases/index.html#use-cases

You don't have to change anything about your notebook. Our goal is to include all possible platforms where pangeo can be run!

On Wed, Jun 27, 2018 at 5:16 PM Rich Signell notifications@github.com wrote:

@mrocklin https://github.com/mrocklin, unfortunately not recorded, but I can show this demo at ESIP summer meeting, which will be recorded.

@rabernat https://github.com/rabernat, sure, I can submit a PR, although obviously the notebook won't work on pangeo.pydata.org if geoviews is not working. I guess I should also switch the data source bucket from AWS to GCS, right?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pangeo-data/pangeo/issues/145#issuecomment-400831769, or mute the thread https://github.com/notifications/unsubscribe-auth/ABJFJghG2x3fs_56Kek5BwDsCGpbPzZ7ks5uA_Y1gaJpZM4Sgwa3 .

[rsignell-usgs]

@rabernat , okay, cool. I'll fix the metadata in the dataset, add a bit more descriptive text in the notebook and submit.

[rsignell-usgs]

@rabernat & @jhamman, I'm ready to submit a PR for this Hurricane Ike example notebook, but I'm not at all sure where!

[martindurant]

Into https://github.com/pangeo-data/pangeo-example-notebooks ?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it had not seen recent activity. The issue can always be reopened at a later date.

[rsignell-usgs]

I've submitted my trimesh hurricane notebook and updated the SSH example notebook with hvplot but to run these on pangeo.pydata.org would require adding geoviews and hvplot to the environment.

I saw that a request to add cartopy was met with a suggestion to install a custom pangeo instance.

So I'm not sure if the same logic applies here, or whether is sufficient general interest to warrant the inclusion of these packages in the pydata instance?

[philippjfr]

Note that there is a geoviews-core package which has slightly lighter dependencies than the default geoviews package, basically just holoviews + cartopy.

[jhamman]

@rsignell-usgs - for context, I think some of the slow movement on these sorts of issues falls into these categories:

1. pangeo.pydata.org's environment image is somewhat fragile and building those images has led to some unpleasant effort by maintainers
2. we're moving away from pangeo.pydata.org as a one-stop-shop and encouraging smaller deployments that can be more easily customized
3. adding some of these packages results in some fairly large increases in docker image sizes which has a negative impact on the UX.

In total, we just don't have a way to make cost/benefit based decisions on stuff like this. I recognize this is probably frustrating as someone who developing some really cool demonstrations (like you). The binder deployment is meant to help with some of this but obviously doesn't hit all of the points.

[mrocklin]

Perhaps we can make Rich a custom deployment that he can control? Probably not rich.pangeo.io (though I'm sure that would be a fun place), perhaps there is some domain that a few colleagues would enjoy collaborating on as well?

On Fri, Oct 19, 2018 at 2:51 PM Joe Hamman notifications@github.com wrote:

@rsignell-usgs https://github.com/rsignell-usgs - for context, I think some of the slow movement on these sorts of issues falls into these categories:

1. pangeo.pydata.org's environment image is somewhat fragile and building those images has led to some unpleasant effort by maintainers
2. we're moving away from pangeo.pydata.org as a one-stop-shop and encouraging smaller deployments that can be more easily customized
3. adding some of these packages results in some fairly large increases in docker image sizes which has a negative impact on the UX.

In total, we just don't have a way to make cost/benefit based decisions on stuff like this. I recognize this is probably frustrating as someone who developing some really cool demonstrations (like you). The binder deployment is meant to help with some of this but obviously doesn't hit all of the points.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pangeo-data/pangeo/issues/145#issuecomment-431461820, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszOJ60RwyG7n9nytqjY2ubhA1MtiDks5umh67gaJpZM4Sgwa3 .

[jhamman]

@rsignell-usgs - you should call it pangeo.esipfed.org 😉 😉 😉

[rsignell-usgs]

Hee hee. Yes, @jhamman, our pangeo.esipfed.org instance on AWS is working quite nicely with the full PyViz stack enabled (including geoviews, cartopy and panel).

I just wanted to share with a broader community. 😸

I understand the fragility of the current pangeo.pydata.org environment, with a pool of five channels to pull from and numerous pinned package versions. I think a safer approach would be to specify conda-forge only as the channel to draw from, and only pull specific packages from specific channels if needed.

I would love to have a site on pangeo.io to test this potentially-more-robustly-built notebook container out. Maybe something like dev.pangeo.io ?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it had not seen recent activity. The issue can always be reopened at a later date.