Loading local tiles to geoviews

[Mossi8]

Hi, I have already been using for a long time datashader in combination to holoviews and bokeh to plot high-throughput microscopy data from my lab and the results are great. But, we are interested in plotting at the same time the images which have a really large size: can be more than 5GB.

I tried to generate tiles with gdal, but I am not sure if these can be loaded locally using geoviews. And, I don't seem to find a way in geoviews to load local tiles, so that the whole large image does not have to be loaded as gv.Image (in this case it will crash).

I would really like to know if what I want to do would be possible and if you would have any suggestions.

I really appreciate your help and time, thank you so much.

[jbednar]

Glad to hear you're finding it useful for that research area! You can surely launch a WMTS tile server on a local port and pass that URL to gv.WMTS(), if you want to. GeoViews shouldn't care whether it's local or remote, as long as it's a working URL.

But using GeoViews and WMTS is obviously not a great solution, because the data isn't geo located in the first place. The underlying tile support in Bokeh seems like it should work for non-geo coordinates (bokeh.models.TileSource), and I've seen an example using it for non-geo data in private code several years ago, but I don't have any public examples to work from for adding a tile source object to HoloViews to cover non-geo cases like this.

In any case, tiles on disk like that are probably only really useful for publishing to a static web server; for interactive exploration of novel data, it would be better if things just worked with the real data, albeit more slowly. Dask should let you do that, with out of core and distributed computation, though we may need to do work in Datashader and HoloViews to support that better.

You can see discussion of this use case in https://github.com/bokeh/datashader/issues/478 , and an example of navigating an EM stack at pangeo.pydata.org (launch a server and select scikit-image-example.ipynb from the examples). If you try that example on your own machine, you should use the very latest HoloViews alpha release 1.10.0a2 (from conda install -c ioam/label/dev holoviews), which has massive speed improvements for this case. Does this approach get close to handling the data size you need?

[Mossi8]

Thank you so much, I still need to fix the code for the visualization, but so far simply using the regrid() did the trick to plot the large image, as in the example bokeh/datashader#478. Even though, I still need to try it with larger images.

I will also try the example at pangeo.pydata.org and see how it goes. But, so far it looks feasible.

Thank you so much, this will help us a lot!!!

[VolkerH]

Thanks for referring me to this example. I just tried to run the EM stack example on the Pangeo server which is running holoviews '1.10.0a2.post26+g3f015c0'. Unfortunately, the import of regrid fails

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-8-bedc7f2208a4> in <module>()
      2 import holoviews as hv
      3 import numpy as np
----> 4 from holoviews.operation.datashader import regrid
      5 hv.extension('bokeh')

/opt/conda/lib/python3.6/site-packages/holoviews/operation/datashader.py in <module>()
    184 
    185 
--> 186 class AggregationOperation(ResamplingOperation):
    187     """
    188     AggregationOperation extends the ResamplingOperation defining an

/opt/conda/lib/python3.6/site-packages/holoviews/operation/datashader.py in AggregationOperation()
    200         'any':   rd.any,
    201         'count': rd.count,
--> 202         'first': rd.first,
    203         'last':  rd.last,
    204         'mode':  rd.mode,

AttributeError: module 'datashader.reductions' has no attribute 'first'

[VolkerH]

Ok, I can see that it has to do with the datashader version on the pangeo server not being up to date (it has datashader.__version__ = '0.6.4dev1'). Probably need to set all of this up on a local machine to try it out.

[jbednar]

Hmm; that's odd; it worked when I tried it a few weeks ago. The pangeo project is happy to engage with external contributors, and I'm sure they'd like to see a PR to update the datashader version to be the 0.6.5 release since it seems broken there right now.

[philippjfr]

They are installing HoloViews from master, which had some recent changes that made it incompatible with datashader < 0.6.5. You can easily pop up a console on pangeo and upgrade datashader yourself. I'll get in touch with them to get datashader updated globally though.

[kidpixo]

first of all : Thanks for the super nice work on this, @jbednar !

Just for the note:

1. I generate some tiles using roblabs/gdal2tilesp: Enhancements to tile cutter for parallelism and image format support with the option -o xyz that generate the tiles in Slippy map tilenames, slightly different from TMS.
2. serve the tiles locally with a simple webserver (python -m http.server)
3. display the tiles with geoviews via gv.WMTS(WMTSTileSource(url='http://0.0.0.0:8000/temp_xyz/{Z}/{X}/{Y}.png')

Note the swap X<>Y . Don't know why, I found this workaround once with jupyter-widgets/ipyleaflet and works here too.

Those aren't Earth data, but from Mercury, via a the NASA MESSENGER mission.

In the picture, on the right I overlay ~6M points via holoviews.operation.datashader.rasterize and this image doesn't wrap the boundary like the Tiles.

One thing I'm not sure of: if I create a layout via tms+rasterized, how can I automatically link the zoom/pan around? I set the same rasterized kdims name like the Tiles Map, but it is not working.

[philippjfr]

As long as the data corresponds to one of the known tile formats this is already possible.