search

holoviz / holoviews

With Holoviews, your data visualizes itself.
https://holoviews.org
BSD 3-Clause "New" or "Revised" License
2.66k stars 396 forks source link

datashade of NdOverlay of QuadMesh does not work #6204

Open peterroelants opened 2 months ago

peterroelants commented 2 months ago

Description of expected behavior and the observed behavior

Datashading an NdOverlay of Quadmeshes (with different axis bins) does not work and results in: ValueError: Supplied Image bounds do not match the coordinates defined in the data. Bounds only have to be declared if no coordinates are supplied, otherwise they must match the data. To change the displayed extents set the range on the x- and y-dimensions.

I tried setting hv.config.image_rtol = 1 but with no result

Complete, minimal, self-contained example code that reproduces the issue

import numpy as np
import xarray as xr

import datashader
import holoviews as hv
from holoviews.operation.datashader import datashade, shade, dynspread, spread, rasterize
import panel as pn
import param

hv.extension('bokeh', logo=False)
pn.extension(comms='vscode')
np.random.seed(1)

def generate_data(direction: str) -> xr.DataArray:
    # Non shared axis doesn't work
    x_axis = np.sort(np.random.rand(300)) * 20
    y_axis = np.sort(np.random.rand(50)) * 15
    # Shared axis works
    # x_axis = np.linspace(0, 20)
    # y_axis = np.linspace(0, 15)
    if direction == "vertical":
        data = np.sin(np.expand_dims(x_axis, -1) * np.expand_dims(np.ones_like(y_axis), 0))
    elif direction == "horizontal":
        data = np.sin(np.expand_dims(np.ones_like(x_axis), -1) * np.expand_dims(y_axis, 0))
    else:
        assert False
    da = xr.DataArray(
        data=data,
        name="D",
        dims=["x", "y"],
        coords={
            "x": x_axis,
            "y": y_axis,
        },
    )
    return da

samples = {
    k: generate_data(k)
    for k in ["vertical", "horizontal"]
}

meshes = {
    k: hv.QuadMesh(da.where(da>0), kdims=["x", "y"], vdims=["D"], label=k).opts(alpha=0.5)
    for k, da in samples.items()
}

overlay = hv.NdOverlay(meshes, kdims=["sample"], sort=False)

# Set rtol to high value
hv.config.image_rtol = 1

# Rasterize by itself works, but doesn't shade
# rasterize(overlay)
# Calling `datashade` does not work
# datashade(overlay)
shade(
    rasterize(overlay),
    aggregator=datashader.by("sample"),
)

Stack traceback and/or browser JavaScript console output

Stack trace ```python-traceback WARNING:param.RGB01051: RGB dimension(s) x and y are not evenly sampled to relative tolerance of 1. Please use the QuadMesh element for irregularly sampled data or set a higher tolerance on hv.config.image_rtol or the rtol parameter in the RGB constructor. --------------------------------------------------------------------------- ValueError Traceback (most recent call last) File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/IPython/core/formatters.py:977](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/IPython/core/formatters.py#line=976), in MimeBundleFormatter.__call__(self, obj, include, exclude) 974 method = get_real_method(obj, self.print_method) 976 if method is not None: --> 977 return method(include=include, exclude=exclude) 978 return None 979 else: File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/core/dimension.py:1286](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/core/dimension.py#line=1285), in Dimensioned._repr_mimebundle_(self, include, exclude) 1279 def _repr_mimebundle_(self, include=None, exclude=None): 1280 """ 1281 Resolves the class hierarchy for the class rendering the 1282 object using any display hooks registered on Store.display 1283 hooks. The output of all registered display_hooks is then 1284 combined and returned. 1285 """ -> 1286 return Store.render(self) File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/core/options.py:1428](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/core/options.py#line=1427), in Store.render(cls, obj) 1426 data, metadata = {}, {} 1427 for hook in hooks: -> 1428 ret = hook(obj) 1429 if ret is None: 1430 continue File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/ipython/display_hooks.py:287](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/ipython/display_hooks.py#line=286), in pprint_display(obj) 285 if not ip.display_formatter.formatters['text[/plain](http://localhost:8888/plain)'].pprint: 286 return None --> 287 return display(obj, raw_output=True) File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/ipython/display_hooks.py:261](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/ipython/display_hooks.py#line=260), in display(obj, raw_output, **kwargs) 259 elif isinstance(obj, (HoloMap, DynamicMap)): 260 with option_state(obj): --> 261 output = map_display(obj) 262 elif isinstance(obj, Plot): 263 output = render(obj) File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/ipython/display_hooks.py:149](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/ipython/display_hooks.py#line=148), in display_hook..wrapped(element) 147 try: 148 max_frames = OutputSettings.options['max_frames'] --> 149 mimebundle = fn(element, max_frames=max_frames) 150 if mimebundle is None: 151 return {}, {} File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/ipython/display_hooks.py:209](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/ipython/display_hooks.py#line=208), in map_display(vmap, max_frames) 206 max_frame_warning(max_frames) 207 return None --> 209 return render(vmap) File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/ipython/display_hooks.py:76](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/ipython/display_hooks.py#line=75), in render(obj, **kwargs) 73 if renderer.fig == 'pdf': 74 renderer = renderer.instance(fig='png') ---> 76 return renderer.components(obj, **kwargs) File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/plotting/renderer.py:397](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/plotting/renderer.py#line=396), in Renderer.components(self, obj, fmt, comm, **kwargs) 395 if embed or config.comms == 'default': 396 return self._render_panel(plot, embed, comm) --> 397 return self._render_ipywidget(plot) File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/plotting/renderer.py:419](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/plotting/renderer.py#line=418), in Renderer._render_ipywidget(self, plot) 417 def _render_ipywidget(self, plot): 418 # Handle rendering object as ipywidget --> 419 widget = ipywidget(plot, combine_events=True) 420 if hasattr(widget, '_repr_mimebundle_'): 421 return widget._repr_mimebundle_(), {} File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/panel/io/notebook.py:560](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/panel/io/notebook.py#line=559), in ipywidget(obj, doc, **kwargs) 558 from ..pane import panel 559 doc = doc if doc else Document() --> 560 model = panel(obj, **kwargs).get_root(doc=doc) 561 widget = BokehModel(model, combine_events=True) 562 if hasattr(widget, '_view_count'): File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/panel/pane/base.py:422](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/panel/pane/base.py#line=421), in PaneBase.get_root(self, doc, comm, preprocess) 420 root = wrapper.get_root(doc, comm, preprocess) 421 else: --> 422 root_view, root = self._get_root_model(doc, comm, preprocess) 423 ref = root.ref['id'] 424 state._views[ref] = (root_view, root, doc, comm) File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/panel/pane/base.py:351](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/panel/pane/base.py#line=350), in PaneBase._get_root_model(self, doc, comm, preprocess) 349 root_view = self 350 else: --> 351 root = self.layout._get_model(doc, comm=comm) 352 root_view = self.layout 353 if preprocess: File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/panel/layout/base.py:186](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/panel/layout/base.py#line=185), in Panel._get_model(self, doc, root, parent, comm) 184 root = root or model 185 self._models[root.ref['id']] = (model, parent) --> 186 objects, _ = self._get_objects(model, [], doc, root, comm) 187 props = self._get_properties(doc) 188 props[self._property_mapping['objects']] = objects File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/panel/layout/base.py:168](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/panel/layout/base.py#line=167), in Panel._get_objects(self, model, old_objects, doc, root, comm) 166 else: 167 try: --> 168 child = pane._get_model(doc, root, model, comm) 169 except RerenderError as e: 170 if e.layout is not None and e.layout is not self: File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/panel/pane/holoviews.py:429](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/panel/pane/holoviews.py#line=428), in HoloViews._get_model(self, doc, root, parent, comm) 427 plot = self.object 428 else: --> 429 plot = self._render(doc, comm, root) 431 plot.pane = self 432 backend = plot.renderer.backend File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/panel/pane/holoviews.py:525](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/panel/pane/holoviews.py#line=524), in HoloViews._render(self, doc, comm, root) 522 if comm: 523 kwargs['comm'] = comm --> 525 return renderer.get_plot(self.object, **kwargs) File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/plotting/bokeh/renderer.py:68](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/plotting/bokeh/renderer.py#line=67), in BokehRenderer.get_plot(self_or_cls, obj, doc, renderer, **kwargs) 61 @bothmethod 62 def get_plot(self_or_cls, obj, doc=None, renderer=None, **kwargs): 63 """ 64 Given a HoloViews Viewable return a corresponding plot instance. 65 Allows supplying a document attach the plot to, useful when 66 combining the bokeh model with another plot. 67 """ ---> 68 plot = super().get_plot(obj, doc, renderer, **kwargs) 69 if plot.document is None: 70 plot.document = Document() if self_or_cls.notebook_context else curdoc() File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/plotting/renderer.py:217](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/plotting/renderer.py#line=216), in Renderer.get_plot(self_or_cls, obj, doc, renderer, comm, **kwargs) 214 raise SkipRendering(msg.format(dims=dims)) 216 # Initialize DynamicMaps with first data item --> 217 initialize_dynamic(obj) 219 if not renderer: 220 renderer = self_or_cls File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/plotting/util.py:270](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/plotting/util.py#line=269), in initialize_dynamic(obj) 268 continue 269 if not len(dmap): --> 270 dmap[dmap._initial_key()] File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/core/spaces.py:1217](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/core/spaces.py#line=1216), in DynamicMap.__getitem__(self, key) 1215 # Not a cross product and nothing cached so compute element. 1216 if cache is not None: return cache -> 1217 val = self._execute_callback(*tuple_key) 1218 if data_slice: 1219 val = self._dataslice(val, data_slice) File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/core/spaces.py:984](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/core/spaces.py#line=983), in DynamicMap._execute_callback(self, *args) 981 kwargs['_memoization_hash_'] = hash_items 983 with dynamicmap_memoization(self.callback, self.streams): --> 984 retval = self.callback(*args, **kwargs) 985 return self._style(retval) File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/core/spaces.py:552](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/core/spaces.py#line=551), in Callable.__call__(self, *args, **kwargs) 550 return self.callable.rx.value 551 elif not args and not kwargs and not any(kwarg_hash): --> 552 return self.callable() 553 inputs = [i for i in self.inputs if isinstance(i, DynamicMap)] 554 streams = [] File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/util/__init__.py:1038](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/util/__init__.py#line=1037), in Dynamic._dynamic_operation..dynamic_operation(*key, **kwargs) 1036 def dynamic_operation(*key, **kwargs): 1037 key, obj = resolve(key, kwargs) -> 1038 return apply(obj, *key, **kwargs) File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/util/__init__.py:1030](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/util/__init__.py#line=1029), in Dynamic._dynamic_operation..apply(element, *key, **kwargs) 1028 def apply(element, *key, **kwargs): 1029 kwargs = dict(util.resolve_dependent_kwargs(self.p.kwargs), **kwargs) -> 1030 processed = self._process(element, key, kwargs) 1031 if (self.p.link_dataset and isinstance(element, Dataset) and 1032 isinstance(processed, Dataset) and processed._dataset is None): 1033 processed._dataset = element.dataset File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/util/__init__.py:1012](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/util/__init__.py#line=1011), in Dynamic._process(self, element, key, kwargs) 1010 elif isinstance(self.p.operation, Operation): 1011 kwargs = {k: v for k, v in kwargs.items() if k in self.p.operation.param} -> 1012 return self.p.operation.process_element(element, key, **kwargs) 1013 else: 1014 return self.p.operation(element, **kwargs) File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/core/operation.py:194](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/core/operation.py#line=193), in Operation.process_element(self, element, key, **params) 191 else: 192 self.p = param.ParamOverrides(self, params, 193 allow_extra_keywords=self._allow_extra_keywords) --> 194 return self._apply(element, key) File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/core/operation.py:141](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/core/operation.py#line=140), in Operation._apply(self, element, key) 139 if not in_method: 140 element._in_method = True --> 141 ret = self._process(element, key) 142 if hasattr(element, '_in_method') and not in_method: 143 element._in_method = in_method File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/operation/datashader.py:1340](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/operation/datashader.py#line=1339), in shade._process(self, element, key) 1338 else: 1339 img = tf.shade(array, **shade_opts) -> 1340 return RGB(self.uint32_to_uint8_xr(img), **params) File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/element/raster.py:696](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/element/raster.py#line=695), in RGB.__init__(self, data, kdims, vdims, **params) 690 if ((hasattr(data, 'shape') and data.shape[-1] == 4 and len(vdims) == 3) or 691 (isinstance(data, tuple) and isinstance(data[-1], np.ndarray) and data[-1].ndim == 3 692 and data[-1].shape[-1] == 4 and len(vdims) == 3) or 693 (isinstance(data, dict) and tuple(dimension_name(vd) for vd in vdims)+(alpha.name,) in data)): 694 # Handle all forms of packed value dimensions 695 vdims.append(alpha) --> 696 super().__init__(data, kdims=kdims, vdims=vdims, **params) File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/element/raster.py:315](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/element/raster.py#line=314), in Image.__init__(self, data, kdims, vdims, bounds, extents, xdensity, ydensity, rtol, **params) 313 if non_finite: 314 self.bounds = BoundingBox(points=((np.nan, np.nan), (np.nan, np.nan))) --> 315 self._validate(data_bounds, supplied_bounds) File [~/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/element/raster.py:381](http://localhost:8888/home/peter/mambaforge/envs/lcms_polymer_env/lib/python3.11/site-packages/holoviews/element/raster.py#line=380), in Image._validate(self, data_bounds, supplied_bounds) 379 not_close = True 380 if not_close: --> 381 raise ValueError('Supplied Image bounds do not match the coordinates defined ' 382 'in the data. Bounds only have to be declared if no coordinates ' 383 'are supplied, otherwise they must match the data. To change ' 384 'the displayed extents set the range on the x- and y-dimensions.') ValueError: Supplied Image bounds do not match the coordinates defined in the data. Bounds only have to be declared if no coordinates are supplied, otherwise they must match the data. To change the displayed extents set the range on the x- and y-dimensions. ```

More info:

ALL software version info

Python implementation: CPython
Python version       : 3.11.9
IPython version      : 8.23.0
holoviews : 1.18.3
param     : 2.1.0
numpy     : 1.26.4
panel     : 1.4.1
datashader: 0.16.1
xarray    : 2024.3.0
hoxbro commented 2 months ago

It seems to work if you use Overlay over 'NdOverlay`?

meshes = [
    hv.QuadMesh(da.where(da > 0), kdims=["x", "y"], vdims=["D"]).opts(alpha=0.5)
    for da in samples.values()
]
overlay = hv.Overlay(meshes, kdims=["sample"])
shade(
    rasterize(overlay),
    aggregator=datashader.by("sample"),
)
peterroelants commented 2 months ago

It seems to work if you use Overlay over 'NdOverlay`?

Using Overlay instead of NdOverlay does not color the meshes ("sample" dimension) differently: bokeh_plot

hoxbro commented 2 months ago

What about this? Screenshot 2024-04-26 15 59 50

peterroelants commented 2 months ago

What about this?

Hey, thanks, however it still seems like it's missing some shading between the two groups.

I've been playing around with the image_rtol value and if I set it to something really large (e.g. hv.config.image_rtol = 1e6) it seems to work:

bokeh_plot (1)

However, I wonder what the implications are of setting hv.config's image_rtol to such a high value. Is image_rtol only used for validation and nothing else?