search

holoviz / hvplot

A high-level plotting API for pandas, dask, xarray, and networkx built on HoloViews
https://hvplot.holoviz.org
BSD 3-Clause "New" or "Revised" License
1.08k stars 105 forks source link

Quadmesh + datashader + dask raises numba error #434

Closed rabernat closed 4 years ago

rabernat commented 4 years ago

ALL software version info

I am using the latest released versions of all the relevant libraries.

Hvplot 0.5.2 Datshader 0.10.0 Numba 0.48.0

Description of expected behavior and the observed behavior

Complete, minimal, self-contained example code that reproduces the issue

I am trying to plot an xarray dataset backed by dask using hvplot. Here is how I load the data. (Should be reproducible with intake installed.)

from intake import open_catalog
cat = open_catalog("https://raw.githubusercontent.com/pangeo-data/pangeo-datastore/master/intake-catalogs/ocean.yaml")
ds  = cat["sea_surface_height"].to_dask()

The following code works fine:

ds.sla.hvplot.image('longitude', 'latitude', rasterize=True, dynamic=True)

However, if I try to do quadmesh instead

ds.sla.hvplot.quadmesh('longitude', 'latitude', rasterize=True, dynamic=True)

I get a big long error.

Stack traceback and/or browser JavaScript console output

WARNING:param.dynamic_operation: Callable raised "TypingError('Failed in nopython mode pipeline (step: nopython frontend)\nnon-precise type pyobject\n[1] During: typing of argument at <extend_cpu> (4)\n\nFile "<extend_cpu>", line 4:\n<source missing, REPL/exec in use?>\n\nThis error may have been caused by the following argument(s):\n- argument 4: cannot determine Numba type of <class \'dask.array.core.Array\'>\n')".
Invoked as dynamic_operation(numpy.datetime64('1993-01-01T00:00:00.000000000'), height=300, scale=1.0, width=700, x_range=None, y_range=None)
WARNING:param.dynamic_operation: Callable raised "TypingError('Failed in nopython mode pipeline (step: nopython frontend)\nnon-precise type pyobject\n[1] During: typing of argument at <extend_cpu> (4)\n\nFile "<extend_cpu>", line 4:\n<source missing, REPL/exec in use?>\n\nThis error may have been caused by the following argument(s):\n- argument 4: cannot determine Numba type of <class \'dask.array.core.Array\'>\n')".
Invoked as dynamic_operation(numpy.datetime64('1993-01-01T00:00:00.000000000'), height=300, scale=1.0, width=700, x_range=None, y_range=None)
---------------------------------------------------------------------------
TypingError                               Traceback (most recent call last)
/srv/conda/envs/pangeo/lib/python3.7/site-packages/IPython/core/formatters.py in __call__(self, obj, include, exclude)
    968 
    969             if method is not None:
--> 970                 return method(include=include, exclude=exclude)
    971             return None
    972         else:

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/core/dimension.py in _repr_mimebundle_(self, include, exclude)
   1302         combined and returned.
   1303         """
-> 1304         return Store.render(self)
   1305 
   1306 

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/core/options.py in render(cls, obj)
   1393         data, metadata = {}, {}
   1394         for hook in hooks:
-> 1395             ret = hook(obj)
   1396             if ret is None:
   1397                 continue

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/ipython/display_hooks.py in pprint_display(obj)
    280     if not ip.display_formatter.formatters['text/plain'].pprint:
    281         return None
--> 282     return display(obj, raw_output=True)
    283 
    284 

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/ipython/display_hooks.py in display(obj, raw_output, **kwargs)
    256     elif isinstance(obj, (HoloMap, DynamicMap)):
    257         with option_state(obj):
--> 258             output = map_display(obj)
    259     elif isinstance(obj, Plot):
    260         output = render(obj)

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/ipython/display_hooks.py in wrapped(element)
    144         try:
    145             max_frames = OutputSettings.options['max_frames']
--> 146             mimebundle = fn(element, max_frames=max_frames)
    147             if mimebundle is None:
    148                 return {}, {}

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/ipython/display_hooks.py in map_display(vmap, max_frames)
    204         return None
    205 
--> 206     return render(vmap)
    207 
    208 

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/ipython/display_hooks.py in render(obj, **kwargs)
     66         renderer = renderer.instance(fig='png')
     67 
---> 68     return renderer.components(obj, **kwargs)
     69 
     70 

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/plotting/renderer.py in components(self, obj, fmt, comm, **kwargs)
    363             doc = Document()
    364             with config.set(embed=embed):
--> 365                 model = plot.layout._render_model(doc, comm)
    366             return render_model(model, comm) if embed else render_mimebundle(model, doc, comm)
    367         else:

/srv/conda/envs/pangeo/lib/python3.7/site-packages/panel/viewable.py in _render_model(self, doc, comm)
    363         if comm is None:
    364             comm = state._comm_manager.get_server_comm()
--> 365         model = self.get_root(doc, comm)
    366 
    367         if config.embed:

/srv/conda/envs/pangeo/lib/python3.7/site-packages/panel/viewable.py in get_root(self, doc, comm)
    534         """
    535         doc = doc or _curdoc()
--> 536         root = self._get_model(doc, comm=comm)
    537         self._preprocess(root)
    538         ref = root.ref['id']

/srv/conda/envs/pangeo/lib/python3.7/site-packages/panel/layout.py in _get_model(self, doc, root, parent, comm)
    134         if root is None:
    135             root = model
--> 136         objects = self._get_objects(model, [], doc, root, comm)
    137         props = dict(self._init_properties(), objects=objects)
    138         model.update(**self._process_param_change(props))

/srv/conda/envs/pangeo/lib/python3.7/site-packages/panel/layout.py in _get_objects(self, model, old_objects, doc, root, comm)
    124             else:
    125                 try:
--> 126                     child = pane._get_model(doc, root, model, comm)
    127                 except RerenderError:
    128                     return self._get_objects(model, current_objects[:i], doc, root, comm)

/srv/conda/envs/pangeo/lib/python3.7/site-packages/panel/pane/holoviews.py in _get_model(self, doc, root, parent, comm)
    225             plot = self.object
    226         else:
--> 227             plot = self._render(doc, comm, root)
    228 
    229         plot.pane = self

/srv/conda/envs/pangeo/lib/python3.7/site-packages/panel/pane/holoviews.py in _render(self, doc, comm, root)
    284             kwargs = {}
    285 
--> 286         return renderer.get_plot(self.object, **kwargs)
    287 
    288     def _cleanup(self, root):

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/plotting/bokeh/renderer.py in get_plot(self_or_cls, obj, doc, renderer, **kwargs)
     71         combining the bokeh model with another plot.
     72         """
---> 73         plot = super(BokehRenderer, self_or_cls).get_plot(obj, doc, renderer, **kwargs)
     74         if plot.document is None:
     75             plot.document = Document() if self_or_cls.notebook_context else curdoc()

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/plotting/renderer.py in get_plot(self_or_cls, obj, doc, renderer, comm, **kwargs)
    200 
    201         # Initialize DynamicMaps with first data item
--> 202         initialize_dynamic(obj)
    203 
    204         if not renderer:

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/plotting/util.py in initialize_dynamic(obj)
    249             continue
    250         if not len(dmap):
--> 251             dmap[dmap._initial_key()]
    252 
    253 

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/core/spaces.py in __getitem__(self, key)
   1278         # Not a cross product and nothing cached so compute element.
   1279         if cache is not None: return cache
-> 1280         val = self._execute_callback(*tuple_key)
   1281         if data_slice:
   1282             val = self._dataslice(val, data_slice)

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/core/spaces.py in _execute_callback(self, *args)
   1052 
   1053         with dynamicmap_memoization(self.callback, self.streams):
-> 1054             retval = self.callback(*args, **kwargs)
   1055         return self._style(retval)
   1056 

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/core/spaces.py in __call__(self, *args, **kwargs)
    691 
    692         try:
--> 693             ret = self.callable(*args, **kwargs)
    694         except KeyError:
    695             # KeyError is caught separately because it is used to signal

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/util/__init__.py in dynamic_operation(*key, **kwargs)
    983 
    984         def dynamic_operation(*key, **kwargs):
--> 985             key, obj = resolve(key, kwargs)
    986             return apply(obj, *key, **kwargs)
    987 

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/util/__init__.py in resolve(key, kwargs)
    976             elif isinstance(map_obj, DynamicMap) and map_obj._posarg_keys and not key:
    977                 key = tuple(kwargs[k] for k in map_obj._posarg_keys)
--> 978             return key, map_obj[key]
    979 
    980         def apply(element, *key, **kwargs):

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/core/spaces.py in __getitem__(self, key)
   1278         # Not a cross product and nothing cached so compute element.
   1279         if cache is not None: return cache
-> 1280         val = self._execute_callback(*tuple_key)
   1281         if data_slice:
   1282             val = self._dataslice(val, data_slice)

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/core/spaces.py in _execute_callback(self, *args)
   1052 
   1053         with dynamicmap_memoization(self.callback, self.streams):
-> 1054             retval = self.callback(*args, **kwargs)
   1055         return self._style(retval)
   1056 

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/core/spaces.py in __call__(self, *args, **kwargs)
    691 
    692         try:
--> 693             ret = self.callable(*args, **kwargs)
    694         except KeyError:
    695             # KeyError is caught separately because it is used to signal

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/util/__init__.py in dynamic_operation(*key, **kwargs)
    984         def dynamic_operation(*key, **kwargs):
    985             key, obj = resolve(key, kwargs)
--> 986             return apply(obj, *key, **kwargs)
    987 
    988         operation = self.p.operation

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/util/__init__.py in apply(element, *key, **kwargs)
    980         def apply(element, *key, **kwargs):
    981             kwargs = dict(util.resolve_dependent_kwargs(self.p.kwargs), **kwargs)
--> 982             return self._process(element, key, kwargs)
    983 
    984         def dynamic_operation(*key, **kwargs):

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/util/__init__.py in _process(self, element, key, kwargs)
    962         elif isinstance(self.p.operation, Operation):
    963             kwargs = {k: v for k, v in kwargs.items() if k in self.p.operation.param}
--> 964             return self.p.operation.process_element(element, key, **kwargs)
    965         else:
    966             return self.p.operation(element, **kwargs)

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/core/operation.py in process_element(self, element, key, **params)
    170             self.p = param.ParamOverrides(self, params,
    171                                           allow_extra_keywords=self._allow_extra_keywords)
--> 172         return self._apply(element, key)
    173 
    174 

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/core/operation.py in _apply(self, element, key)
    130         element_pipeline = getattr(element, '_pipeline', None)
    131 
--> 132         ret = self._process(element, key)
    133         for hook in self._postprocess_hooks:
    134             ret = hook(self, ret, **kwargs)

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/operation/datashader.py in _process(self, element, key)
   1413                                        if k in transform.param})
   1414             op._precomputed = self._precomputed
-> 1415             element = element.map(op, predicate)
   1416             self._precomputed = op._precomputed
   1417 

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/core/data/__init__.py in pipelined_fn(*args, **kwargs)
    212 
    213             try:
--> 214                 result = method_fn(*args, **kwargs)
    215 
    216                 op = method_op.instance(

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/core/data/__init__.py in map(self, *args, **kwargs)
   1167 
   1168     def map(self, *args, **kwargs):
-> 1169         return super(Dataset, self).map(*args, **kwargs)
   1170     map.__doc__ = LabelledData.map.__doc__
   1171 

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/core/dimension.py in map(self, map_fn, specs, clone)
    703             return deep_mapped
    704         else:
--> 705             return map_fn(self) if applies else self
    706 
    707 

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/core/operation.py in __call__(self, element, **kwargs)
    190             elif ((self._per_element and isinstance(element, Element)) or
    191                   (not self._per_element and isinstance(element, ViewableElement))):
--> 192                 return self._apply(element)
    193         elif 'streams' not in kwargs:
    194             kwargs['streams'] = self.p.streams

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/core/operation.py in _apply(self, element, key)
    130         element_pipeline = getattr(element, '_pipeline', None)
    131 
--> 132         ret = self._process(element, key)
    133         for hook in self._postprocess_hooks:
    134             ret = hook(self, ret, **kwargs)

/srv/conda/envs/pangeo/lib/python3.7/site-packages/holoviews/operation/datashader.py in _process(self, element, key)
   1081 
   1082         vdim = getattr(agg_fn, 'column', element.vdims[0].name)
-> 1083         agg = cvs.quadmesh(data[vdim], x.name, y.name, agg_fn)
   1084         xdim, ydim = list(agg.dims)[:2][::-1]
   1085         if xtype == "datetime":

/srv/conda/envs/pangeo/lib/python3.7/site-packages/datashader/core.py in quadmesh(self, source, x, y, agg)
    770                 dims=list(xarr.dims)))
    771 
--> 772         return bypixel(source, self, glyph, agg)
    773 
    774     # TODO re 'untested', below: Consider replacing with e.g. a 3x3

/srv/conda/envs/pangeo/lib/python3.7/site-packages/datashader/core.py in bypixel(source, canvas, glyph, agg)
   1159     with np.warnings.catch_warnings():
   1160         np.warnings.filterwarnings('ignore', r'All-NaN (slice|axis) encountered')
-> 1161         return bypixel.pipeline(source, schema, canvas, glyph, agg)
   1162 
   1163 

/srv/conda/envs/pangeo/lib/python3.7/site-packages/datashader/utils.py in __call__(self, head, *rest, **kwargs)
     91         typ = type(head)
     92         if typ in lk:
---> 93             return lk[typ](head, *rest, **kwargs)
     94         for cls in getmro(typ)[1:]:
     95             if cls in lk:

/srv/conda/envs/pangeo/lib/python3.7/site-packages/datashader/data_libraries/xarray.py in xarray_pipeline(df, schema, canvas, glyph, summary)
     17 def xarray_pipeline(df, schema, canvas, glyph, summary):
     18     cuda = cupy and isinstance(df[glyph.name].data, cupy.ndarray)
---> 19     return glyph_dispatch(glyph, df, schema, canvas, summary, cuda)
     20 
     21 

/srv/conda/envs/pangeo/lib/python3.7/site-packages/datashader/utils.py in __call__(self, head, *rest, **kwargs)
     94         for cls in getmro(typ)[1:]:
     95             if cls in lk:
---> 96                 return lk[cls](head, *rest, **kwargs)
     97         raise TypeError("No dispatch for {0} type".format(typ))
     98 

/srv/conda/envs/pangeo/lib/python3.7/site-packages/datashader/data_libraries/pandas.py in default(glyph, source, schema, canvas, summary, cuda)
     43 
     44     bases = create((height, width))
---> 45     extend(bases, source, x_st + y_st, x_range + y_range)
     46 
     47     return finalize(bases,

/srv/conda/envs/pangeo/lib/python3.7/site-packages/datashader/glyphs/quadmesh.py in extend(aggs, xr_ds, vt, bounds)
    186                 do_extend = extend_cpu
    187 
--> 188             do_extend(xs, ys, *aggs_and_cols)
    189 
    190         return extend

/srv/conda/envs/pangeo/lib/python3.7/site-packages/numba/dispatcher.py in _compile_for_args(self, *args, **kws)
    399                 e.patch_message(msg)
    400 
--> 401             error_rewrite(e, 'typing')
    402         except errors.UnsupportedError as e:
    403             # Something unsupported is present in the user code, add help info

/srv/conda/envs/pangeo/lib/python3.7/site-packages/numba/dispatcher.py in error_rewrite(e, issue_type)
    342                 raise e
    343             else:
--> 344                 reraise(type(e), e, None)
    345 
    346         argtypes = []

/srv/conda/envs/pangeo/lib/python3.7/site-packages/numba/six.py in reraise(tp, value, tb)
    666             value = tp()
    667         if value.__traceback__ is not tb:
--> 668             raise value.with_traceback(tb)
    669         raise value
    670 

TypingError: Failed in nopython mode pipeline (step: nopython frontend)
non-precise type pyobject
[1] During: typing of argument at <extend_cpu> (4)

File "<extend_cpu>", line 4:
<source missing, REPL/exec in use?>

This error may have been caused by the following argument(s):
- argument 4: cannot determine Numba type of <class 'dask.array.core.Array'>

I recognize that the best place for this issue might be in another repo, but I thought I would try here first.

poplarShift commented 4 years ago

Here another MWE without intake/google cloud storage.

import xarray as xr
import hvplot.xarray
ds = xr.tutorial.open_dataset('rasm', chunks={'x': 25, 'y': 25, 'time': -1})
ds = ds.assign_coords(x=ds.x, y=ds.y)

import holoviews as hv
hv.extension('bokeh')

I think the error above boils down to:

# fails:
ds.Tair.isel(time=0).hvplot.quadmesh('x', 'y', rasterize=True)

# works
ds.Tair.isel(time=0).hvplot.image('x', 'y', rasterize=True)

As a workaround, you could shade a Points element (also allows irregular spacing). I know it's not the same, but maybe it helps you in the short term?

# works:
import datashader as dsh
from holoviews.operation.datashader import datashade
ds.hvplot.points('x', 'y', c='Tair', rasterize=True, x_sampling=1, y_sampling=1, aggregator=dsh.mean('Tair'))

# equivalent:
l = hv.DynamicMap(
    lambda t: hv.Points(ds.sel(time=t), ['x', 'y'], 'Tair'),
    kdims=hv.Dimension('time', values=ds.time.values),
)
datashade(l, x_sampling=1, y_sampling=1, aggregator=dsh.mean('Tair'))
philippjfr commented 4 years ago

I believe this should be fixed in the next release but @jonmmease or @jbednar can correct me.

philippjfr commented 4 years ago

So I was wrong, it seems we don't yet have dask quadmesh rendering support in datashader, for now I'll have to insert a compute sadly.

rabernat commented 4 years ago

Thanks for the update @philippjfr. My understanding was that quadmesh just underwent a big refactor to improve performance. Datashader quadmesh + dask is very important to the Pangeo project. @jbednar - I'm happy to see our NSF grant support such work.

philippjfr commented 4 years ago

Oh it turns out it was done by @jonmmease here, it was just never merged! We'll get that merged this weekend and hopefully get a release out by Monday.

jbednar commented 4 years ago

I'm happy to see our NSF grant support such work.

Indeed, your NSF grant did support this work (datashader PR #855). :-) We just need to track down the last few issues and release it. Sorry for the confusion!

philippjfr commented 4 years ago

Didn't think to look in the PRs, since I assumed such a great change would have been merged asap but I did have a vague recollection of Jon working on it so I was surprised. Anyway, the to-do list for a datashader release is pretty short and I'll do my best to tick off items tonight and tomorrow.

philippjfr commented 4 years ago

@rabernat I'm trying to test this but am running into this error:

ValueError: Bad Request: https://www.googleapis.com/download/storage/v1/b/pangeo-cmems-duacs/o/.zmetadata?alt=media
b'User project specified in the request is invalid.'
rabernat commented 4 years ago

The data are requester pays. So you'll need a google cloud account to pay the egress fees. @martindurant can probably advise. (xref https://github.com/pangeo-data/pangeo/issues/774)

philippjfr commented 4 years ago

Okay, I've tested with this example instead, which should be sufficient:

from intake import open_catalog
ds = xr.tutorial.load_dataset('rasm').Tair

ds.xc.values = da.from_array(ds.xc.values)
ds.yc.values = da.from_array(ds.yc.values)
ds.values = da.from_array(ds.values)

ds.hvplot.quadmesh('xc', 'yc', rasterize=True, dynamic=True)
philippjfr commented 4 years ago

Should be fixed now datashader 0.11 is released.

kthyng commented 3 years ago

I am trying to use quadmesh + datashader + dask which is requiring datashader to be at version 0.11 or higher because of the above problem.

But, I cannot have datashader be v0.11 because it requires that numba cannot be 0.49. or 0.50.

And I am using xgcm.grid.transform which requires that numba be version 0.49 or above.

Any suggestions for how to proceed from either the datashader or xgcm (@rabernat @jbusecke) side? Thanks.

philippjfr commented 3 years ago

Numba 0.51 should be available now. Could you try that?

kthyng commented 3 years ago

@philippjfr Yes! Thank you a newer version is available and it has fixed the conundrum. I had checked the numba release notes to see the newest version and they apparently aren't quite updated so I thought I was stuck at 0.50.*. Thanks again!