Rasterize not working in combination with multiple rectangles with DateTime coordinates

StevenWarmelink commented 2 years ago

ALL software version info

pandas==1.3.5 holoviews==1.14.8 bokeh==2.3.3 python==3.8.5

Description of expected behavior and the observed behavior

When rasterizing multiple hv.Rectangles, I expect to be able to supply Timestamp coordinates; this works fine without the rasterization or for single Rectangles. However, when multiple Rectangles with Timestamp coordinates are rasterized, the supplied timestamps are out of bounds.

EDIT: This also occurs when a single rectangle is supplied in a list when calling hv.Rectangles.

Complete, minimal, self-contained example code that reproduces the issue

from holoviews.operation.datashader import rasterize
import pandas as pd
import holoviews as hv
import bokeh 
hv.extension("bokeh",logo=False)

dts         = pd.to_datetime(['1.15.2019', '2.15.2019'])
rectangle_1 = (dts[0], 0, dts[1], 0.3)
rectangle_2 = (dts[0], 0.5, dts[1], 0.9)
rects       = hv.Rectangles([rectangle_1, rectangle_2])
rast_rects  = rasterize(rects)
display(rast_rects)

Stack traceback and/or browser JavaScript console output

WARNING:param.dynamic_operation: Callable raised "OutOfBoundsDatetime('Out of bounds nanosecond timestamp: 51008-08-27 00:00:00')".
Invoked as dynamic_operation(height=400, scale=1.0, width=400, x_range=(numpy.datetime64('2019-01-15T00:00:00.000000000'), numpy.datetime64('2019-02-15T00:00:00.000000000')), y_range=(0.1, 0.9))
WARNING:param.dynamic_operation: Callable raised "OutOfBoundsDatetime('Out of bounds nanosecond timestamp: 51008-08-27 00:00:00')".
Invoked as dynamic_operation(height=400, scale=1.0, width=400, x_range=(numpy.datetime64('2019-01-15T00:00:00.000000000'), numpy.datetime64('2019-02-15T00:00:00.000000000')), y_range=(0.1, 0.9))
Traceback (most recent call last):
  File "/home/steven/anaconda3/lib/python3.8/site-packages/holoviews/plotting/util.py", line 277, in get_plot_frame
    return map_obj[key]
  File "/home/steven/anaconda3/lib/python3.8/site-packages/holoviews/core/spaces.py", line 1344, in __getitem__
    val = self._execute_callback(*tuple_key)
  File "/home/steven/anaconda3/lib/python3.8/site-packages/holoviews/core/spaces.py", line 1111, in _execute_callback
    retval = self.callback(*args, **kwargs)
  File "/home/steven/anaconda3/lib/python3.8/site-packages/holoviews/core/spaces.py", line 708, in __call__
    ret = self.callable(*args, **kwargs)
  File "/home/steven/anaconda3/lib/python3.8/site-packages/holoviews/util/__init__.py", line 1043, in dynamic_operation
    key, obj = resolve(key, kwargs)
  File "/home/steven/anaconda3/lib/python3.8/site-packages/holoviews/util/__init__.py", line 1032, in resolve
    return key, map_obj[key]
  File "/home/steven/anaconda3/lib/python3.8/site-packages/holoviews/core/spaces.py", line 1344, in __getitem__
    val = self._execute_callback(*tuple_key)
  File "/home/steven/anaconda3/lib/python3.8/site-packages/holoviews/core/spaces.py", line 1111, in _execute_callback
    retval = self.callback(*args, **kwargs)
  File "/home/steven/anaconda3/lib/python3.8/site-packages/holoviews/core/spaces.py", line 708, in __call__
    ret = self.callable(*args, **kwargs)
  File "/home/steven/anaconda3/lib/python3.8/site-packages/holoviews/util/__init__.py", line 1044, in dynamic_operation
    return apply(obj, *key, **kwargs)
  File "/home/steven/anaconda3/lib/python3.8/site-packages/holoviews/util/__init__.py", line 1036, in apply
    processed = self._process(element, key, kwargs)
  File "/home/steven/anaconda3/lib/python3.8/site-packages/holoviews/util/__init__.py", line 1018, in _process
    return self.p.operation.process_element(element, key, **kwargs)
  File "/home/steven/anaconda3/lib/python3.8/site-packages/holoviews/core/operation.py", line 194, in process_element
    return self._apply(element, key)
  File "/home/steven/anaconda3/lib/python3.8/site-packages/holoviews/core/operation.py", line 141, in _apply
    ret = self._process(element, key)
  File "/home/steven/anaconda3/lib/python3.8/site-packages/holoviews/operation/datashader.py", line 1514, in _process
    element = element.map(op, predicate)
  File "/home/steven/anaconda3/lib/python3.8/site-packages/holoviews/core/data/__init__.py", line 207, in pipelined_fn
    result = method_fn(*args, **kwargs)
  File "/home/steven/anaconda3/lib/python3.8/site-packages/holoviews/core/data/__init__.py", line 1223, in map
    return super(Dataset, self).map(*args, **kwargs)
  File "/home/steven/anaconda3/lib/python3.8/site-packages/holoviews/core/dimension.py", line 710, in map
    return map_fn(self) if applies else self
  File "/home/steven/anaconda3/lib/python3.8/site-packages/holoviews/core/operation.py", line 214, in __call__
    return self._apply(element)
  File "/home/steven/anaconda3/lib/python3.8/site-packages/holoviews/core/operation.py", line 141, in _apply
    ret = self._process(element, key)
  File "/home/steven/anaconda3/lib/python3.8/site-packages/holoviews/operation/datashader.py", line 765, in _process
    df[x0d.name] = cast_array_to_int64(df[x0d.name].astype('datetime64[us]'))
  File "/home/steven/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 5815, in astype
    new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
  File "/home/steven/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 418, in astype
    return self.apply("astype", dtype=dtype, copy=copy, errors=errors)
  File "/home/steven/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 327, in apply
    applied = getattr(b, f)(**kwargs)
  File "/home/steven/.local/lib/python3.8/site-packages/pandas/core/internals/blocks.py", line 593, in astype
    new_values = maybe_coerce_values(new_values)
  File "/home/steven/.local/lib/python3.8/site-packages/pandas/core/internals/blocks.py", line 1878, in maybe_coerce_values
    values = ensure_wrapped_if_datetimelike(values)
  File "/home/steven/.local/lib/python3.8/site-packages/pandas/core/construction.py", line 439, in ensure_wrapped_if_datetimelike
    return DatetimeArray._from_sequence(arr)
  File "/home/steven/.local/lib/python3.8/site-packages/pandas/core/arrays/datetimes.py", line 336, in _from_sequence
    return cls._from_sequence_not_strict(scalars, dtype=dtype, copy=copy)
  File "/home/steven/.local/lib/python3.8/site-packages/pandas/core/arrays/datetimes.py", line 355, in _from_sequence_not_strict
    subarr, tz, inferred_freq = sequence_to_dt64ns(
  File "/home/steven/.local/lib/python3.8/site-packages/pandas/core/arrays/datetimes.py", line 2096, in sequence_to_dt64ns
    data = conversion.ensure_datetime64ns(data)
  File "pandas/_libs/tslibs/conversion.pyx", line 256, in pandas._libs.tslibs.conversion.ensure_datetime64ns
  File "pandas/_libs/tslibs/np_datetime.pyx", line 120, in pandas._libs.tslibs.np_datetime.check_dts_bounds
pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 51008-08-27 00:00:00

Screenshots or screencasts of the bug in action

hoxbro commented 2 years ago

If I change the following lines from us to nsor remove the datetime conversion, I can get the code to work.

https://github.com/holoviz/holoviews/blob/0defcef994d6dd6d2054f75a0e332d02d121f8b0/holoviews/operation/datashader.py#L769-L774

maximlt commented 2 years ago

Thanks for looking into this @Hoxbro !

From what I've seen after some debugging geom_aggregate._process is called twice, the first time it updates the dataframe referenced aselement.data by converting the relevant datetime columns to int, the second time it tries to do so again but fails.

Questions are:

Are the two calls to geom_aggregate._process expected?
Should an operation be allowed to transform element.data? I find that somewhat surprising.

I could fix the issue by adding these two lines above the failing code.

        if xtype == 'datetime' or ytype == 'datetime':
            df = df.copy()

maximlt commented 2 years ago

The traceback reported in this recent blog post is very similar to the one reported in this issue. They have the same root cause and will certainly have the same fix. Specifically the blog post is plotting a QuadMesh, and the issue lies in quadmesh_rasterize instead of geom_aggreate.

holoviz / holoviews