Wrong hover values on hv.operation.decimate when data is np.datetime64

[ludwigVonKoopa]

Hi all,

i created a mirror bugreport on discourse.holoviz.org I did not know where to explain my problem.

it seems thats with datetime64 dtype, holoviews operation break hover formatting (lon/lat projection, and datetime64 formating) :

import xarray as xr
import holoviews as hv
import geoviews as gv
import numpy as np
print("xarray    : ", xr.__version__)
print("holoviews : ", hv.__version__)
print("geoviews  : ", gv.__version__)

gv.extension("bokeh")

xarray    :  0.18.2
holoviews :  1.14.4
geoviews  :  1.9.1

dataset creation :

dates = np.arange(np.datetime64("2003-01-01"), np.datetime64("2003-01-31"))

ds = xr.Dataset(
    data_vars=dict(
        ssh=(["obs"], np.random.rand(dates.size)),
        time=(["obs"], dates),
    ),
    coords=dict(
        lon=(["obs"], np.linspace(-120, 120, dates.size)),
        lat=(["obs"], np.linspace(-80, 80, dates.size)),
    ),
)

ds

Dimensions:  (obs: 30)
Coordinates:
    lon      (obs) float64 -120.0 -111.7 -103.4 -95.17 ... 103.4 111.7 120.0
    lat      (obs) float64 -80.0 -74.48 -68.97 -63.45 ... 63.45 68.97 74.48 80.0
Dimensions without coordinates: obs
Data variables:
    ssh      (obs) float64 0.501 0.3243 0.01972 0.5372 ... 0.7374 0.1138 0.2572
    time     (obs) datetime64[ns] 2003-01-01 2003-01-02 ... 2003-01-30

Plot :

from holoviews.operation import decimate

pts = gv.Points(
    (ds.lon, ds.lat, ds.ssh, ds.time), 
    kdims=["lon", "lat"], 
    vdims=["ssh", "time"]
).opts(tools=["hover"], size=8)

pts.opts(title="without decimate") + decimate(pts).opts(title="with decimate")

expectation :

I would like to see time values in hover with proper formatting, and the longitude/latitude formatting not broken

Thanks all

[ludwigVonKoopa]

for clarifications, without np.datetime64 data, holoviews.operation.decimate works fine :

pts = gv.Points(
    (ds.lon, ds.lat, ds.ssh, ds.time), 
    kdims=["lon", "lat"], 
    vdims=["ssh"]   # changes : time values not plotted
).opts(tools=["hover"], size=8)

pts.opts(title="without decimate") + decimate(pts).opts(title="with decimate")

[philippjfr]

Thanks, for filing the issue. I believe this was resolved in https://github.com/holoviz/holoviews/pull/5039 and will be fixed as part of the 1.14.6 hotfix release in the coming days.

[philippjfr]

I was wrong this issue persists but only in GeoViews.

[ludwigVonKoopa]

Hi, i would like to know if this issue has any updates ?

I tried to investigate myself:

in the class decimate, even with a minimalist decimate class the problem still appear :

class decimate(Operation):
    dynamic = param.Boolean(default=True, doc="""
       Enables dynamic processing by default.""")

    link_inputs = param.Boolean(default=True, doc="""
         By default, the link_inputs parameter is set to True so that
         when applying shade, backends that support linked streams
         update RangeXY streams on the inputs of the shade operation.""")

    max_samples = param.Integer(default=5000, doc="""
        Maximum number of samples to display at the same time.""")

    random_seed = param.Integer(default=42, doc="""
        Seed used to initialize randomization.""")

    streams = param.ClassSelector(default=[RangeXY], class_=(dict, list),
                                   doc="""
        List of streams that are applied if dynamic=True, allowing
        for dynamic interaction with the plot.""")

    x_range  = param.NumericTuple(default=None, length=2, doc="""
       The x_range as a tuple of min and max x-value. Auto-ranges
       if set to None.""")

    y_range  = param.NumericTuple(default=None, length=2, doc="""
       The x_range as a tuple of min and max y-value. Auto-ranges
       if set to None.""")

    _per_element = True

    def _process_layer(self, element, key=None):
        return element

    def _process(self, element, key=None):
        return element

As with holoviews.Points this probleme doesn't appear, but with geoviews.Points yes, my main guess is that holoviews.operation.element.Operation is somehow bypassing the custom hover made by geoviews.

I found the declaration of geo hover in GeoPlot class and even a update hover function, but i lack some holoviews backend experience and did not find how to / when / who use this function.

How can i access the GeoPlot.handles dictionnary with CustomJSHover in it from an gv.Points object ?

I would happily help to correct this but do not fully understand how holoviews works in backend.

Thanks

[ludwigVonKoopa]

Hi, i have some update for this issue:

It looks like geoviews is re-creating the hover when using decimate.

Considere this :

pts = gv.Points(
    (ds.lon, ds.lat, ds.ssh, ds.time), 
    kdims=["lon", "lat"], 
    vdims=["ssh", "time"]
).opts(tools=["hover"], size=8)
fig_base = hv.render(pts)
fig_decimate = hv.render(decimate(pts))

import pprint
pprint.pprint(fig_base.tools[2].tooltips)
print()
pprint.pprint(fig_decimate.tools[2].tooltips)

[('lon', '$x{custom}'),
 ('lat', '$y{custom}'),
 ('ssh', '@{ssh}'),
 ('time', '@{time}{%F %T}')]

[('lon', '$x{custom}'),
 ('lat', '$y{custom}'),
 ('ssh', '@{ssh}'),
 ('time', '@{time}{custom}')]

We see that the tooltip for the datetime dimension is changed, and delete the %F %T format

To prevent this from happening, i only deleted the tag "hv_created" when gv create the hover at geoviews.plotting.bokeh.plot._postprocess_hover, which trigger the creation to be done again.

https://github.com/holoviz/geoviews/blob/1bd771233dfbd1208e86224f987c9003f121164a/geoviews/plotting/bokeh/plot.py#L153

...

                formatter += '{custom}'
            tooltips.append((name, formatter))
        hover.tooltips = tooltips
        hover.formatters = formatters
        hover.tags = list(set(hover.tags) - set(["hv_created"]))  # => this delete the tag, and then force rebuild

    def _update_hover(self, element):
        ...

WIth this patch, it doesn't change the tooltip:

pprint.pprint(fig_base.tools[2].tooltips)
print()
pprint.pprint(fig_decimate.tools[2].tooltips)

[('lon', '$x{custom}'),
 ('lat', '$y{custom}'),
 ('ssh', '@{ssh}'),
 ('time', '@{time}{%F %T}')]

[('lon', '$x{custom}'),
 ('lat', '$y{custom}'),
 ('ssh', '@{ssh}'),
 ('time', '@{time}{%F %T}')]

However, i don't have enough experience to know if this impact something else.. Any advices @philippjfr ? Do you think it will break other things, if yes, do you have some examples that i can run to test ?

Thanks