Make Bokeh objects always appear in overlay order

[jbednar]

The HoloViews * syntax for constructing overlays has a clear left-to-right interpretation. In an expression like e1 * e2 * e3, e1 is normally drawn first, then overlaid with e2, then with e3 on top.

However, at least with the Bokeh backend, this order is not always respected. E.g. GeoViews map tiles always appear underneath any plotted data, which sometimes is what you want (for a tile layer showing geographic context), but is often not what you want (for a tile layer showing geographic place names, which should not usually be obscured by data points).

I propose that we eliminate any cases where displayable items appear at any order other than that specified in the overlay expression, unless the user has explicitly manipulated a special parameter (e.g. modifying the "level" parameter of a Bokeh object to bump it up or down in draw order). Specifically, from what @philippjfr has described, it seems like the WMTS element in GeoViews should be declared to appear at glyph level by default, to match other HoloViews objects so that sorting will be determined by overlay order. But I've raised the issue here rather than in GeoViews because it applies to all displayable items in both HoloViews and GeoViews; in my opinion if any of them (even Annotations) appear at some different order than specifed in the overlay, it's a bug in HoloViews according to the semantics of Overlay.

Does that sound correct to everyone?

[jlstevens]

I agree the order of overlay should be respected everywhere if possible though there are some issues to think about:

• Is there any valid reason for annotations not to always appear on the top?
• I believe Bokeh doesn't have a completely straightforward z-ordering model like matplotlib does: as I understand it 'renderers' get ordered.

Other than these two caveats, I agree that the overlay ordering should be the z-ordering whenever possible.

[jbednar]

Is there any valid reason for annotations not to always appear on the top?

Sure. Just because we call them annotations doesn't mean that they will always be used as such. E.g. someone could use splines, circles, etc. to draw some crazy grid lines:

and want their data to go on top of that, as it's background information, not plot elements. I don't think it's up to us to decide whether that's useful; it's much simpler just to layer things in order by default.

[jlstevens]

I agree the utility of having annotations underneath other elements does vary. For instance, while I can imagine wanting a box/ellipse in the background, I can't think of any reason you would you ever want something to render over a text or arrow annotation...

[philippjfr]

I'd suggest we make all elements default to the glyph render level but also expose the level as a plot/style option to ensure we have the flexibility in certain cases. The easiest thing would be to simply make it a style option.

[jbednar]

That sounds perfect.

[jlstevens]

I'm not sure that is ideal. Shouldn't HoloViews handle the z-order based on the overlay definition like @jbednar initially suggested? If that can work properly, I would rather do that than expose another plot/style option.

[jbednar]

The proposal is to handle it based on the overlay definition, by default. That's the way we expect most people to use it. But it does seem useful to provide a style option that lets the user force some Element types to go on top or bottom, which could be very convenient for some workflows. Optional, but seems handy!

[jlstevens]

I would rather keep the semantics clear if we can and introduce additional options only if we know they are necessary.

[jbednar]

I don't think there is any semantic problem here. For each render level, Overlays are constructed in the order the Elements are specified, left to right. All Elements default to the glyph render_level, but if you want to force some Element types to be in front of the others, you can set that Element's render_level to annotation. If you want to force some Element types to be behind others, you can set the render_level to underlay. Seems clear to me, though I wouldn't cry if it were not available.

That said, aren't some of the render_levels special, in that they are not cropped to the viewport? If so having the ability to set the level explicitly will surely eventually be required, not just as a convenience.

[philippjfr]

That said, aren't some of the render_levels special

That's correct, I think the overlay level lets you draw outside the bounds.

[jbednar]

Drawing outside the bounds is a very important capability in some cases, and so I think the combined proposal (all at glyph level by default, style option to set explicit level) is appropriate.

That said, I think Jean-Luc may have already been exploiting the ability of Annotations to appear outside of the axis boundaries, e.g. to fake multi-line titles? If so, I guess there would be backwards compatibility implications of changing Annotations to use glyph level, as they would now require this style option? I'm personally ok with that, as I think the ability to draw outside of the axis boundaries is sometimes useful but can be very surprising and is probably more often a bug (e.g. when the position is calculated from a function), so I think I still come down in favor of glyph level for all by default.

[philippjfr]

I doubt that a little bit, the Text element is at the glyph level already unless I'm completely mistaken.

[jbednar]

I hope that's true, but I did see Jean-Luc make a plot a couple of days ago with items I think he said were hv.Text appearing outside the plot boundaries...

[philippjfr]

Doesn't work for me:

hv.Curve(range(10)) * hv.Text(0, 11, 'A')

[jbednar]

Nor me:

@jlstevens?

[jlstevens]

I hope that's true, but I did see Jean-Luc make a plot a couple of days ago with items I think he said were hv.Text appearing outside the plot boundaries...

The text wasn't outside the plot boundaries, it just had the y-axis disabled.

[jbednar]

Ah! You fooled me. :-) Ok, then I stand by my proposal to have all Elements use glyph type consistently now.

[philippjfr]

If we're actually worried about the semantics we could also just add a clip_bounds option (or similar) rather than providing full control over the render levels. I do think we should expose one of these options though.

[jbednar]

I have no preference between those two alternatives, and support either one. They each have advantages; clip_bounds is a simpler, clearer concept, and meaningful outside of Bokeh, but render_level is more powerful and exposes more of the Bokeh machinery to those who want that. Either is fine by me.

[jlstevens]

Whatever we decide, please let's make sure to keep improvements to the current behavior (i.e respecting the ordering as specified in the overlay) separate from any new plot options. We all agree with the former so such a PR won't need much discussion whereas new plot options will need more thought.

[philippjfr]

I'm not actually aware of any elements in holoviews itself that are not at the glyph level at this point.

[jlstevens]

Ok, so the original issues really only applies to geoviews.

[philippjfr]

Appears what I said is not quite true, I noticed Bars are generally on top of other elements. That should be fixed.

[Dr-Irv]

I'm just learning HoloViews and doing the tutorial from pyviz. I've noticed the issue with bars (and area) being on top. For example, in the notebook https://github.com/pyviz/pyviz/blob/master/notebooks/02_Annotating_Data.ipynb , if you do the exercise in the middle that is listed as # Exercise: Make an overlay of the Spikes object from layout on top of the filled trajectory area of labelled_layout, and then try something like hv.Area(trajectory) * hv.Spikes(trajectory), the spikes don't show.

But you can make the spikes show by using the fill_alpha=0.5 option for Area.

So now the question is whether it is the responsibility of the user to specify the alpha/transparency values correctly when doing overlays, rather than assuming some particular drawing order.

Being VERY new to all of this, I may be totally misunderstanding things, but I thought I would add my observation.

[philippjfr]

No your intuition is right here, something is off about the Area zorder in this example. This should however already be resolved by this recent fix.

[jbednar]

@philippjfr, how can we change the default level of tile sources to 'glyph', and find out if anything other than tile sources has an inappropriate level? I can see references to 'level' in TilePlot._init_glyph, but it's not setting the value to anything by default. Presumably it's inherited from Bokeh, so I can't tell (a) how to change it by default and (b) how to detect what else might inherit a non-glyph default.

[philippjfr]

Presumably it's inherited from Bokeh, so I can't tell (a) how to change it by default and (b) how to detect what else might inherit a non-glyph default.

Two options, we can either set level as a default style option on WMTS, or we could have WMTSPlot override the bokeh default if no custom value is supplied. I've never quite worked out which I prefer.

[jbednar]

Doing it on WMTSPlot seems more suitable to me, because this is a very Bokeh-specific option and nothing to do with the general notion of WMTS objects. Would you be able to make a PR for that? I still don't know how to address (b).

[philippjfr]

I still don't know how to address (b).

You could write a little script to crawl the bokeh API and make a list of the default level settings for different glyphs.

[jbednar]

I did look at that briefly, but a naive grep-based approach won't work, as "level" is used for logging and other purposes, so it would need to be a fairly sophisticated script.

[tzuni]

I just ran into possibly a related issue but with the Matplotlib backend. When you make an overlay with more than three curves and then a horizontal or vertical line the fourth curve and beyond are drawn on top of the HLine or VLine thus breaking the layout order.

An example:

import numpy as np
import holoviews as hv
hv.extension('matplotlib')

one = hv.Curve([(0.1*i, np.sin(0.1*i)) for i in range(100)])
two = hv.Curve([(0.1*i, np.sin(0.1*i+0.75)) for i in range(100)])
three = hv.Curve([(0.1*i, np.sin(0.1*i+1.5)) for i in range(100)])
four = hv.Curve([(0.1*i, np.sin(0.1*i+2.25)) for i in range(100)])
five = hv.Curve([(0.1*i, np.sin(0.1*i+3.0)) for i in range(100)])
vline = hv.VLine(4.6).opts(color='black')
hline = hv.HLine(0.0).opts(color='purple')
(one * two * three * four * five * vline * hline).opts(
    hv.opts.Curve(linewidth=5),
    hv.opts.VLine(linewidth=5),
    hv.opts.HLine(linewidth=5))

Curves one, two, and three are drawn below the hline and vline but curves four and `five are drawn over them.

I'm working in linux on Python 3.7.5 with these versions: holoviews==1.12.7 matplotlib==3.1.2

[brunorpinho]

Just in case anyone else is confused.

Using the bokeh backend I was having an issue of a slope line that was supposed to be plotted on top of a datashaded scatterplot being plotted under it, so I couldn't see it.

I solved it by passing the level to the slope opts doing:

hv.Slope(1, 0).opts(level='overlay') * datashade_image

More info on level

[kitaev-chen]

I solved it by passing the level to the slope opts doing:

hv.Slope(1, 0).opts(level='overlay') * datashade_image

errorbars = hv.ErrorBars(pd.concat([x, y_mean, y_std], axis=1)).opts(level='overlay')

ValueError: Unexpected option 'level' for ErrorBars type across all extensions. No similar options found.

... ...

I have to say, holoviews is super unfriendly to the people who know little about the various parameters/opts of boken/matplot etc. Usually it will cost a lot of time to find the setting for a very simple function

[jbednar]

@brunorpinho: Using the bokeh backend I was having an issue of a slope line that was supposed to be plotted on top of a datashaded scatterplot being plotted under it, so I couldn't see it. I solved it by passing the level to the slope opts doing: hv.Slope(1, 0).opts(level='overlay') * datashade_image

Here the problem wasn't the level, but the ordering. The HoloViews way of fixing the ordering is not by trying to mess with the level, but simply by putting items in the order you want them to overlay, left to right (bottom to top in the overlay):

datashade_image * hv.Slope(1, 0)

@kitaev-chen, maybe the Bokeh ErrorBars element doesn't support a level option, but that shouldn't matter if so, because the correct way to handle overlays in HoloViews in general is simply to order them left to right.

I agree that using backend-specific options (when they are needed!) could be a lot easier; see my proposal in https://github.com/holoviz/holoviews/issues/1820. We only lack time to implement it! If anyone has time and expertise (or money so we can hire someone with time and expertise) to do this, please let me know!

[kitaev-chen]

@kitaev-chen, maybe the Bokeh ErrorBars element doesn't support a level option, but that shouldn't matter if so, because the correct way to handle overlays in HoloViews in general is simply to order them left to right.

I agree that using backend-specific options (when they are needed!) could be a lot easier; see my proposal in #1820. We only lack time to implement it! If anyone has time and expertise (or money so we can hire someone with time and expertise) to do this, please let me know!

That's what bothers me a whole day. You can see the ErrorBars works, but when composing with bar chart, half of the errorbars was blocked no matter what order.

fig = errorbars

fig = bars * errorbars

[jlstevens]

With bars * errorbars, the error bars should definitely be on top! Not sure what is going on in that screenshot but is looks like a bug to me. @philippjfr ?

I have to say, holoviews is super unfriendly to the people who know little about the various parameters/opts of bokeh/matplot etc. Usually it will cost a lot of time to find the setting for a very simple function

In the FAQ, see the Q: Why don’t you let me pass matplotlib_option as a style through to matplotlib? question which also applies to Bokeh. You should be able to make level available if you really want to, but as @jbednar says, the order of * should be respected (though not in your screenshot for some reason!).

[kitaev-chen]

@jlstevens

fig = bars * errorbars is the return of the single_fig function, then I did:

fig_dict_2D = OrderedDict({(rval, cval):single_fig(...) for rval in rvals for cval in cvals})
grid = hv.GridSpace(fig_dict_2D, kdims=[rlabel, clabel], sort=False)
ndlayout = hv.NdLayout(grid, kdims=[rlabel, clabel], sort=False)
nd_fig = ndlayout.opts(opts.Bars(width=250, height=200)).cols(n_cols)
show(hv.render(nd_fig))

But I try to show the single plot, the errorbars is still blocked by bar chart

bars = hv.Bars(
        df_draw, [xft], [yft1]).opts(
            opts.Bars(
                ylim=(0, 1.2),
                bar_width=0.5, 
                color=xft,
                show_legend=False,
                xlabel = xlabel, 
                ylabel = ylabel1, 
                cmap=colors, 
                )).opts(
                    fontsize={
                        'xticks': 7, 
                    },
                )

    errorbars = hv.ErrorBars(
        pd.concat([x, y_mean, y_std], axis=1)
        )#.opts(level='overlay')

fig = bars * errorbars

[kitaev-chen]

In the FAQ, see the _Q: Why don’t you let me pass matplotliboption as a style through to matplotlib? question which also applies to Bokeh. You should be able to make level available if you really want to, but as @jbednar says, the order of * should be respected (though not in your screenshot for some reason!).

Finally it works.

from holoviews import Store
Store.add_style_opts(hv.Bars, ['level'], backend='bokeh')

hv.Bars(...)
.opts(
                    fontsize={
                        'xticks': 7, 
                    },
                    level='image',
                )

Many thanks! You save me lots of time!

[brunorpinho]

@jbednar What is the purpose of level? I'm asking because I use it all the time for ordering, haha

[jbednar]

Bokeh is independent of HoloViews and uses levels for its own purposes, and I don't think there's any compelling reason to set the level in HoloViews except to fix cases where the level was set incorrectly internally (which is the problem this issue is about). Named levels like Bokeh has only give you a very rough control over ordering, and you have to remember what order the names go in, which seems much less clear than just putting the expression in the order you want. I'd really recommend that people only mess with levels if they find a bug where the HoloViews left to right (bottom to top) ordering is not being respected.

[brunorpinho]

Bokeh is independent of HoloViews and uses levels for its own purposes, and I don't think there's any compelling reason to set the level in HoloViews except to fix cases where the level was set incorrectly internally (which is the problem this issue is about). Named levels like Bokeh has only give you a very rough control over ordering, and you have to remember what order the names go in, which seems much less clear than just putting the expression in the order you want. I'd really recommend that people only mess with levels if they find a bug where the HoloViews left to right (bottom to top) ordering is not being respected.

Got it! Thanks

[jbednar]

Here's another case where the ordering has to be overridden with level; not sure why:

import panel as pn, hvplot.pandas, holoviews as hv, pandas as pd, colorcet as cc

df = pd.DataFrame(dict(x=[6E6,5E6,4E6,3E6,2E6,1E6], 
                       y=[2E6,1E6,3E6,4E6,0E6,5E6], val=[7,8,9,7,8,9]))
dfi = df.interactive

map_tiles = hv.element.tiles.EsriImagery().opts(level='underlay')

value = pn.widgets.Select(options=list(df.val.unique()), name='Value')

map_tiles * dfi[dfi.val==value].hvplot('x','y', kind='scatter')

(the scatter points are obscured by the map if .opts(level='underlay') is omitted). I strongly vote for all Bokeh elements to use the same level to avoid this problem!!!!!!!!!

[philippjfr]

That one is pretty bizarre, don't know how that would even happen.

[jbednar]

Bad things happen when we let there be multiple levels anywhere in the system. :-)

[TheoMathurin]

+1 on this one

It seems the issue with Errorbars is not limited to overlays with hv.Bars, other elements like hv.Curve are also affected.

errors = [(0.1*i, np.sin(0.1*i), np.random.rand()/2) for i in np.linspace(0, 100, 11)]
hv.Curve(errors).opts(line_width=5) * hv.ErrorBars(errors)

Solved it using hv.Store.add_style_opts(hv.ErrorBars, ['level'], backend='bokeh') + hv.ErrorBars(...).opts(level='overlay')

[G-Guillard]

hv.Slope also ignores natural ordering for overlay. It has already be mentioned by @brunorpinho in combination with a Datashader plot, but can be generalized with other kinds of plots, and from my experience (hv 1.17.1, bokeh 3.1.1) the slope is usually above the other plot(s).

(Using level="underlay" as suggested above does the trick.)