hv.save() yields increasing side margins in exported png when using datashader

[SebastianSchafer]

Hi, first, thanks for providing an awesome library. It seems the export feature using hv.save seems broken, though I'm not sure whether this is a holoviews or bokeh issue.

ALL software version info

bokeh 2.1.0 chromedriver 2.24.1 jupyterlab 2.1.4 numpy 1.18.5 pandas 1.0.4 panel 0.9.5 Pillow 7.1.2 selenium 3.141.0

python 3.7.4 macos 10.15.5 chrome 83

Description of expected behavior and the observed behavior

When plotting data from a number of dataframes and exporting them to .png files, I noticed that the margin at the sides of the plot is increasing every time a new plot is exported. See a simplified example below.

Observations, some might not be directly related:

• This seems only to happen with datashader=True
• The last time I remember this working was beginning of this year. I do not know which libraries I updated since then, sorry.
• Currently, hv.save() quietly saves as .html even with extension .png both in the path and in fmt='png' defined; this might be related to PR #4304 though I'm not sure
• Also noticed that the version using the hv.Store.renderers was quietly saving as html when chromedriver was not installed - not sure if I should file this as an issue?
• When trying to figure out which option will size the plot when exported to png (something I still don't fully understand), I noticed a mention (here) that the canvas for gridplots changed sometime around boke 2.0 - these threads are on the bokeh export_png function
• I did try inserting hv.output(size=200) in the loop to reset the plot sizing, but that does not affect the outcome
• So far the only way I found to reset the margins/sizing was to restart the notebook kernel

Complete, minimal, self-contained example code that reproduces the issue

import numpy as np
import pandas as pd
from time import time
import holoviews as hv
import datashader as ds
import hvplot.pandas
from holoviews.operation.datashader import datashade, dynspread

df = pd.DataFrame()
df['x'] = np.arange(100)
df['y'] = np.random.rand(len(df.x))
df['c1'] = df.x // 10
df['c1'] = df.x // 50

for i in range(9):
    plot = df.hvplot(x='x', y='y', row='c1', datashade=True, dynspread=True).opts(toolbar=None)

    renderer = hv.Store.renderers['bokeh'].instance(fig='png', holomap='widgets')
    renderer.save(plot, f'plot{i}.png')
#     hv.save(plot, f'plot_save_{i}.png', fmt='png') # currently ouputs .html w/o warning

Screenshots or screencasts of the bug in action

First and last plot result from that loop:

plot when not using datashader, does not change during loop:

[jbednar]

That seems very strange and worrisome!

[SebastianSchafer]

Had some time to check this on my Ubuntu machine with above example. Getting different results, but still see difference depending on whether datashader is True or not.

• Plot w/o datashader looks looks almost the same as above example w/o ds - difference is that now it actually plots w/o the toolbar. This was another issue I had (though not in the past), but forgot when trying to get the scaling/margin to work: I tried to put toolbar=None in a bout a dozen different .opts() places, but it would still save with toolbar present.
• Using datashader, the margin does not change during the loop, but the aspect ratio and margin do not look right (the png has 1366x277 pixels), looking similar to what I'd get after several plots in the macos environment.

• The environment is slightly different: bokeh 2.1.0 geckodriver 0.26.0 jupyterlab 2.0.1 numpy 1.18.1 pandas 1.0.2 panel 0.9.6 Pillow 7.0.0 selenium 3.141.0

  python 3.7.4 Ubuntu 18.04 Firefox 77

Again, not sure if this is holoviews related, or whether this could be selenium or bokeh?

plot with datashader (does not change when looping):

Without datashader, same as on macos but w/o toolbar (As intended):

[jlstevens]

I agree it is very bizarre that this only happens when datashader=True!

@philippjfr I'm going to assign this to the 1.13.x milestone, what do you think?

[SebastianSchafer]

Just a minor update: I tried older bokeh releases (as panel 0.9.5+ stopped working for me in another script with bokeh 2.1). There was no change going down to 2.0, but using bokeh 1.4.0 (with panel 0.8.1, though that should not matter here), changes the behavior. The background is transparent, and while the margin is still very large, it does not change during loop. I have to check whether with setting the sizing I can remove the large margin. All other libraries are the same, so that should exclude changes in selenium or others (as I was not sure when I updated those).

• I'm not familiar with bokeh itself, so haven't tested this natively in bokeh; if someone could check a similar script using bokeh directly we could see whether to file this with bokeh directly or if it's an issue with the changed bokeh 2.0 API?
• Somewhat off-topic: how do I best debug a panel servable in a notebook? This is for another issue (mentioned above), where panel just stopped showing neither the panel nor a warning/error, even with pn.config.console_output = 'accumulate' explicitly set to print all output.

[jlstevens]

I am confident the datashader version is irrelevant here but the bokeh export machinery could be the cause. Interestingly, trying your self-contained example with bokeh 1.4.0 doesn't look like particularly large margins to me:

However with a new environment using bokeh 2.1.1 what I see are very large margins though they aren't expanding:

That said, one of the plots seems to be shifted relative to the rest (some sort of jitter is happening).

Here are the versions I used:

hv.__version__, bokeh.__version__, selenium.__version__
('1.13.2', '2.1.1', '3.141.0')

Based on this, I agree there is an issue and the export quality can be improved.

[marcbernot]

It took some time to understand what was going on! Here are the elements I got.

Ultimately this behaviour comes from the fact that bokeh does not handle well a spacer with _stretchwidth sizing mode. This can't really be considered to be a bokeh bug though. Indeed, there is a warning in the get_screenshot_as_png bokeh function that says _Responsive sizingmodes may generate layouts with unexpected size and aspect ratios. It is recommended to use the default fixed sizing mode.

Here is a minimal bokeh code to reproduce the strange increasing size of the pngs.

from bokeh.io.export import export_png
from bokeh.layouts import Spacer, Row
from PIL import Image

filename = 'plot.png'
sp = Spacer(margin=[5,5,5,5],sizing_mode='stretch_width')
row = Row(sp)

for k in range(3):
    export_png(row,filename=filename)
    print(Image.open(filename).size)

So where do these spacers come from in the first place? This is not related to datashader per se but rather related to hv.DynamicMap. Indeed png exports are correct with hv.Curve(df[['x','y']]) but incorrect with hv.DynamicMap(hv.Curve(df[['x','y']])).

By following the code trail, it appears that : hv.DynamicMap => center = True in holoviews/plotting/renderer.py/_validate code => layout = [HSpacer(), self, HSpacer()] in panel/pane/holoviews.py/_update_layout => bokeh spacer "bug" since the default sizing mode for HSpacer is _stretchwidth

Here is the code to check this:

import numpy as np
import pandas as pd
import holoviews as hv
hv.extension('bokeh')
from PIL import Image

n_samples = 100
filename = 'plot.png'
df = pd.DataFrame(dict(x = np.arange(n_samples),
                       y = np.random.rand(n_samples)))

renderer = hv.renderer('bokeh')
h = hv.DynamicMap(hv.Curve(df))
plot, fmt = renderer._validate(h,'png')
print(plot.layout)

Where should this bug be fixed without breaking legit use cases?

Maybe by changing this test

elif dynamic or (self._render_with_panel and fmt == 'html')

that results in toggling center=True for the holoviews pane.

I don't see the logic behind why a hv.DynamicMap should result in a center=True holoviews pane but I don't have a clear view of the current use cases that could be broken by changing this line.

Waiting for a fix, a quick workaround is simply to create yourself the panel holoviews pane and save it.

import panel as pn
pn.pane.HoloViews(plot).save(filename)

[Tsarpf]

Thanks for the workaround! Helped as I'm exporting a ton of graphs as PNG, and the size kept growing.

However, the save function in pn.pane.HoloViews(plot).save(filename) does not support setting DPI?

[cdeciampa]

The large white space still appears to be an issue, but using hv.save(center=False) appears to fix it for me. I never had increasingly large white space like you, just comically large white space on the sides. I should note that I use Datashader exclusively for rasterized plots.

This is my environment info:

holoviews=11.15.3
bokeh=2.4.3
python=3.8.15
selenium=4.7.2

[d33bs]

Thanks all for the discussion here and for the workarounds. I ended up using @cdeciampa's solution (thank you!) regarding hv.save(center=False) to workaround the very large widths. For my scenario I found that when building multiple plots in a single module each one sequentially had larger and larger widths in the exports.