export_png not exporting figures correctly

[jnettels]

This is a continuation of #7621

export_png (and export_svgs for that matter) do not work as expected when attempting to export figures after they were added to a layout curdoc().

Here is the example code:

# -*- coding: utf-8 -*-
'''
After the layout has been added to the root, exporting the png of a fig is
not possible anymore
With "bokeh serve export_test.py", the console output for me is the following:
doc_layout works
fig works
Layout is added to root
doc_layout works
fig throws error
'''
from bokeh.io import curdoc
from bokeh.models import ColumnDataSource
from bokeh.plotting import figure
from bokeh.layouts import layout, widgetbox
from bokeh.io import export_png
from bokeh.models.widgets import Button

def download():
    try:
        export_png(doc_layout, filename="export doc_layout.png")
        print('doc_layout works')
    except Exception as ex:
        print('doc_layout throws error:', ex)
    try:
        export_png(fig, filename="export fig.png")
        print('fig works')
    except Exception as ex:
        print('fig throws error:', ex)

source = ColumnDataSource({'x': [2, 3, 4], 'y': [0.5, 1.5, 2.5]})
fig = figure(plot_width=1000, plot_height=300)
fig.hbar(y='y', right='x', height=0.4, source=source)

button = Button(label="Download")
button.on_click(download)
doc_layout = layout([[widgetbox(button)], [fig]], sizing_mode='fixed')

#download()  # Test 1

print('Layout is added to root')
curdoc().add_root(doc_layout)
curdoc().title = "Export PNG test"

#download()  # Test 2

I start the server with bokeh serve export_test.py. After I click the Download button, the following output is generated:

You are generating standalone HTML/JS output, but trying to use real Python
callbacks (i.e. with on_change or on_event). This combination cannot work.

Only JavaScript callbacks may be used with standalone output. For more
information on JavaScript callbacks with Bokeh, see:

    http://bokeh.pydata.org/en/latest/docs/user_guide/interaction/callbacks.html

Alternatively, to use real Python callbacks, a Bokeh server application may
be used. For more information on building and running Bokeh applications, see:

    http://bokeh.pydata.org/en/latest/docs/user_guide/server.html

doc_layout works
fig throws error: Message: {"errorMessage":"undefined is not an object (evaluating 'document.getElementsByClassName('bk-root')[0].children[0].getBoundingClientRect')","request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"171","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:62171","User-Agent":"selenium/3.12.0 (python linux)"},"httpVersion":"1.1","method":"POST","post":"{\"script\": \"\\nreturn document.getElementsByClassName('bk-root')[0].children[0].getBoundingClientRect()\\n\", \"args\": [], \"sessionId\": \"5f2d8590-7620-11e8-94ca-bf888776a2b5\"}","url":"/execute","urlParsed":{"anchor":"","query":"","file":"execute","directory":"/","path":"/execute","relative":"/execute","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/execute","queryKey":{},"chunks":["execute"]},"urlOriginal":"/session/5f2d8590-7620-11e8-94ca-bf888776a2b5/execute"}}
Screenshot: available via screen

If I comment out the line download() # Test 1, the correct png is generated. This is only possible before curdoc().add_root(doc_layout). I still cannot save the png of a figure, once it has been included in a layout and added to root. Am I doing someting wrong? Does it have something to do with the new standalone HTML warning?

Software: Bokeh = 0.13.0 Tornado = 5.0.2 Selenium = 3.12.0 Pillow = 5.1.0

[bryevdv]

@AzraelDD to be clear, this is not really a continuation of #7621, and the problem has nothing to to with things being added to any layout.

The issue is that Bokeh 0.13.0 added an error condition and detailed exception when a user tries to use on_change in a standalone document (i.e. not in a Bokeh server app). The support burden of answering users who tried to do this and then asked "why doesn't on_change work?" had become entirely too heavy. This was an attempt to provide immediate, actionable feedback to users trying to use on_change outside of a Bokeh server app.

Unfortunately, it was not appreciated at the time that the mechanism for export_png is essentially to create a standalone document for a headless browser to render and take a screenshot of! This is always the way export works (even in a Bokeh server app), and so that process unintentionally triggers the new error message (even in a bokeh server app).

There are two options:

• remove the new error check entirely, and continue to deal with the tide of "why doesn't this work" questions with ongoing human labor, or

• find a way to special case things with export_png so that the error check is short-circuited only during exports

Since "human labor" is basically just me and my time, I'd obviously prefer to find a way to make the latter work.

[jnettels]

Dear @bryevdv, I am sorry if there is something I do not understand. But while I think that I get what you are trying to say, it does not match what I am currently experiencing. Consider this reduced piece of code from the example above:

from bokeh.models import ColumnDataSource
from bokeh.plotting import figure
from bokeh.layouts import layout, widgetbox
from bokeh.io import export_png
from bokeh.models.widgets import Button

def download():
    try:
        export_png(doc_layout, filename="export doc_layout.png")
        print('doc_layout works')
    except Exception as ex:
        print('doc_layout throws error:', ex)
    try:
        export_png(fig, filename="export fig.png")
        print('fig works')
    except Exception as ex:
        print('fig throws error:', ex)

source = ColumnDataSource({'x': [2, 3, 4], 'y': [0.5, 1.5, 2.5]})
fig = figure(plot_width=1000, plot_height=300)
fig.hbar(y='y', right='x', height=0.4, source=source)

button = Button(label="Download")
button.on_click(download)
doc_layout = layout([[widgetbox(button)], [fig]], sizing_mode='fixed')

download()  # Test 1

Then do bokeh serve export_test.py. Then connect to the server with your browser. It creates the output:

You are generating standalone HTML/JS output, but trying to use real Python
callbacks (i.e. with on_change or on_event). This combination cannot work.

Only JavaScript callbacks may be used with standalone output. For more
information on JavaScript callbacks with Bokeh, see:

    http://bokeh.pydata.org/en/latest/docs/user_guide/interaction/callbacks.html

Alternatively, to use real Python callbacks, a Bokeh server application may
be used. For more information on building and running Bokeh applications, see:

    http://bokeh.pydata.org/en/latest/docs/user_guide/server.html

doc_layout works
fig works

Despite your "standalone HTML" warning, exporting both the layout and the figure works. The expected files are generated. Since curdoc() is not even referred to, nothing is visible in the broswer, but that is expected. My point is: The "standalone HTML" warning does not prevent export_png from working correctly. It is irritating in this case, but nothing more.

While you said that it has nothing to do with the problem, the only line of code that causes export_png to fail is curdoc().add_root(doc_layout) - at least from my point of view.

Following that, from my point of view there are two seperate issues:

• A useful "standalone HTML" warning, that is irritating in this special case but does no harm
• export_png not working under some conditions

Since I trust your explanation, I expect to have a misunderstanding in the above train of thought. But reiterating my experience seems to be the only helpful thing I could do.

A related question that might help clear things up: Is the "standalone HTML" message intended as a) an error (preventing all following code from working) or as b) a warning (not interfering with the rest of the code)? To me it seems like b) is the case. But from your explanation I feel like a) was intended.

[bryevdv]

Partly I think you are using a definition I don't recognize. Adding things to the document (using curdoc().add_root) is not what I would think of when "adding to a layout", which I would only consider to be adding to something in bokeh.layouts e.g. a row or column. Also this is the relevant error message, that was not included:

fig throws error: Message: {"errorMessage":"undefined is not an object (evaluating 'document.getElementsByClassName('bk-root')[0].children[0].getBoundingClientRect')","request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"171","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:49953","User-Agent":"Python http auth"},"httpVersion":"1.1","method":"POST","post":"{\"script\": \"\nreturn document.getElementsByClassName('bk-root')[0].children[0].getBoundingClientRect()\n\", \"args\": [], \"sessionId\": \"060e11a0-788c-11e8-83b8-2779613502fa\"}","url":"/execute","urlParsed":{"anchor":"","query":"","file":"execute","directory":"/","path":"/execute","relative":"/execute","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/execute","queryKey":{},"chunks":["execute"]},"urlOriginal":"/session/060e11a0-788c-11e8-83b8-2779613502fa/execute"}} Screenshot: available via screen

@mattpap when you are back next week, my offhand guess is this has to do with the DOM structure changes for the custom HTML template work. Perhaps the export_png code for the headless browser needs some kind of update.

In any case this is not the same issue as before, even if it presents a similar outcome.

[mattpap]

Lets simplify the code a bit:

export_png(doc_layout, filename="export doc_layout.png") # works
export_png(fig, filename="export fig.png")               # works

curdoc().add_root(doc_layout)
# navigate

export_png(doc_layout, filename="export doc_layout.png") # works
export_png(fig, filename="export fig.png")               # fails

The difference between the former and the later set of export_png() calls, is that the former doesn't have a document assigned to models, whereas the later has. This results in the later export_png(fig) being understood as (~) export_png(fig.document.roots), which results in a mismatch between roots and their assigned element ids and fig. I think we should simply disallow usage of sub-models as roots after a document is assigned. One can always clone a model to start from scratch and avoid confusion. An alternative we could ignore the document in certain scenarios, but I think that would be too messy. Perhaps we may need to introduce a concept of (lets call it) bound and unbound models to further clarify what happens when a document is assigned and what when it's not.

[jnettels]

Dear @mattpap, you hinted at

One can always clone a model to start from scratch and avoid confusion.

as a workaround, if I understand correctly. I tried some (stupid, I guess) stuff like

fig_copy = figure()
fig_copy._property_values = fig._property_values

which always results in errors like: fig throws error: Sub-model (id='f8b61051-2e18-4a08-97a4-8267f28e4386', ...) of the root model Figure(id='af8f75c4-f9e2-49d3-9f07-8b9242bea45b', ...) is already owned by another document (Models must be owned by only a single document). This may indicate a usage error.

Or did you mean to actually recreate the whole figure from scratch?

---

Anyways, I found another workaround: Temporarily removing the figure from the document. When you click "Download", the figure will vanish until the export is completed. This is not a good solution, though. In my main project it fails, with the error message shown above. I guess there are more dependencies between documents and models that I don't understand. Also locating your figure in this way doc_layout.children[1].children seems cumbersome.

from bokeh.io import curdoc
from bokeh.models import ColumnDataSource
from bokeh.plotting import figure
from bokeh.layouts import layout, widgetbox
from bokeh.io import export_png
from bokeh.models.widgets import Button

def download():
    try:
        export_png(doc_layout, filename="export doc_layout.png")
        print('doc_layout works')
    except Exception as ex:
        print('doc_layout throws error:', ex)
    try:
        doc_layout.children[1].children.remove(fig)  # remove to allow export
        export_png(fig, filename="export fig.png")
        doc_layout.children[1].children.append(fig)  # add fig back in again
        print('fig works')
    except Exception as ex:
        print('fig throws error:', ex)

source = ColumnDataSource({'x': [2, 3, 4], 'y': [0.5, 1.5, 2.5]})
fig = figure(plot_width=1000, plot_height=300)
fig.hbar(y='y', right='x', height=0.4, source=source)

button = Button(label="Download")
button.on_click(download)
doc_layout = layout([[widgetbox(button)], [fig]], sizing_mode='fixed')

#download()  # Test 1

print('Layout is added to root')
curdoc().add_root(doc_layout)
curdoc().title = "Export PNG test"

#download()  # Test 2

[jnettels]

This took me quite a while to figure out, but here is another workaround with a more general approach. It works in my main use case, so hopefully it should always work. I hope this helps those with the same issue who come here via search.

If the model we are trying to export must not be assigned to a document, a general solution is to remove the complete root from curdoc() during the export. Instead we show a temporary root with a status message:

from bokeh.io import curdoc
from bokeh.models import ColumnDataSource
from bokeh.plotting import figure
from bokeh.layouts import layout, widgetbox
from bokeh.io import export_png
from bokeh.models.widgets import Button, Div

def download():
    div_temp = Div(text='''<div>Please wait for export</div>''', width=1000)
    root_temp = layout(div_temp)
    curdoc().add_root(root_temp)  # create temporary view
    root_main = curdoc().roots[0]  # store the main root
    curdoc().remove_root(root_main)  # required for export_png to work

    try:
        export_png(doc_layout, filename="export doc_layout.png")
        print('doc_layout works')
    except Exception as ex:
        print('doc_layout throws error:', ex)
    try:
        export_png(fig, filename="export fig.png")
        print('fig works')
    except Exception as ex:
        print('fig throws error:', ex)

    # Restore original view
    curdoc().add_root(root_main)
    curdoc().remove_root(root_temp)

source = ColumnDataSource({'x': [2, 3, 4], 'y': [0.5, 1.5, 2.5]})
fig = figure(plot_width=1000, plot_height=300)
fig.hbar(y='y', right='x', height=0.4, source=source)

button = Button(label="Download")
button.on_click(download)
doc_layout = layout([[widgetbox(button)], [fig]], sizing_mode='fixed')

curdoc().add_root(doc_layout)
curdoc().title = "Export PNG test"

The "standalone HTML" warning may still be shown, but can be ignored.

[bryevdv]

The new OutputDocumentFor context manager might handle this, I will try that out soon

[bryevdv]

@AzraelDD I believe the work to add OutputDocumentFor actually just solved the issue. Your original code is working for me. Can you verify this with a recent dev build, or local dev install?

If so, I will task this issue with suppressing the warning about callbacks when exporting.

[bryevdv]

Closing since, as I said the original code now works for me, and also the annoying warning has been silenced. Please reopen or comment if issues are persisting with latest dev versions.

[znstrider]

Thank you AzraelDD, your solution works for me.

[jnettels]

@bryevdv Now I had a chance to test with 1.0.0dev8+17.ged889df. Thanks very much for getting back to this issue! I can confirm that the original problem is solved now, which is great for most common use cases of export_png and export_svgs.

---

"However"... the workaround I proposed does no longer work. One might argue that such a usage is not supported and I could accept that, but I still wanted to bring the issue up:

In the workaround, I removed the root including the figure during the export and showed a status message instead. Adding the original root back again now throws an error: Models must be owned by only a single document, HBar(id='3f1714ac-f846-4799-b050-d70c157917e2', ...) is already in a doc When exporting a longer list of figures, this was actually quite useful.

Here is an example:

# -*- coding: utf-8 -*-
from bokeh.io import curdoc
from bokeh.models import ColumnDataSource
from bokeh.plotting import figure
from bokeh.layouts import layout, widgetbox
from bokeh.io import export_png
from bokeh.models.widgets import Button, Div

def download():
    div_temp = Div(text='''<div>Please wait for export</div>''', width=1000)
    root_temp = layout(div_temp)
    curdoc().add_root(root_temp)  # create temporary root
    root_main = curdoc().roots[0]  # store the main root
    curdoc().remove_root(root_main)  # required for export_png to work

    try:
        export_png(fig, filename="export fig.png")
        print('fig export works')
    except Exception as ex:
        print('fig throws error:', ex)

    # Restore original root
    curdoc().remove_root(root_temp)
    try:
        # Works in 0.13.0, does not work in 1.0.0dev8+17.ged889df
        curdoc().add_root(root_main)
    except Exception as ex:
        # Error message in 1.0.0dev8+17.ged889df:
        # Models must be owned by only a single document, HBar is already in a doc
        print(ex)

source = ColumnDataSource({'x': [2, 3, 4], 'y': [0.5, 1.5, 2.5]})
fig = figure(plot_width=1000, plot_height=300)
fig.hbar(y='y', right='x', height=0.4, source=source)

button = Button(label="Download")
button.on_click(download)
doc_layout = layout([[widgetbox(button)], [fig]], sizing_mode='fixed')

curdoc().add_root(doc_layout)
curdoc().title = "Export PNG test"

[bryevdv]

@AzraelDD Thanks for getting back to us, I do think that the workaround is really unsupported behavior, the "models belong to one doc" is a pretty foundational assumption.

It's possible using the OutputDocumentFor decorator, or even tweaking _document by hand might work as workarounds, but I'd need more context, specifically:

When exporting a longer list of figures, this was actually quite useful.

I don't really understand what situation this describes. It seems now with this change, you can just loop over the list of figures and export each, without removing/adding back anything. Can you explain a bit further?

[jnettels]

As I said, I can totally accept this being unsupported behavior.

But since you ask, this is my main use case: I let users create a grid of plots from their data, where (up to) all columns are plotted against each other (similar to a typical correlation coefficient matrix). Anyway, the only thing that matters is that the list of figures can become large quite fast. I include two regular Button widgets (export_png and export_svgs) for starting the export. Since iterating over the list of figures takes a while (during which all callback related stuff is unresponsive), I need to show a status message or something similar.

• With my workaround, I can hide the default UI. Instead I show nothing but a Div, where the text displays the path of the current export. Not the prettiest solution, but gives all needed information

• Now (it seems like) I cannot hide the UI anymore. If I just add the Div as another root, it is shown at the bottom of the page, and thus potentially out of view. But with enough time I think I will find a solution, like moving the Div to the top of the page. Maybe something similar to a js alert could be possible, too. Also I can use the spinning mouse wheel document.body.style.cursor = "wait" which I already use in other situations.

• Ultimately I would like to replace the default save tool in the gridplot toolbar. Only Chrome seems to be able to download all pngs simultaneously. In Firefox you have to accept the download for every individual file. And Egde cannot handle the downloads at all. Also all files have the same name - in my own export function I can give meaningful file names. But then there is a huge difference between exporting directly to the file system and sending a file via the browser. Furthermore, I didn't have the time yet to look into custom tools. All in all, this is stuff for another topic :-)