Lets-Plot library plots are not rendering

[OSuwaidi]

I'm quite new to Panel and I would really like to start using it since it supports a wide range of Python plotting libraries and is excellent for dashboarding even for professional settings.

However, I happen to be predominantly using the Lets-Plot Python visualization library, which unfortunately, doesn't seem to be supported by Panel!

When I bind a panel widget to a plotting function that returns a lets-plot plot object, the plot is not rendered (laid out) upon using a Panel layout such as Column.

Here's a MRE:

from lets_plot import *
import panel as pn
import pandas as pd

LetsPlot.setup_html()
pn.extension()
df = pd.DataFrame({
    'x': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    'y': [2, 4, 1, 8, 5, 7, 6, 3, 9, 10],
})

def create_lets_plot(rng):
    p = (ggplot(df[:rng]) + geom_line(aes(x='x', y='y')))
    return p

slider = pn.widgets.IntSlider(name="range", value=5, start=0, end=len(df))
bound_plot = pn.bind(create_lets_plot, slider)
pn.Column(slider, bound_plot)

The only way to actually visualize anything is by forcing a plot via p.show(), but that will stack the plots multiple times which is obviously undesirable.

[MarcSkovMadsen]

Hi @OSuwaidi

The plot probably need an export to SVG or HTML to be displayed in Panel.

I'll take a look when I get back to my computer. My starting point would be see how it exports it self in https://github.com/JetBrains/lets-plot/blob/master/python-package/lets_plot/export/ggsave_.py

Maybe we should add a lets-plot pane to automatically do this for you and other users?

[OSuwaidi]

Hey @MarcSkovMadsen,

Thanks for the hint, after messing around with the exports a bit, a temporary solution that seems to work is by passing the lets-plot plot object into the export_svg() function and then returning it from create_lets_plot():

def export_svg(plot: Union[PlotSpec, SupPlotsSpec, GGBunch]):
    """
    Export plot or `bunch` to a file in SVG format.

    Parameters
    ----------
    plot: `PlotSpec`, `SupPlotsSpec` or `GGBunch` object
            Plot specification to export.

    Returns
    -------
        An SVG file.

    """
    if not (isinstance(plot, PlotSpec) or isinstance(plot, SupPlotsSpec) or isinstance(plot, GGBunch)):
        raise ValueError("PlotSpec, SupPlotsSpec or GGBunch expected but was: {}".format(type(plot)))

    from lets_plot import _kbridge as kbr

    return kbr._generate_svg(plot.as_dict())

However, the generated plot(s) are static (no tooltips), and I was unsuccessful in returning the plots in an HTML format (just displays blank upon serving the app).

Appreciate if we can support the plots in an interactive/dynamic way!

[MarcSkovMadsen]

Hi @OSuwaidi

You can try using the below. If you experience issues please report back with minimum, reproducible examples

https://github.com/holoviz/panel/assets/42288570/3e7243cb-e618-46de-825f-af2916a04818

import param
from panel.reactive import ReactiveHTML

from lets_plot import __version__ as _lets_plot_version
from lets_plot.plot.core import PlotSpec

class LetsPlotPane(ReactiveHTML):
    object: PlotSpec = param.ClassSelector(class_=PlotSpec, precedence=-1)

    _plot_spec_as_dict = param.Dict()

    _template = '<div id="pn-container" style="height:100%;width:100%"></div>'

    __javascript__ = [
        f"https://cdn.jsdelivr.net/gh/JetBrains/lets-plot@v{_lets_plot_version}/js-package/distr/lets-plot.min.js"
    ]

    @param.depends("object", watch=True, on_init=True)
    def _update_config(self):
        if not self.object:
            self._plot_spec_as_dict = {}
        else:
            spec = self.object.as_dict()
            if "data" in spec and isinstance(spec["data"], pd.DataFrame):
                spec["data"] = spec["data"].to_dict(orient="list")

            self._plot_spec_as_dict = spec

    _scripts = {
        "render": "state.height=-10",
        "after_layout": "self._plot_spec_as_dict()",
        "_plot_spec_as_dict": """
var height=pn_container.clientHeight
if (state.height-5<=height & height<=state.height+5){height=state.height}
state.height=height
pn_container.innerHTML=""
LetsPlot.buildPlotFromProcessedSpecs(data._plot_spec_as_dict, pn_container.clientWidth-5, height, pn_container);
""",
    }

from lets_plot import ggplot, geom_line, aes
import pandas as pd
import panel as pn

pn.extension(design="bootstrap")

df = pd.DataFrame(
    {
        "x": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
        "y": [2, 4, 1, 8, 5, 7, 6, 3, 9, 10],
    }
)

def create_lets_plot(rng):
    p = ggplot(df[:rng]) + geom_line(aes(x="x", y="y"))
    return p

def create_lets_plot_pane(rng):
    p = create_lets_plot(rng)
    return LetsPlotPane(object=p, sizing_mode="stretch_width")

slider = pn.widgets.IntSlider(name="Range", value=5, start=0, end=len(df))
bound_plot = pn.bind(create_lets_plot_pane, slider)
component = pn.Column(
    pn.Row(
        "# Lets-Plot",
        pn.layout.HSpacer(),
        pn.pane.Image(
            "https://panel.holoviz.org/_static/logo_horizontal_light_theme.png",
            height=50,
        ),
    ),
    pn.layout.Divider(),
    slider,
    bound_plot,
    sizing_mode="stretch_width",
).servable()

panel serve app.py

Notes before taking this to next level

• Lets-plot does not support responsive mode. You have to specify the exact width and height in the LetsPlot.buildPlotFromProcessedSpecs.
  • view.height, view.width are null and cannot be used to set the height and width.
  • after_layout is rerun when ever the window size changes
  • after_layout is rerun in infinity if I don't use the state.height hack.
  • Lets-plot does not provide any method to reset, clear or update a plot. Thus I use .innerHTML="".
• I don't know how the LetsPlotPane could take a bound function returning a PlotSpec instead of a PlotSpec. See https://github.com/holoviz/panel/issues/5476
• If I don't set precedence=-1 on the object parameter I will get a serialization error. We should document in the ReactiveHTML docs how to best solve this issue.

[MarcSkovMadsen]

Looking at https://github.com/randyzwitch/streamlit-letsplot/blob/master/streamlit_letsplot/__init__.py i can see that both PlotSpec and GGBunch needs to be supported.

[MarcSkovMadsen]

Would Panel be interested in add a LetsPlot Pane natively @philippjfr ? Or should it be a separate extension?

[OSuwaidi]

Any updates on this @philippjfr ?

[OSuwaidi]

Hey @MarcSkovMadsen 👋🏼, upon experimenting a bit with the work around you provided, it seems to break down when using plots having their stat parameter not defaulted to 'identity'. If you try to plot geom_density() or geom_smooth() for example, it spits out an error because lets-plot starts looking for the dataframe's meta data such as ..quantile.. and ..ymin/max.. for geom_density() and geom_smooth(), respectively.

But even upon manually providing those variables by augmenting the dataframe before feeding it into ggplot(), the rendered plots just reduce to something that you would get from geom_line().

Also, for any of the ordinary plots with their stat parameter defaulted to 'identity', if you change their stat to 'smooth' or 'density', nothing happens, the transformation doesn't take place and you get back the original plot!

[MarcSkovMadsen]

Hey @MarcSkovMadsen 👋🏼, upon experimenting a bit with the work around you provided, it seems to break down when using plots having their stat parameter not defaulted to 'identity'. If you try to plot geom_density() or geom_smooth() for example, it spits out an error because lets-plot starts looking for the dataframe's meta data such as ..quantile.. and ..ymin/max.. for geom_density() and geom_smooth(), respectively.

But even upon manually providing those variables by augmenting the dataframe before feeding it into ggplot(), the rendered plots just reduce to something that you would get from geom_line().

Also, for any of the ordinary plots with their stat parameter defaulted to 'identity', if you change their stat to 'smooth' or 'density', nothing happens, the transformation doesn't take place and you get back the original plot!

Hi @OSuwaidi . Could you provide a minimum, reproducible example? That would make it very easy for me or someone else to start root cause analysis. Thanks.

[OSuwaidi]

Sure! We first create the custom lets-plot pane (I used polars over pandas because plot wouldn't even render with pandas):

import param
from panel.reactive import ReactiveHTML

from lets_plot import __version__ as _lets_plot_version
from lets_plot.plot.core import PlotSpec

import polars as pl

class LetsPlotPane(ReactiveHTML):
    object: PlotSpec = param.ClassSelector(class_=PlotSpec, precedence=-1)

    _plot_spec_as_dict = param.Dict()

    _template = '<div id="pn-container" style="height:100%;width:100%"></div>'

    __javascript__ = [
        f"https://cdn.jsdelivr.net/gh/JetBrains/lets-plot@v{_lets_plot_version}/js-package/distr/lets-plot.min.js"
    ]

    @param.depends("object", watch=True, on_init=True)
    def _update_config(self):
        if not self.object:
            self._plot_spec_as_dict = {}
        else:
            spec = self.object.as_dict()
            if "data" in spec and isinstance(spec["data"], pl.DataFrame):
                spec["data"] = spec["data"].to_dict(as_series=False)

            self._plot_spec_as_dict = spec

    _scripts = {
        "render": "state.height=-10",
        "after_layout": "self._plot_spec_as_dict()",
        "_plot_spec_as_dict": """
var height=pn_container.clientHeight
if (state.height-5<=height & height<=state.height+5){height=state.height}
state.height=height
pn_container.innerHTML=""
LetsPlot.buildPlotFromProcessedSpecs(data._plot_spec_as_dict, pn_container.clientWidth-5, height, pn_container);
""",
    }

Below is sample data:

np.random.seed(0)
sample_revenue = np.random.randint(1, 10000, size=100)
sample_time = range(100)

df = pl.DataFrame({
    'revenue': sample_revenue,
    'time': sample_time
})
p = (ggplot(df.with_columns(pl.min('revenue').alias('..ymin..'), pl.max('revenue').alias('..ymax..')))
 + geom_smooth(aes(x='time', y='revenue'), method='loess', span=0.43, size=1.5)
  )
LetsPlotPane(object=p).servable()

Result:

Expected:

[alshan]

Hi @OSuwaidi . buildPlotFromProcessedSpecs() doesn't apply statistical transformation you needed. Try using buildPlotFromRawSpecs() instead.

[OSuwaidi]

@alshan Ah, you're right! Works perfectly now, thanks!

[OSuwaidi]

Hey guys,

Sorry to revive this randomly but, isn't the lets-plot pane class that @MarcSkovMadsen has written (with a minor tweak) is almost complete:

import param
from panel.reactive import ReactiveHTML

from lets_plot import __version__ as _lets_plot_version
from lets_plot.plot.core import PlotSpec

import polars as pl

class LetsPlotPane(ReactiveHTML):
    object: PlotSpec = param.ClassSelector(class_=PlotSpec, precedence=-1)

    _plot_spec_as_dict = param.Dict()

    _template = '<div id="pn-container" style="height:100%;width:100%"></div>'

    __javascript__ = [
            f"https://cdn.jsdelivr.net/gh/JetBrains/lets-plot@v{_lets_plot_version}/js-package/distr/lets-plot.min.js"
            ]

    @param.depends("object", watch=True, on_init=True)
    def _update_config(self):
        if not self.object:
            self._plot_spec_as_dict = {}
        else:
            spec = self.object.as_dict()
            if "data" in spec and isinstance(spec["data"], pl.DataFrame):
                spec["data"] = spec["data"].to_dict(as_series=False)

            self._plot_spec_as_dict = spec

    _scripts = {
            "render": "state.height=-10",
            "after_layout": "self._plot_spec_as_dict()",
            "_plot_spec_as_dict": """
var height=pn_container.clientHeight
if (state.height-5<=height & height<=state.height+5){height=state.height}
state.height=height
pn_container.innerHTML=""
LetsPlot.buildPlotFromRawSpecs(data._plot_spec_as_dict, pn_container.clientWidth-5, height, pn_container);
""",
            }

I believe it just requires supporting SupPlotsSpec class as well (gggrid layer)

[MarcSkovMadsen]

Its a good starting point. But Maybe we need to consider other things. Like performance.

But the main blocker is for @philippjfr to provide feedback on whether a PR would be appreciated or whether this should live in a seperate package.

[OSuwaidi]

Hey @MarcSkovMadsen and @alshan 👋🏼!

Guys really sorry to bring this up again (and thanks a ton for your continuous support) but I discovered two subtle, yet super annoying bugs in the lets-plot pane class (LetsPlotPane) @MarcSkovMadsen has implemented:

class LetsPlotPane(ReactiveHTML): object: PlotSpec = param.ClassSelector(class_=PlotSpec, precedence=-1) ...

1. If the data source is of type numpy array (numpy.ndarray), it spits out an error:

   The value of data variable [x] must be a list but was LinkedHashMap

   and here's an MRE (can be run in a notebook):

   from lets_plot import *
   import numpy as np
   import panel as pn
   
   LetsPlot.setup_html()
   pn.extension() 
   
   data = dict(x=np.random.rand(100), y=np.random.randn(100))
   plot = ggplot(data) + geom_line(aes(x='x', y='y'))
   LetsPlotPane(object=plot)

   Note that variable plot above itself is viewable and works fine. Fix: You would have to manually cast the numpy arrays in both x and y into either a list or a tuple for LetsPlotPane(object=plot) to work normally.

2. If the data source was of type DataFrame (either pandas or polars), and it was supplied as an argument in the LayerSpec (geom_x()) object rather than the PlotSpec (ggplot()) object, it spits out the following error:

   The value of data variable [0] must be a list but was Double

   and here's an MRE (can be run in a notebook):

   import polars as pl
   from lets_plot import *
   import numpy as np
   import panel as pn
   
   LetsPlot.setup_html()
   pn.extension() 
   df = pl.DataFrame(dict(x=list(np.random.rand(100)), y=list(np.random.randn(100))))
   plot = ggplot() + geom_line(aes(x='x', y='y'), data=df)
   LetsPlotPane(object=plot)

   Again, variable plot works fine, but the issue arises when wrapping it within LetsPlotPane(object=). Fix: You have to move the data source into ggplot() as in:

   ggplot(df) + geom_line(aes(x='x', y='y'))

   then it works fine. But this is very annoying because in my actual code I have a single ggplot() with two geom layers reading from different dataframes, hence I have to use the data=df in the geom_x() layers, but it gives me the above error.

Appreciate your time and assistance; the second bug specifically took me a lot of time to understand because the error: The value of data variable [0] must be a list but was Double makes no sense to me and I had to debug my codebase for a long time!

[alshan]

Hi guys, try to "standardize" plot specs before passing it to buildPlotFromRawSpecs():

from lets_plot._type_utils import standardize_dict

plot_spec_std = standardize_dict(plot.as_dict())

[OSuwaidi]

Hi guys, try to "standardize" plot specs before passing it to buildPlotFromRawSpecs():

from lets_plot._type_utils import standardize_dict

plot_spec_std = standardize_dict(plot.as_dict())

@alshan Massive! 🚀

Works like a charm, just add: self._plot_spec_as_dict = standardize_dict(spec) in our previous LetsPlotPane() class

[Ayfri]

Hey, I'm trying let's plot library in 2024.1 but I'm not getting any output anywhere ? What is the current state of this issue ?

[alshan]

The last I heard is that @OSuwaidi got a near-production-ready LetsPlotPanel().