Consider PygWalker extension alternative

[MarcSkovMadsen]

This is a great enabler 👍, but:

Please consider whether it will be a better long term solution to extend PygWalker instead like its done for Jupyter, Gradio and Streamlit https://github.com/Kanaries/pygwalker/tree/main/pygwalker/communications.

PygWalker is much more than just a thin wrapper around Graphic Walker. My understanding is that it

• provides a client-server architecture which scales to larger datasets than what this solution will do, as it will be limited by Tornado, Jupyter or Kubernetes message size settings and by the Browser max memory limit. That is what I'm dreaming of and increasingly need.
• support many different databackends (See)
• Provides a python api that is nice and quick to use in a data exploration setting

Strategically PygWalker will support and upgrade the solutions in the PygWalker repository. They will also promote them. Panel will probably not be a part of that journey when building its own solution. Except if it provides unique and more powerful features.

PygWalker also supports bidirectional communication. Being able to get hold of the configuration is a key enabler for persisting the user selections.

Where Panel can really make a difference is if it scale interactively to very large datasets. Like Datashader for HoloViews/ hvPlot. Or like what we support for Tabulator. Or like I've requested for Perspective in https://github.com/holoviz/panel/issues/7359. Or like modern tools developed with a DuckDB backend - for example Marimo and Quack. Or like Vega-Fusion.

[MarcSkovMadsen]

There is now a PygWalker AnyWidget implementation with https://github.com/Kanaries/pygwalker/pull/651. So maybe Panel should change to this implementation @philippjfr ?

[ObservedObserver]

Thanks again for the amazing extensions. Here is what I think:

We do have plans to extend pygwalker to more Python web frameworks. Our long-term goal is simple: make pygwalker work directly with most web frameworks.

These frameworks fall into two main categories:

1. Frontend-focused runtimes (like Streamlit, Gradio, and the new Shiny-py, which we prefer to maintain at pygwalker):

   • These need pygwalker to bind its frontend state with the Python frontend runtime
   • This is more about real-time interaction

2. Backend-focused servers (like Flask or Django):

   • These mainly need pygwalker to export HTML for hosting
   • This is more about serving static content

For the future, I plan to introduce a new API in pygwalker that can bind frontend state with Python runtime state. It would work like this:

spec = pygwalker.walk(df)

The spec will automatically update when users interact with the pygwalker UI.

Regarding Panel specifically: if it can work like Streamlit/Gradio, we should merge it into pygwalker and provide a standard API once it finished early stage development. If not, keeping two separate repos would be better.

---

We should also consider the maintenance aspect differently based on who maintains the repo:

1. If maintained by individual developers:

   • We recommend merging it into pygwalker
   • This ensures long-term maintenance and support

2. If maintained by a company-backed team:

   • The decision should align with the company's preferences
   • While separate maintenance is feasible given the company's resources
   • Separate repos would be more suitable if the company:
     • Plans to add commercial features
     • Needs to integrate with their platform-specific APIs
     • Wants to maintain independent release cycles
   • We'd appreciate hearing the company's thoughts on whether to merge or keep separate

[philippjfr]

Thanks @ObservedObserver! Panel is definitely closest to the Streamlit/Gradio model. In terms of maintainers I think it's primarily going to be @MarcSkovMadsen and I and both of us would be doing so in our capacity as individual developers, not employees of any company. I'd be happy to see it folded into pygwalker directly, but am also happy to incubate it here a little more and then integrating it when we consider it stable.

[MarcSkovMadsen]

@philippjfr . Most of the time here would be waisted as I see it if we migrate.

• We would throw away the JS. Pygwalker is something very different than what we have. It does not bind GraphicWalker it binds to a much more complicated app built on top of GraphicWalker. See pygwalker index.tsx.
• We would also throw away most of the Python code because its all there already in Pygwalker. See pygwalker anywidget.

The things we would have to do is

• Build a Parameterized model similar to the Pygwalker AnyWidget model. Probably fix a lot of "bugs" in the AnyWidget js to accomodate working with two slight variations of the same AnyWidget api.
  • Make sure our GraphicWalker widget has all the parameters and not just one props parameter as the AnyWidget implementation to align with how our widgets normally work.
• Build the comms.

Its also a matter of "philosophy". Pygwalker can be extended into a commercial product with chat AI etc. While this is a simpler implementation closer to "open source and free" concept. And closer to all the other components we have.

If we have "ideas" like enabling the .hvplot api on it, then it would be hard to do in PygWalker as there is already a plotting api. From what I've seen I'm also pretty sure our comms are more efficient than PygWalker. As far as I remember dataframes are transferred as json in Pygwalker.

Its not to say we should not do it as there are many benefits of working closely together. We should just know the consequences.

[MarcSkovMadsen]

One key architectural difference is that each instance of our GraphicWalker python class would be independent of the others. I.e. we could end up sending the same data to the frontend X times if we have X GraphicWalker instances the shows the same data in different ways. Pygwalker shares the data among all GraphicWalker js instances.

[ObservedObserver]

For the convenient of future development, we have just opened more graphic walker docs to the public. Access it here https://platform.kanaries.net/graphic-walker/overview

Some APIs might be not develper-friendly, like storeRef, we will keep working on it. Any feedbacks or suggestions are welcome!