Jupyter Community Workshop - The Future of Jupyter Widgets [London]

[ibdafna]

HackMD for the event: https://hackmd.io/ygqeDBoLRPC7piEcfYKk2g

[jasongrout-db]

For archiving, here is the current contents of the hackmd:

Jupyter Community Workshop - The Future of Jupyter Widgets

Tuesday 18th of October - Friday 21st of October, London

Week Agenda:

Google Docs Agenda

Friday 2022-10-21

Zoom link for remote attendees: https://bloomberg.zoom.us/j/95137727255?pwd=OGpmYmI0RkJZelk1V2ErUWoyc2w4QT09

• 8:30-9:30: Breakfast
• 9:30-12:00: Knowledge sharing / Hacking
• 12:00-13:00: Lunch
• 13:00-18:00: Socials / Hacking!

ipywidgets overview

• package overview:
  • js
    • base/widget.ts -> basic widget class
    • base-manager -> inherts from base
    • html manager -> based on base manager
    • schema -> useful documenation around
  • python

Thursday 2022-10-20

Zoom link for remote attendees: https://bloomberg.zoom.us/j/92527562243?pwd=ODFuR2VlZjVia0xLcCtPemRhUkFPUT09

• 8:30-9:30: Breakfast
• 9:30-12:00: Building larger applications using ipywidgets (with discussion of performance and developer experience)
• 12:00-13:00: Lunch
• 13:00-17:00: Hacking!
• 17:00-18:00: Demos
• 18:30 (optional): Impromptu drinks/dinner

Wednesday 2022-10-19

Zoom link for remote attendees: https://bloomberg.zoom.us/j/97006618705?pwd=SnM3OXdWYUtDbWUwbFZZenY5S2ZWZz09

• 8:30-9:30: Breakfast
• 9:30-12:00: Reviewing the entire ipywidgets stack
• 12:00-13:00: Lunch
• 13:00-17:00: Hacking!
• 17:00-18:00: Demos
• 18:30 (optional): Dinner @ Apulia (https://apuliarestaurant.co.uk/)

Tuesday 2022-10-18

Zoom link for remote attendees: https://bloomberg.zoom.us/j/93920480098?pwd=SnkvNGlCWTZiUGY4b3EzWlNMMHEydz09

15 Attendees representing academia and industry, with core developers, third-party widget developers, widget users, etc.

• 8:30-9:30: Breakfast
• 9:30-12:00: Introductions and demos
• 12:00-13:00: Lunch
• 13:00-17:00: Discussion about widgets in various frontends
• 17:00-18:00: Demos
• 18:30 (optional): Drinks at The Last Talisman

Demos

• Maarten Breddels on Reacton
  • Provides an answer for how to build large applications in ipywidgets, basically a port of React to Python, which uses ipywidgets, etc., as the equivalent of the DOM elements
  • Wrapped ipywidgets, and can autogenerate wrapper as long as constructor arguments match the traits. Currently ipywidgets, ipycanvas, bqplot, wrappers are shipped with reacton
  • Compare with idom: idom is driven by react. Reacton does everything in Python
  • How to create a new wrapped react component? Create a widget, then generate the wrapper in Reacton.
  • How do we embed an ipywidget into a React application?
  • Maarten does embed into Vue
  • We could have better examples for embedding an ipywidgets into existing web applications
  • You can embed reacton inside an ipywidgets app, and embed ipywidgets inside a reacton app
• Max Fordham: ipyautoui project
  • Generates a ui based on pydantic types
• Nate Rush on mito

• Jeremy Tuloup, Martin Renou:

  • JupyterLite
  • Jupyter Notebook 7 (based on JupyterLab components)
  • JupyterLab App Cookiecutter
  • https://xeus-python-kernel.readthedocs.io/en/latest

  Discussion: Widgets in Other Locations

  • Don Jayamanne on ipywidgets in VS Code
    • https://github.com/microsoft/vscode-jupyter/wiki/IPyWidget-Architecture-Challenges
    • Details of how VS code handles message passing from frontend to kernel
    • Widget echos: https://github.com/jupyter-widgets/ipywidgets/pull/3195
  • Pete Blois on colab:
    • How colab handles this: https://github.com/googlecolab/colabtools/blob/main/google/colab/output/_widgets.py
  • Finding all dependencies of a set of widgets including jslink
    • Issue in ipycanvas where there was a missing ref between widgets https://github.com/martinRenou/ipycanvas/pull/290 resulting in iframed canvases failing to render.

Issues for working groups

• iframe isolation for widgets
• lightweight kernel manager that can support multiple versions of the widget manager
• output widgets, and how outputs are updated (kernel vs frontend)
• ipywidgets 7+8
  • Want to support ipywidgets 7 core controls in ipywidgets 8 widget manager
• Layering the apis in the frontend to separate concerns:
  • synchronization
  • model instantiation
  • view rendering
• Clearly documenting the entry points for AMD modules (like in JLab has metadata in package.json)
• Provide a RESTful end point to expose the list of widgets and their corresponding entry points in JLab and Jupyter Notebook.
• Rich outputs without Widgets (HTML & Comms)
• Freezing part of the stack to give predictability and prevent breakageshttps://github.com/jpmorganchase/jupyterlab_templates
• Testing
  • currently have unit tests and some Galata ui tests
  • expand to more comprehensive protocol end-to-end protocol tests
• ipywidgets in other non-notebook contexts: streamlit, solara, shiny
• ipywidgets as a protocol?
  • Moving the documentation (which exists in an npm package somewhere) to the docs
• Should the kernel handle the Output widgets redirection?
• error handling for comm messages that don't get through
• Handling the "comm target does not exist". Should we improve the jupyter kernel protocol?
• Remove the ipykernel and IPython dependencies:
  • Rely on a Comm modules that's separated from ipykernel: https://github.com/martinRenou/comm / https://github.com/jupyter-widgets/ipywidgets/pull/3533 / https://github.com/ipython/ipykernel/pull/973 / https://github.com/jupyter-xeus/xeus-python/issues/342
• ipywidget author education:
  • Finding documentation in the first place
  • Custom widget dev tutorials
  • Checking for broken links / tutorials
  • Run codespell on the docs/code-base
• Linting: https://github.com/jupyter-widgets/ipywidgets/pull/3566
• Packaging: https://github.com/jupyter-widgets/ipywidgets/pull/3587
• async/await: https://github.com/jupyter-widgets/ipywidgets/pull/2779
• What problems are we solving with widgets vs. what we set out to solve:
  • devs who rely on the packaging aspect: widgets as a ways to distribute to multiple UIs
  • users who rely on widgets: quick and dirty prototypes vs. full applications (or, as a gateway from one to the other)

End of day Demos

• Matt Craig: Demoed
  • Templating extensions:
  • https://pypi.org/project/jupyterlab-multicontents-templates/
  • https://github.com/jpmorganchase/jupyterlab_templates (pypi)
  • https://github.com/trungleduc/jupyter_app_launcher
  • https://jupyterstarters.readthedocs.io/en/stable/

Wednesday, Oct 19

Full ipywidgets stack discussion

Traitlets discussion

• pydantic (typing) -> value: Syncable[Int]
  • Can we rebuild trailets on top of pydantic?
  • Outside our ecosystem, so is it worth the risk?
    • Perhaps the best bet is just improving traitlets
  • We don't have to abondon trailets for widget authors; we can let widget users use pydantic
• Refactor traitlets:
  • Performance:
    • Creating traitlets with static default values can be improved 20x (maarten has working PR)
    • Event processing: a lot of time appears to be spent in bunch
  • Exposing typing to static typers
  • Removing the metaclass that attaches the name to each traitlet
    • pep 487
  • Adding typing using dataclass transform

JS Stack

Backbone

• Uses:
  • Eventing
  • Framework for instantiating and syncing
  • Equality comparison (deciding if you need to sync something); _.isEqual
  • Lifycycle
• State of Backbone as of October 2022:
  • There is a pinned issue on the repo stating the project is still maintained: https://github.com/jashkenas/backbone/issues/4244#issuecomment-991381707
  • There was a release on February 26th 2022: https://backbonejs.org/#changelog
  • So there might not be a need to fork and maintain a custom Backbone for now
• Issues with backbone:
  • jquery: exposed as part of the API for backwards compatibilty
    • Would be nice to remove this as it's not used by jupyter
    • How to remove this without breaking custom widgets, unclear
    • $ as a property, first time you access it, it loads?
  • We don't fit the syncronization model, have had to strip a lot of this
  • Maintainablity: it's fossilized, hard to get changes merged.
    • Maarten had issue getting equality checking for typed arrays merged
• The goal perhaps is removing jquery
  • It's very hard to do a lazy-load of jquery, as these are syncronous APIs
  • Removing jquery is far less of a breaking change than ipywidgets 7 -> ipywidgets 8
  • It's like 50kb for 2022. The modern browser versions are super small, as they punt to the browser for everything
  • What's even the goal of getting rid of jquery?
    • The biggest thing you get is the a simpler interface - reimplementing widgets is easier
• Options:
  • Fork backbone, have more control, and we can get rid of jquery
    • Leave in all the syncing stuff, etc - in case people rely on this
    • Could remove the collections, etc
  • What would it take to drop backbone?
    • If we had lumino widget class from the start, would we have used backbone?
      • Backbone has a more nuianced event system: e.g. the value change events, there's more advanced logic here
        • Changing the event handling is high-risk
        • e.g. ipythreejs relies on specific event ordering
    • Forking the portion of backbone that does this event handling?
      • Issues with typing? Only for when widget authors upgrade, so not the worst sort of breaking change
  • Could we just have a strategy for when backbone fails us?
    • E.g. if you could put in a custom equals, this typed array equality comparison would be ok
  • Deprication messages?
    • When you use $ on the page
    • In webpack, we could log a warning
• On ipywidgets 7 -> 8:
  • This could be a test case for if widget authors are willing to upgrade
  • Potentially, the upgrade fatigue will just build up
  • Maybe we should insist on being future compatible and backward compatible

Lumino

• lumino widget

• processLuminoMessage

• Can we rely on the browser APIs?

  • before attach, after attach
  • onresize, presumably you could shim this to the the browsers resize event w/ ResizeObserver

• Do widgets need to use lumino?

  • Not necessarily

• Questions around keeping the API backward compatible + a smaller subset stable API?

  • A working group looks at the API, decides what a minimal API looks like for building widgets. Can we build a large portion of the widgets going forward just with this smaller API?
  • Two approaches:
    • Slimming down the widgets API
    • Starting from scratch

• Lumino is going to be around for as long as we use it to build the frontends ; there's less concern than JQuery

• Working group on widget API changes:

  • Option 1: removing things (e.g. jquery, Lumino) from the API
  • Option 2: reccomending a specific subset of the API that will be long-term maintained
  • Option 3: keeping the API the same, and then adding a new API that is long-term maintained
  • Etc.

• Dicsussion on the above working group:

  • removing things from existing API gets less pressing if we introduce a new minimal subset API that users can rely on
  • although one of the things we're loosing is composability of widgets if we just have this minimal API
    • How much composition of indivigual outputs really occurs?
    • The layout widgets are extremly common
  • if different people do it different ways, then there will be libraries that effectively do this widget management
  • The hard part of the minimal API is the composability of the widgets:
    • But if both widgets are syncing back to Python, then you can still do the side-by-side widgets
      • But this is where jslink becomes an issue, but maybe this can be exposed as an API
  • How do widgets communicate on the frontend:
    • Pete argues that this different frameworks do this differently, and trying to be opinioned on this in a long term contract is a loosing game.
    • Don points out that this means that widgets interoperating requires widget authors to opt-in in ways that they don't
  • Plan: punt the discussion to this afternoon, and come up with a concrete proposal
    • A group of people have a more concrete proposal:
      • Here's the API
      • Here's how we handle embeding a widget inside another widget
      • Here's how they talk to eacother on the front-end / backend
      • Here's how a widget would work with the current stack
      • Here's how a widget that doesn't work with the current stack would work (and here's how it communicates with the current stack)

Webpack

• This is an internal detail of widgets:
• We could benefit from using another technology
• esbuild:
  • Does es6 module output
• We put a checkmark on this, and move on. We're not gonna consider this one now

AMD modules

• Current cookiecutter, there are currently a federated module for jupyterlab, and then two amd modules (for notebooks and for cdn) that look pretty much the same.

• Options and benefits:

  • es6 modules are the modern one, most things are moving to this. The primary benefit is that it's a bit more standarized.
    • When loading as an es6 module, async dependencies, this is standardized.
    • This is the fully standardized API.
    • Potentially prefering this becuase this is the browser standard

• What would a managed transition to es6 look like?

  • What steps are the breaking changes, how to make them less painful?

• If we're going to go to the minimal API, then we can leave AMD modules in the old API

  • Aka, one option is just adding the es6 as another module to the webpack
  • What is the carrot for widget authors? This allows them to load these modules from a CDN (jlab would need to support this?)

• This question is about: how do widget managers pull in the widgets?

  • Don, pete agree that the important this is that we all agree on how to load the widgets

• What non-jupyter notebooks do:

  • Collab has their own impl of require js that avoids polluting the global namespace
  • Microsoft used to have their own require js for .ipynb, but moved to just having standard require js in the webview last week

• Working group for looking into ES6

  • Answering two questions:
    • What are the benefits, if any?
    • Is it feasible to do it?
      • Major question around external modules - need some external configuration in the bundler or something
  • If there are benefits, and it is feasible, then could consider adding es6 modules as another bundle?
  • Grant Nestor had a demo of widget modules as es6 modules

• What would you love to see in widgets?

Minimal Interactive Outputs API

• Previous brainstorming: https://github.com/Quansight-Labs/jupyter-output-spec
• Colab's stable API: https://github.com/googlecolab/colabtools/blob/main/packages/outputframe/lib/index.d.ts
• How can this be integrated into an improved %%javascript magic? https://github.com/ipython/ipython/issues/13376#issuecomment-989040562
• Examples: https://github.com/blois/js-module-renderer/blob/master/tests.ipynb

Working Groups

Minimal API

Attendee: Pete, Vidar, Paul, Jeremy, Jason

An es6 module that exports a render method taking a context argument.

A context has a discovery function and functions for:

• comms
• update output state

Colab has a comms api:

• open("target"): Promise<Comm>

An example mimebundle:

{
  'application/vnd.jupyter.es6-rich-output': 'a string that will be the src of a script tag, i.e., a url, data url, etc.',
  'application/vnd.jupyter.datagrid+json': {'data': ['some', 'data', 'in', 'a', 'structure', 'which', 'the', 'js', 'can', 'access']},
  'text/html': 'fallback rendering of your data',
  'text/plain': 'fallback rendering of your data'
}

The mimebundle renderer for application/vnd.jupyter.es6-rich-output does a System.import of the url as an object, then calls the render method if it exists with a context object, awaiting the returned Promise<void> before moving to the next output.

The context object has mandatory fields:

• output: {'data': {...}, 'metadata': {...}} (which is the display_data or execute_result message data/metadata fields). Changes are not respected - treat it as immutable.
  • Should the transient data also be passed in???
• element: HTMLDivElement

Optional fields:

• render(output, element): Promise<void>:
  • output: a kernel iopub stream, display_data, execute_result, error message (i.e., inputs into the rendering system)
  • Should these be formatted for nbformat (e.g., transient field is not included)??? Should they include the trusted field that is passed in from JLab???
  • element: an HTMLDivElement
  • Promise resolves when the rendering callback returns
• comms: See Colab interface...
  • Comm interface
  • open("target"):
• sharedState: Map([@@Symbol]: Any) - shared state that might span render calls or contexts. This is considered mutable, in that a renderer can share things with other render calls, replace implementations, etc.
  • Let's explore exactly the scope of shared state (e.g., per kernel, per document, etc.). Does there need to be different scopes, or a way to introspect the scopes, etc.???

JupyterLab branch for prototype implementation: https://github.com/jupyterlab/jupyterlab/tree/es6-rich-output

Traitlet improvements

• Integration with more natural typing
  • ?
• Removing the metaclass
  • ?
• Performance improvements

https://github.com/ipython/traitlets/pull/788

Thursday

Building larger applications with ipywidgets

• Initial load performance?
  • There are hard limits on the
  • Server-side rendering
  • Doing some things client-side with a wasm runtime, doing what you need on the backend
    • Could we statically figure out which is which?
• Performance:
  • Users sometimes end up with hundreds to thousands of widgets, pushing memory usage (mostly on Python?)
  • Where are the performance bottlenecks with hundreds of widgets?
    • Where is the time spent with creating composition lots of widgets?
    • How does creating 100 button widgets compare to creating 100 normal html buttons -- what is the overhead?
• Solara:
  • Viola slower than and solara on initial load, due to starting a different Python process per
• Potential benchmarks:
  • 100 widget buttons vs. 100 HTML buttons
  • Where is memory used for hudreds/thousands of widgets that are all visible?
• On scaling application complexity:
  • Why do applications get complex?
    • No seperation of concerns
    • n^2 communication problem
    • Callback hell
  • There are no state management libraries for Python
    • It's also not clear that these Python users are looking for state management solutions
  • What are the user profiles who are using ipywidgets?
    • Recent college grad: goes to a bank, starts writing Pythonm uses ipywidgets to create an application. Not a software engineer, etc - it grows organically.
      • This is an iterative proces. Takes week if not months to work through issues.
      • E.g. memory that hangs around, this is really slow. There is no notion of lifecycle, etc.
    • More typical quants: they are using Jupyter for more prototyping stuff
  • This conversation is very similar to notebook vs. raw Pyhton files. It's very easy to get to a place in your code development where it's extremly hard to figure out what state you have vs. not, and then u.r.done....
• Potential documentation changes:
  • Should we have a section on the application page like: you built your first app.
  • Perhaps we should think about it as "how can we allow users to just copy and paste and it just works"
• Potentially a clever opinionated way of a way of building applications:
  • Sometimes this gets built internally within companies -
  • This could be something that we distribute, and allow users to use
  • Ideas:
    • Class with inheritance?
    • Reacton
    • Providing a visual programming interface? We could give a GUI builder
      • Concerns about degree of customizability
• Doc changes:
  • More complex examples of building complex widget applications
• Testing and typing:
  • Good practices for testing (although people might not use it)
  • Static typing wherever possible
• Working groups:
  • Proper state management for
    • Perhaps form input, simple output, that you regenerate
    • Show an example of building small components that then transitions to composing them together
• Create a contrib organization on GitHub, similar to jupyterlab-contrib:
  • There could be a list of custom widgets like on the jupyterlab-contrib website: https://jupyterlab-contrib.github.io/extensions.html
  • The Jovyan github org is also a place for Jupyter community projects. See https://github.com/jupyter/docker-stacks/issues/358 for more info about the name "Jovyan".
  • The name jupyter-widgets-contrib could make it more explicit about what it is about? And easier to find via search engines. It would also be similar to the existing jupyterlab-contrib.

Friday

Contributing

• Walk-through of fixing a bug in ipywidgets.
  • Installation along the way: wrong (old) version of jupyterlab installed when creating conda environment. Known issues, see: https://github.com/mamba-org/mamba/issues/1581

Yjs investigations

• https://github.com/jupyter-widgets/ipywidgets/pull/3195 already helped quite significantly to keep the state synced between the kernel and multiple frontends
  • Enabled by default in 8.x: https://github.com/jupyter-widgets/ipywidgets/blob/e3d8fe576334dc84d24ba8b20b68423e9e72ffe7/python/ipywidgets/ipywidgets/widgets/widget.py#L42
  • Disabled by default in 7.x: https://github.com/jupyter-widgets/ipywidgets/blob/892166dd0524e93b912d8826a6da26c09afd4366/ipywidgets/widgets/widget.py#L49
• Looking into using yjs / ypy for widgets:
  • probably the simplest for now would be to start a completely separate project rather than trying to replace the internals of ipywidgets directly
  • There could be an ephemeral YDoc created in the kernel
  • A first step could be to create a new Websocket connection between the web clients (JS) and the Python kernel. This would bypass the current contract of message ordering but could be fine for now for testing. There could be a simple JupyterLab extensions exposing a simple widget like the IntSlider via a mimerender extension
  • Widgets state are managed in the kernel via the YDoc
  • Ideally use a custom websocket provider over the comms?

[jgonggrijp]

Sorry for the bump, just noticed this reference in jashkenas/backbone#4244.

• So there might not be a need to fork and maintain a custom Backbone for now

Indeed, please submit PRs to the main repo instead! I realize I have not been very proactive in the past months, but I'm still keeping an eye on my notifications and will handle PRs swiftly.

[martinRenou]

Concerning the point:

Create a contrib organization on GitHub, similar to jupyterlab-contrib

I created the https://github.com/jupyter-widgets-contrib organization, moving ipycanvas there already as a test.

The website is here: https://jupyter-widgets-contrib.github.io/. It is simply a fork of the JupyterLab contrib website.

Feel free to open transfer requests so we can start feeding this organization! Happy to give admin rights to people as well in the process.