jupyter ipywidgets -- update cocalc to work with newest version of ipywidget

[williamstein]

Ipywidgets v8 breaks cocalc's Jupyter. Not everything is broken, but this is:

import ipywidgets as widgets
from IPython.display import display
build_label = widgets.Label(value="Build:")
json_button = widgets.Button(description="Bank from cache")
json_regen_button = widgets.Button(description="Regenerated bank")
buttons = widgets.HBox([build_label,json_button,json_regen_button])
display(buttons)

WORKAROUND: Restart your project. We already fixed this in the default image.

[williamstein]

This is turning out to be much, much harder than expected, because the ipywidgets devs (@jasongrout) fully switched to es2017 as their build target (good for them, of course), but cocalc is are on es5, so now we have to update to es6 at least, and that's nontrivial given how complicated cocalc is.

The non-es6 related changes that are required are also taking a while, though I might have figured them out mostly. For example, the migration guide https://github.com/jupyter-widgets/ipywidgets/blob/master/docs/source/migration_guides.md#migrating-from-7x-to-80 talks about a change with ManagerBase as if you just have to change where you import it from. I'm sure that's true going from 7x to 8... but that's NOT what we're doing. We're moving from 4.x to 8, and with 4.x ManagerBase was a parametrized type.

I've spent many hours on this, and still can't tell if it is a two day project or two month project. Part of the problem is that we use the same ipywidgets code on the server under nodejs and import/modules/etc. ended up pretty weird there. So just switching to es6 is fairly easy on the frontend, but we also have to switch part of our backend. Given there's still a mess of coffeescript in there, etc., this gets complicated.

Another option might be to build our own ipywidgets packages that target es5.

I'm not asking for help or comments really, but just recording some of my thoughts here so it's easier to pick up on this when I come back to it (hopefully soon).

[williamstein]

The guide is for "migrating from 7x to 8". However, the migration I'm doing feels much more like "migrating @jupyter-widgets/base from 4.x to 6.x? [1] E.g., 3 months ago the official version of @jupyter-widgets/base was 4.1.1, and now it is 6.0.1. For me, by far the hardest part of this, which I'm still not done with, is that 4.1.1 had a target of es5, but 6.0.1 has a target of es6. Since I define a custom widget manager that derives from the built in one, it seems like my only option (keeping that architecture) is to be forced to update my entire codebase to target es6. That should be mentioned in the migration guide.

It might also be nice to clarify how the overall version of ipywidgets relates to the version of the actual npm packages, e.g. "there is no relationship", since I got pretty confused by that.

Also, the migration guide doesn't address the fact that ManagerBase was a parameterized type before, and now it isn't, and that's another confusing thing to hit.

[1] See https://www.npmjs.com/package/@jupyter-widgets/base for the version history.

[williamstein]

I chatted with the ipywidgets devs just now about this issue. Jason said we could probably fork all the npm packages and rebuild everything to target es5, and publish that to npm (under @cocalc). On the other hand, I really wish we targeted es6 with cocalc, since it is more efficient at runtime, and easier to look at if you see a traceback without a source map. So this ticket is going to take a while.

[williamstein]

There's a website that analyzes code to say much faster something will be upon switching a "modern" target (with webpack).

https://estimator.dev/#cocalc.com

it doesn't finish, but it does give some estimates for many particular parts of cocalc and it looks pretty good:

[williamstein]

This looks very useful:

https://gist.github.com/silassare/cc3db860edeea07298cb03f64738c997

as is this discussion

https://stackoverflow.com/questions/66378682/nodejs-loading-es-modules-and-native-addons-in-the-same-project

and this esm module.

[williamstein]

Better version of that script mentioned above: https://gist.github.com/williamstein/5d1f5f77614c9a48d5adc9c2078647a8

[williamstein]

First, after a deep dive into ESM modules, etc., I realize that the core problem is that we just need to change how webpack is configured. I've done that, and now Jupyter widgets javascript works. So never mind about updating any our codebase to es modules for this reason.

[williamstein]

I'm pushing another change to https://github.com/sagemathinc/cocalc/pull/6142, but it seems like there are so many changes to ipywidgets, that it could be incredibly hard to ever get things to work with the new version of ipywidgets. Everything I relied on about how the base manager works seems to have just arbitrarily changed, and I don't even know where to start, except doing a ton of reading code again, almost like starting over. It's frustrating to have to do that again.

Maybe a better approach is to just write a complete widget manager from scratch, rather than trying to derive from the builtin one. I could possibly even take the code from the previous ipywidgets.

[williamstein]

A little update. I've now switched everything over officially to using swc-loader, and that solves any import problems. I also changed cocalc-docker to install Jupyter packages via:

RUN pip3 install pyyaml matplotlib jupyter jupyterlab "ipywidgets<8" "jupyter-client<7"

(see https://github.com/sagemathinc/cocalc-docker/blob/master/Dockerfile) and we'll just stick with that for a now.

[williamstein]

A little PSA: Here's an example of a core widget, which works fine with the older widgets, but breaks with ipywidgets 8 backend and our frontend.

import ipywidgets as widgets
tab_contents = ['P0', 'P1', 'P2', 'P3', 'P4']
children = [widgets.Text(description=name) for name in tab_contents]
tab = widgets.Tab()
tab.children = children
for i in range(len(children)):
    tab.set_title(i, str(i))
tab

The error is

TextStyleModel not found in module @jupyter-widgets/controls

This "TextStyleModel" isn't in @jupyter-widgets/controls from a few months ago (e.g., version 3.1.0 which is in cocalc, and is from about a year ago), but it is in the latest version.

Installing something with a dependency on newer widgets locally can easily break everything in this way, so this is further motivation to upgrade our frontend to use the latest widgets. Last time I tried, I failed, but it's got to be possible. I think the trick is to just start over and implement a widget manager from scratch, rather than trying to derive from the built-in one, which has become too inflexible for some reason (?). We'll see.

[williamstein]

NOTE: it seems we need to do this also in order to get ipyml to work... even in JupyterLab (?).

[williamstein]

I've spent many hours on this, and still can't tell if it is a two day project or two month project.

It turned out to be about 2-3 focused weeks -- see https://github.com/sagemathinc/cocalc/pull/7687/commits .

Also ipympl does work now... but mpld3 is probably better and it also works. It depends on what you need though.