This PR fixes 2 bugs uncovered in #261. Both are fairly major fixes to interactive/animated plotting functionality that we should try to get into a release version ASAP:
As of IPython v8.17, callback functions registered through the EventManager interface must now accept all arguments expected by the prototype function for the relevant event. This affects the _deferred_reset_cb() callback that Hypertools registers when temporarily changing the Matplotlib plotting backend to display an interactive/animated plot, in order to reset it to its previous state during the pre_run_cell phase of the next cell executed. Callbacks triggered by the pre_run_cell event expect to be passed an object containing various info about the to-be-executed cell, but since the next cell's contents aren't relevant to our use case, _deferred_reset_cb() had simply omitted this argument. However, the changes in ipython/ipython/pull/14216 now cause this function to throw an error when the user goes to execute the next cell, and continue to do so for every subsequent cell executed until the kernel is restarted (or the callback is removed manually from get_ipython().events.callbacks['pre_run_cell']).
To fix this, I've updated the _deferred_reset_cb() function to just accept and ignore any arbitrary *args passed to it.
Version 7.0 of the base Jupyter Notebook app switched over to using the same JavaScript codebase as JupyterLab, which doesn't expose the global IPython and Jupyter JS objects that were available in previous versions of Jupyter Notebooks. Matplotlib's default backend for displaying inline interactive/animated plots in notebooks (nbAgg) relied on these objects for communicating between the frontend and kernel, and therefore no longer works in Notebook v7.0+ (nor in JupyterLab). Since Hypertools uses nbAgg for displaying interactive plots by default (i.e., unless changed via hypertools.set_interactive_backend() or hypertools.plot()'s mpl_backend argument, both of which are unfortunately undocumented), interactive and/or animated Hypertools plots now throw an error with no clear fix when displayed in Notebook v7.0+ or JupyterLab.
To fix this, I've added some extra logic to the function in hypertools.plot.backend that runs when the package is imported and initializes the Matplotlib backend used for interactive/animated plots based on the importing environment. When imported into a Jupyter notebook, Hypertools will now determine whether that notebook is being run through the "classic" notebook interface (i.e., notebook<7.0) or the newer JupyterLab interface (i.e., JupyterLab or notebook>=7.0) and choose a default plotting backend that will work automatically in either case. In classic notebooks, it'll continue to use nbAgg. In JupyterLab & Notebook v7+, it'll now use the ipympl (a.k.a., "widget") backend, which is Matplotlib's replacement for nbAgg that works with Jupyter's new JS frontend.
This turned out to be rather tricky to do programmatically. I think I've settled on pretty good solutions throughout, but there are a few imperfections/edge cases I'm not sure can be handled perfectly in reality:
I've used a combination of a few different tricks to figure out which frontend is being used to run the importing notebook while adding as little overhead as possible to the import, but there may be some weird scenarios that aren't handled correctly (e.g., an IDE that runs its own notebook server in a managed process that can't be externally inspected)
The JS widget inside of which ipympl displays animated plots isn't identical to nbAgg's. One thing it's missing is a dedicated "stop animation" button, which I find quite annoying because the only way to stop a running animation is to repeatedly click the "interrupt kernel" button in the notebook menu bar until you happen to hit it while the next frame is being drawn, because it has no effect between frames.
The new ipympl Matplotlib backend is distributed as its own separate package, so I've also added it to requirements.txt to ensure it's installed and available. However, the package works similarly to ipywidgets in that it provides both Python components needed by the IPython kernel and JavaScript components needed by the notebook frontend. So if a user runs their notebook server and IPython kernel from different environments, ipympl would need to be installed in both, and simply pip-installing hypertools wouldn't guarantee that interactive/animated plots work out-of-the-box. The best we can really do (which I've implemented in this PR), is to check at runtime (when appropriate) whether the server and kernel environments are different, and if so, check whether ipympl is also installed in the server environment. If it's not, then we issue a warning if and when the user tries to create an interactive/animated plot noting that they need to install ipympl in the server environment manually. This still isn't perfect though because the JS components of ipympl are also available (without installing the full package) as a NodeJS JupyterLab extension called jupyter-matplotlib. So there's a small risk of issuing a false positive warning if the user has installed that extension in the server environment rather than ipympl itself.
I've tested this locally in the "classic" Notebook frontend (v6.4.7), the new Notebook frontend (v7.1.2), and JupyterLab (v4.1.5), as well as with a few different server/kernel environment configurations. Everything's working for me, but additional testing is certainly needed and would be appreciated (cc: @jeremymanning). However, I'm gonna go ahead and merge this in now so that the fix is available on master, since an imperfect fix for these issues is better than no fix at all.
This PR fixes 2 bugs uncovered in #261. Both are fairly major fixes to interactive/animated plotting functionality that we should try to get into a release version ASAP:
As of
IPythonv8.17, callback functions registered through theEventManagerinterface must now accept all arguments expected by the prototype function for the relevant event. This affects the_deferred_reset_cb()callback that Hypertools registers when temporarily changing the Matplotlib plotting backend to display an interactive/animated plot, in order to reset it to its previous state during thepre_run_cellphase of the next cell executed. Callbacks triggered by thepre_run_cellevent expect to be passed an object containing various info about the to-be-executed cell, but since the next cell's contents aren't relevant to our use case,_deferred_reset_cb()had simply omitted this argument. However, the changes in ipython/ipython/pull/14216 now cause this function to throw an error when the user goes to execute the next cell, and continue to do so for every subsequent cell executed until the kernel is restarted (or the callback is removed manually fromget_ipython().events.callbacks['pre_run_cell']).To fix this, I've updated the
_deferred_reset_cb()function to just accept and ignore any arbitrary*argspassed to it.Version 7.0 of the base Jupyter Notebook app switched over to using the same JavaScript codebase as JupyterLab, which doesn't expose the global
IPythonandJupyterJS objects that were available in previous versions of Jupyter Notebooks. Matplotlib's default backend for displaying inline interactive/animated plots in notebooks (nbAgg) relied on these objects for communicating between the frontend and kernel, and therefore no longer works in Notebook v7.0+ (nor in JupyterLab). Since Hypertools usesnbAggfor displaying interactive plots by default (i.e., unless changed viahypertools.set_interactive_backend()orhypertools.plot()'smpl_backendargument, both of which are unfortunately undocumented), interactive and/or animated Hypertools plots now throw an error with no clear fix when displayed in Notebook v7.0+ or JupyterLab.To fix this, I've added some extra logic to the function in
hypertools.plot.backendthat runs when the package is imported and initializes the Matplotlib backend used for interactive/animated plots based on the importing environment. When imported into a Jupyter notebook, Hypertools will now determine whether that notebook is being run through the "classic" notebook interface (i.e.,notebook<7.0) or the newer JupyterLab interface (i.e., JupyterLab ornotebook>=7.0) and choose a default plotting backend that will work automatically in either case. In classic notebooks, it'll continue to usenbAgg. In JupyterLab & Notebook v7+, it'll now use theipympl(a.k.a., "widget") backend, which is Matplotlib's replacement fornbAggthat works with Jupyter's new JS frontend.This turned out to be rather tricky to do programmatically. I think I've settled on pretty good solutions throughout, but there are a few imperfections/edge cases I'm not sure can be handled perfectly in reality:
ipympldisplays animated plots isn't identical tonbAgg's. One thing it's missing is a dedicated "stop animation" button, which I find quite annoying because the only way to stop a running animation is to repeatedly click the "interrupt kernel" button in the notebook menu bar until you happen to hit it while the next frame is being drawn, because it has no effect between frames.ipymplMatplotlib backend is distributed as its own separate package, so I've also added it torequirements.txtto ensure it's installed and available. However, the package works similarly toipywidgetsin that it provides both Python components needed by the IPython kernel and JavaScript components needed by the notebook frontend. So if a user runs their notebook server and IPython kernel from different environments,ipymplwould need to be installed in both, and simply pip-installing hypertools wouldn't guarantee that interactive/animated plots work out-of-the-box. The best we can really do (which I've implemented in this PR), is to check at runtime (when appropriate) whether the server and kernel environments are different, and if so, check whetheripymplis also installed in the server environment. If it's not, then we issue a warning if and when the user tries to create an interactive/animated plot noting that they need to installipymplin the server environment manually. This still isn't perfect though because the JS components ofipymplare also available (without installing the full package) as a NodeJS JupyterLab extension calledjupyter-matplotlib. So there's a small risk of issuing a false positive warning if the user has installed that extension in the server environment rather thanipymplitself.I've tested this locally in the "classic" Notebook frontend (v6.4.7), the new Notebook frontend (v7.1.2), and JupyterLab (v4.1.5), as well as with a few different server/kernel environment configurations. Everything's working for me, but additional testing is certainly needed and would be appreciated (cc: @jeremymanning). However, I'm gonna go ahead and merge this in now so that the fix is available on master, since an imperfect fix for these issues is better than no fix at all.