xwidgets 2.0

[rapgenic]

In this PR is being developed a mainly octave based (i.e. not C++) implementation of Jupyter widgets.

The three main components of this implementation are:

• The basic C++ underlying octave classdef (which is the octave type for classes) implementation, that provides a superclass that native Octave widgets can inherit
• The actual widgets, written in octave code
• A widget template and generator, which goes through all the python widgets and generates a corresponding octave classdef implementation

I'll try to describe each component.

The C++ superclass

The C++ superclass is implemented in the files xwidgets2.cpp and xwidgets2.hpp. Here a __xwidget_internal__ base classdef is defined and registered in octave symbol table (in the register_all2 function) for all widgets to subclass.

This classdef provides the following functionality:

• Comm handling
• Property (e.g. traits) synchronisation towards view when changed from octave context
• Property synchronisation from view when changed in JupyterLab frontend
• Observer pattern for properties
• Custom message handling (not done yet)

To understand how this works it is necessary to know how octave builtin values are stored in C++. For each kind of object octave uses two classes:

• a rep one which implements the actual functionalities of the object
• and a wrapper one, that takes a rep pointer at construction and usually simply forwards function calls to its rep.

This means that the functionality of a type can be changed simply by using a different rep class.

This is what I've done here: I have defined a new class that inherits from both octave::handle_cdef_object (that provides the implementation of handle classdef objects) and xw::xcommon for xwidget functionality.

By overriding a few octave::handle_cdef_object member functions (i.e. put and mark_as_constructed), we can implement comm handling, property synchronisation and observer patterns. Not going into too many details here for legibility.

The "replacement" of the rep class happens inside the classdef constructor (cdef_constructor): that discards the class instance that had been built by the octave interpreter and returns a new one with our modified rep. This is mainly the one slightly hacky part, as it uses some more internal octave API.

The Octave implementation

The `__xwidget_internal__' class can be inherited by octave classdefs to create a new widget. The implementation can be as simple as this:

classdef Button < __xwidget_internal__
    properties (Sync = true)
        _model_module = "@jupyter-widgets/controls";
        _model_module_version = "2.0.0";
        _model_name = "ButtonModel";
        _view_module = "@jupyter-widgets/controls";
        _view_module_version = "2.0.0";
        _view_name = "ButtonView";
        disabled = false;
    endproperties
endclassdef

button = Button;
button.disabled = true; % Synchronised

All properties declared with attribute Sync=true are automatically synchronised between backend and frontend.

Note that at the moment octave does not yet provide classdef property validation, so additional setters and getters should be defined for each property in order to check their correctness.

The generator

In order to overcome the difficulty of implementing by hand each widget, a generator has been provided, taking inspiration from here. The generator simply loops through all defined widgets, and through the use of a python Mako template is able to generate the corresponding octave widget implementation, with some form of validation as well.

The generator binary and template can be found in share/xeus-octave/+widgets/.

Widgets are automatically generated and installed by CMake scripts.

Note: The generator works with ipywidgets version 8

Todo

• Tests (this is for me a bit painful as I have never really used them properly until now). We should be able to test:
  • octave_value -> json conversion and viceversa
  • validation functions
  • communication between frontend and backend
• Documentation
  • Would be nice to have the generator do this from ipywidgets documentation, with some parsing and conversion to texinfo
• Implementing custom messages (e.g. button click)
• Interact function (in the works)
  • It would be nice to use a parse tree walker to identify automatically the parameters of the function and choose which widgets to use for interactivity (as octave does not provide natively a way to specify parameter types)
• A xfigure widget for plots (in the works)
• Probably more, I'll add here...

Note on PR #87: this is a complete rewrite (architecturally as well), the previous PR, which at the moment I'm not developing at all, explored the use of xwidgets defined in C++ (in the xwidgets library) and used some complicated templating system to implement them natively in C++ as single octave classdefs. Of course that prevented extending them from octave and creating new ones without using C++.

Closes: #7

[rapgenic]

@AntoinePrv @SylvainCorlay,

I'm not asking for a review of this, but as you seemed interested in the xwidgets work I just wanted to tell you that this PR is getting a good shape now, so you should be able to poke around with it without getting a head ache in case you're interested.

I've added some documentation on the top comment for information on the architecture.

[AntoinePrv]

Hi @rapgenic sorry for the late reply.

This is still in the back of my head, but, to be fully transparent, I also have other projects to tend to at the moment. I am not sure I can provide a valuable review until I learn how one binds C++ functions in Octave.

We had a couple high level discussion with @SylvainCorlay about how this should get integrated in the ecosystem. There is already a bit of manual work in XWidget to keep it in sync with IPyWidget. This work gets duplicated here, and in any possible binding of XWidgets. The long term solution will be, as you did, to generate the widgets from a script/template (Jinja?). In XWidget we started using a schema file from IPyWidget in testing but we're far from generating them from script. As we do so, we'll make sure to design it in a way that is reusable by all kernels. In the meantime, we do not wish to slow down the great work you are doing here. Quite the contrary, this will provide valuable insight for a more general solution!

Which brings me to the next point, since merging this would tie XOctave to a particular XWidget version, and that we currently do not have an efficient way of updating XWidgets, perhaps it would be more suitable to provide this as a separate Octave package (I think we already discussed this). I know getting started with packaging can be time consuming (maybe you know Octave packaging actually), so in the meantime we could keep this in a separate XOctave branch, and even do a couple dev releases with it (they are handled separately in conda-forge) before moving it to its own package.

What do you think about that plan? Anyways, thanks and congrats for the good work you do here! I myself am just getting started with XWidgets :smiley:

[rapgenic]

This is still in the back of my head, but, to be fully transparent, I also have other projects to tend to at the moment. I am not sure I can provide a valuable review until I learn how one binds C++ functions in Octave.

Don't worry about that, I don't think this is ready for review/merging anyway, I just wanted to signal that it's in a usable state, in case you wanted to try it!

There is already a bit of manual work in XWidget to keep it in sync with IPyWidget. This work gets duplicated here, and in any possible binding of XWidgets.

Just to be clear, I designed this to avoid at all the manual work, as I hate (and do not have time) to maintain a few dozens of widgets manually...

In XWidget we started using a schema file from IPyWidget in testing but we're far from generating them from script. As we do so, we'll make sure to design it in a way that is reusable by all kernels.

I actually had found a schema here but I found much easier iterating directly the Widget instances from the python package (even though I admit it's surely less portable... but anyway we'd only need to maintain one script)

As we do so, we'll make sure to design it in a way that is reusable by all kernels.

When you are designing some way to generate automatically the widgets code, you might want to let me know, I have found a few problems with this approach while developing this (mainly while generating documentation and handling custom comm events) that could surely be improved!

Which brings me to the next point, since merging this would tie XOctave to a particular XWidget version, and that we currently do not have an efficient way of updating XWidgets, perhaps it would be more suitable to provide this as a separate Octave package (I think we already discussed this).

I had not thought about this, but this is actually a good idea. Octave folks have provided a package template that I could look into. This would actually be useful for me as well to avoid building the kernel each time. The only problem might be with plot generation (I'm not sure how to integrate with the toolkit)

Also, this uses a very small part of xwidgets (i.e. only xcommon) and half of it is reimplemented... so I was actually thinking about rewriting the other half and using directly the comm class provided by xeus itself.

I know getting started with packaging can be time consuming (maybe you know Octave packaging actually), so in the meantime we could keep this in a separate XOctave branch, and even do a couple dev releases with it (they are handled separately in conda-forge) before moving it to its own package.

I think we could do both, but in the end I actually think that a package might be the best idea (provided it is feasible), even just if we wanted to develop kernel and widgets at a different pace.

I myself am just getting started with XWidgets

That's great! Some things are quite obscure to me and another pair of eyes will be useful!

[rapgenic]

Done! https://github.com/rapgenic/xoctave-widgets

Can be installed by (even in the current kernel release, provided you install the following conda packages: mako, ipywidgets=8, xwidgets)

pkg install -verbose "https://github.com/rapgenic/xoctave-widgets/archive/refs/heads/main.zip"

Development will probably move there, I guess we can call it "widgets 3.0" :laughing: Thank you @AntoinePrv for the idea!

It can be used like this:

pkg load widgets
b = Button
b.description = "It works!";
b.on('click', @(obj) disp("Yeah!"));

[SylvainCorlay]

Just for the record, I think that this approach is much better than the other PR.