Discussion: Support Dataclass/Attrs decorated classes

[groutr]

Ipywidgets is a very powerful for building interfaces in notebooks. Interact is an amazing concept of automatically creating user interface controls and is the easiest way to get started using ipywidgets. It works by accepting a function and inspecting the signature of the function to discover type information for each argument. This information is used to generate various kinds of interface elements such as sliders, text areas, and dropdowns.

Another approach to automatically generating user interfaces with ipywidgets is via the param and paramnb libraries. One can use param to "annotate" class attributes with type information and other metadata that paramnb then uses to generate a set of widgets to display in a notebook.

In Python 3.7, dataclasses became part of the standard library (PEP 557). It allows one to easily create data oriented classes with type annotated attributes. It was inspired by the attrs library. I feel it would be a natural evolution of ipywidgets to grow support for dataclass/attrs decorated classes (or instances of these classes).

I believe dataclasses/attrs based UI generation to be a compelling goal would like to make that a reality. It is inspired by the param/paramnb approach, but instead uses more widely used methods of annotation.

I'm starting this discussion to address these points:

1. Is this something that should have native support in ipywidgets? Or is this better off in a separate library?
2. Direction for using the more advanced functionality of ipywidgets. I'm not an expert here and I haven't investigated how to cover some of the more advanced use cases of ipywidgets.

Proof of Concept:

A simple example:

from typing import List, Str
from dataclasses import dataclass  # Python 3.7+

@dataclass
class MyObj:
    a: int
    b: List[Str]
    c: float = 3
    d: str = 'Hello'

Produce a widget layout like:

The critical information that interact needs to generate UIs is type information for each attribute. Dataclasses require you to annotate all class attributes with types. These annotations are available via the __annotations__ dictionary. Attrs stores the same kind of information in the __attrs_attrs__ attribute of its decorated classes. These attributes point to a collection of field objects, which contain information such as field type, default values, default factory, etc.

# MyObj.__dataclass_fields__
{'a': Field(name='a',type=<class 'int'>,default=<dataclasses._MISSING_TYPE object at 0x7f9bcc63cd68>,default_factory=<dataclasses._MISSING_TYPE object at 0x7f9bcc63cd68>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),_field_type=_FIELD),
 'b': Field(name='b',type=typing.List[typing.Any],default=<dataclasses._MISSING_TYPE object at 0x7f9bcc63cd68>,default_factory=<dataclasses._MISSING_TYPE object at 0x7f9bcc63cd68>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),_field_type=_FIELD),
 'c': Field(name='c',type=<class 'float'>,default=3,default_factory=<dataclasses._MISSING_TYPE object at 0x7f9bcc63cd68>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),_field_type=_FIELD),
 'd': Field(name='d',type=<class 'str'>,default='Hello',default_factory=<dataclasses._MISSING_TYPE object at 0x7f9bcc63cd68>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),_field_type=_FIELD)}

Some examples from ipywidgets documentation

Interact allows you to set other options for controls such as bounds. Consider this example in the interact documentation:

@interact(x=(0.0,20.0,0.5))
def h(x=5.5):
    return x

Both dataclasses and attrs allow you pass a keyword argument for metadata. It is specifically designed to allow third-party extensions. We can use a special key in this metadata dictionary to hold information for ipywidgets. which using the class method would probably look something like:

@dataclass
class Identity:
    x: float = dataclasses.field(default=5.5, metadata={'__widgets__': {'max': 20.0, 'min': 0.0, 'step': 0.5}})
    @interact
    def h(self):
        return self.x

One more proof of concept from here

a = widgets.IntSlider(description='a')
b = widgets.IntSlider(description='b')
c = widgets.IntSlider(description='c')
def f(a, b, c):
    print('{}*{}*{}={}'.format(a, b, c, a*b*c))

out = widgets.interactive_output(f, {'a': a, 'b': b, 'c': c})

widgets.HBox([widgets.VBox([a, b, c]), out])

Using this dataclass approach, it could look something like:

@dataclass
class MyWidget:
    a: int
    b: int
    c: int
    def mul(self):
        return self.a * self.b * self.c

    def f(self, a, b, c): # the arguments aren't used, but there to satisfy interactive_output
        print('{}*{}*{}={}'.format(self.a, self.b, self.c, self.mul()))

    def _layout(self):
        # classes can define their own layouts
        controls = generate_widgets(self)
        out = ipywidgets.interactive_output(self.f, {c.description: c for c in controls})
        return ipywidgets.HBox([ipywidgets.VBox(controls), out])

x = MyWidget(1, 2, 3)
x._layout()

These examples are obviously a little rough, but I think they get communicate the concept. The main difference to using classes over functions is how and where state is managed.

Some sample code to generate interfaces:

_mapping_types = {
    int: ipywidgets.IntSlider,
    float: ipywidgets.FloatSlider,
    str: ipywidgets.Textarea,
    list: ipywidgets.Dropdown,
    tuple: ipywidgets.Dropdown,
    List: ipywidgets.Dropdown
}

def get_fields(class_or_instance):
    try:
        return getattr(class_or_instance, '__attrs_attrs__')
    except AttributeError:
        fields = getattr(class_or_instance, '__dataclass_fields__').values()
        return tuple(fields)

def generate_widgets(obj, _mapping_types=_mapping_types):
    w = []

    for field in get_fields(obj):
        kwargs = {'description': field.name,
                 'value': getattr(obj, field.name, field.default)}
        kwargs.update(field.metadata.get('__widgets__', {}))
        if field.type in _mapping_types:
            w.append(_mapping_types[field.type](**kwargs))
        elif type(field.default) in _mapping_types:
            w.append(_mapping_types[field.type](**kwargs))
        else:
            raise TypeError("Unable to determine type of field %s" % field.name)

    return w

Hopefully this enough get the discussion started. I think having this approach to generating UIs can the process easier to read and maintain. Ipywidgets is already very powerful, and I think dataclasses/attrs can complement that nicely. I'd love to hear feedback from more experienced users of ipywidgets.

[maartenbreddels]

Hi Ryan 👋

I love the idea, and I think already with traitlets, which can store metadata per trait/property, we could put in enough information to build UI's from Widgets itself. However, to build UI's, like interact from non-widgets, would require one to use traitlets. It would be really nice if we could instead, rely on dataclasses to do this. But since this is 3.7 only, I can imagine this will take a while before this becomes the oldest Python we support, and a new/separate library would make more sense if we take that direction.

PS: I think the metadata is general enough to change this:

> x: float = dataclasses.field(default=5.5, metadata={'__widgets__': {'max': 20.0, 'min': 0.0, 'step': 0.5}})

to

x: float = dataclasses.field(default=5.5, max=20.0, min=0.0, step=0.5)

[groutr]

Thanks @maartenbreddels. I think my motivation/project focuses more on building a UI from non-widgets. Essentially, I want to use widgets to represent and manipulate the attribute state of a class instance and have that representation generated automatically. Dataclasses/attrs, in my mind, provide a natural way to provide the information required generate a UI to represent the attributes as widgets.

dataclasses is a Python 3.7+ feature, however, attrs, the inspiration for dataclasses, is Python 2.7/3.4+. If Ipywidgets were to depend on attrs, these kinds of decorated classes could be used much sooner. I'm also a fan of a new/separate library in order to keep dependencies separate and to also be independent of the ipywidgets release cycle. The reason I focused on dataclasses because 1) I was playing with Python 3.7 and 2) attrs supports more features than dataclasses and I wanted to keep the initial concept more simple.

Regarding the field metadata, the dataclasses (and attrs) suggest the following convention:

It is not used at all by Data Classes, and is provided as a third-party extension mechanism. Multiple third-parties can each have their own key, to use as a namespace in the metadata.

The __widgets__ key was an attempt to carve out a namespace for widget metadata while being nice to other metadata.

In both attrs and dataclasses, this will raise an exception:

> x: float = dataclasses.field(default=5.5, max=20.0, min=0.0, step=0.5)

TypeError: field() got an unexpected keyword argument 'max'

fields use Python 3's keyword-only arguments instead of **kwargs.

[groutr]

I think this is a more succinct statement of what I was trying to explore last night:

ipywidgets.interact[ive] already inspects function arguments. We could extend that to also inspect class instance attributes using the same inspection logic that is used for function arguments. Then if a class instance was decorated with dataclasses or attrs, ipywidgets could understand how to extract field information and other metadata from the field definitions. That's the overall idea. It wouldn't require ipywidgets to depend on Python 3.7+ or attrs.

[vidartf]

@groutr I think this looks cool! I agree that a separate package is probably best. Then, once ipywidgets requires python 3.7 or greater, we can consider the other pros and cons of merging.

On the details, I'm thinking that the dataclasses.field with metadata is not ideal. It is verbose, and does not give any errors if you have a typo in the metadata (setp=0.0). I think using that as a declared and accepted format could be helpful, but that some helper functions could be added. Then people can either use fields directly or the more convenient helpers.

@maartenbreddels If you want to replace traitlets, then you have my vote, but that's not going to be an easy process. 😅

[groutr]

I created a small demo that is available in this repo: https://github.com/groutr/datawidgets

Test notebook. https://github.com/groutr/datawidgets/blob/master/datawidgets/Test.ipynb

ping @maartenbreddels @jasongrout

[stefanseefeld]

I was just looking around for existing modules / widgets for a more convenient and nicer display of dataclasses in notebooks, and stumbled across this discussion. What a great idea ! More than four years have passed since the last post. What is the current state of things ? (Googling for "datawidgets" I find a number of other packages, but apparently with quite a different scope. I'm really interested into the automatic generation of UI elements from dataclass metadata.)