Dynamically add Input, Output, Settings, ...

[woutdenolf]

As we are trying to create a mapping between our common task graph representation (to be scheduled with luigi, dask and other task schedulers), we end up with lots of boilerplate code for each widget:

from Orange.widgets import gui
from Orange.widgets.widget import OWWidget, Input, Output
from Orange.widgets.settings import Setting

from esrf2orange3 import OWWidgetTaskExecutor
from esrf2orange3.gui.parameterform import ParameterForm
from tasklib import SumTask
from esrftaskgraph import Variable

__all__ = ["Adder1"]

class Adder1(OWWidget):
    name = "Adder1"
    description = "Adds two numbers"
    want_main_area = False
    esrftaskclass = SumTask

    class Inputs:
        a = Input("A", Variable)
        b = Input("B", Variable)

    class Outputs:
        result = Output("A + B", Variable)

    static_input = Setting({"a": 0, "b": 0})
    varinfo = Setting({"root_uri": "/tmp"})

    def __init__(self):
        super().__init__()

        self._esrfexecutor = OWWidgetTaskExecutor(self)

        box = gui.widgetBox(self.controlArea, "Static Inputs")
        self._static_input_form = ParameterForm(parent=box)
        for name in self._esrfexecutor.input_names:
            value = self.static_input.get(name, 0)
            self._static_input_form.addParameter(
                name, value=value, changeCallback=self.changeStaticInput
            )

        box = gui.widgetBox(self.controlArea, "Dynamic Inputs")
        self._dynamic_input_form = ParameterForm(parent=box)
        for name in self._esrfexecutor.input_names:
            self._dynamic_input_form.addParameter(name)

        box = gui.widgetBox(self.controlArea, "Outputs")
        self._output_form = ParameterForm(parent=box)
        for name in self._esrfexecutor.output_names:
            self._output_form.addParameter(name)

        self.handleNewSignals()

    def changeStaticInput(self):
        self.static_input.update(self._static_input_form.getParameters())
        self.handleNewSignals()

    @Inputs.a
    def settera(self, var):
        self._esrfexecutor.set_input("a", var)

    @Inputs.b
    def setterb(self, var):
        self._esrfexecutor.set_input("b", var)

    def handleNewSignals(self):
        for name, value in self._esrfexecutor.dynamic_input_values.items():
            self._dynamic_input_form.setParameter(name, value)
            self._static_input_form.disable(name)
        self._esrfexecutor.run()
        for name, value in self._esrfexecutor.output_values.items():
            self._output_form.setParameter(name, value)

The Inputs, Ouputs, Setting as well as the input setters are all boilerplate (can be derived from SumTask). Any suggestions on how we can add those class attributes dynamically? It seems subclassing OWWidget is forbidden for some reason (https://github.com/biolab/orange-widget-base/issues/144). How about adding a metaclass?

[irgolic]

Woah this is really cool. We're thinking of moving core Orange to a task graph system (using dask arrays), so this is right up our alley. Would you be willing to share your code with us? :)

Regarding adding class attributes dynamically, I suggest a metaclass. Make sure the metaclass extends WidgetMetaClass.

[janezd]

See #144. You can derive from any class, for as long as the ancestor (that is, your class containing the boiler code) is marked with openclass=True.

Meta classes are of course an option, too, but I'd strongly encourage going with normal subclassing.

While I was writing this, @irgolic suggested meta classes. If all your widgets have the same inputs and outputs, you don't need meta classes. If they are different, this would be parametrized meta class. Go for it, if you really must. :)

[irgolic]

Judging only by the code snippet you posted here, I think I may've solved this exact problem for orange3-pandas, wherein widget UIs are generated from pandas docstrings/signatures.

It's not quite ready/released yet, but if you say hi on Discord, I can share.

[woutdenolf]

@irgolic We don't even have proper projects yet but you could already have a look at this: https://gitlab.esrf.fr/denolf/workflow_concepts

It's an attempt at finding the optimal ECO system of projects suitable for workflows at the ESRF. Orange3 would be part of that.

We will probably have one core project for representing task graphs (persistent and at runtime): https://gitlab.esrf.fr/denolf/workflow_concepts/-/tree/master/esrftaskgraph

And then bindings for task schedulers and graph design GUIs. This is what we have so far for Orange: https://gitlab.esrf.fr/denolf/workflow_concepts/-/tree/master/esrf2orange3

And this is the binding for dask: https://gitlab.esrf.fr/denolf/workflow_concepts/-/tree/master/esrf2dask

[irgolic]

I don't think I can access your internal gitlab.

[woutdenolf]

Ah sorry, it was not even a public project yet, that's how recent this is ;-). Try again.

[woutdenolf]

I tried with a metaclass and a subclass but I need to do this

class WidgetMetaClass(type(QDialog)):
    def __new__(mcs, name, bases, namespace, openclass=False, **kwargs):
        cls = super().__new__(mcs, name, bases, namespace, **kwargs)
        if not cls.name: # not a widget
            return cls
         # >>> HERE I NEED TO ADD inputs, outputs and settings
        cls.convert_signals()
        cls.settingsHandler = \
            SettingsHandler.create(cls, template=cls.settingsHandler)
        return cls

Any suggestions? Of course I could do something dodgy with convert_signals but that might break in the future.

[woutdenolf]

No in fact I cannot even do that because I have no way of passing new arguments.

class ESRFWidgetMetaClass(WidgetMetaClass):
    def __new__(
        metacls,
        name,
        bases,
        attr,
        esrftaskclass=None,
        input_names=None,
        output_names=None,
        **kw
    ):
        if input_names is None:
            input_names = dict()
        if output_names is None:
            output_names = dict()
        prepare_kw = {
            "esrftaskclass": esrftaskclass,
            "inputnamemap": input_names,
            "outputnamemap": output_names,
        }
        # _prepare_owesrfwidgetclass need to access prepare_kw
    return super().__new__(metacls, name, bases, attr, **kw)

    @classmethod
    def convert_signals(cls):
        cls._prepare_owesrfwidgetclass()
        super().convert_signals()

    @classmethod
    def _prepare_owesrfwidgetclass(cls):
          ...

[irgolic]

You might need to do something funky if you wanted to use the signals syntactic sugar, but if you add to attr (namespace) a list named inputs of (name, type, method_name, flags) tuples, it should be fine.

[woutdenolf]

Ok perfect, I can do what I want now by using WidgetMetaClass and subclassing with openclass=True. Thanks!

[janezd]

I have foreseen that. I wrote that you'll need parametrized meta classes. :)

If all your subclasses have the same inputs and outputs, then add them to your base class, not the meta class. You have an example here: https://github.com/biolab/orange3/blob/master/Orange/widgets/utils/owlearnerwidget.py#L54. This is an open widget class that defines signals, and errors and a part of GUI... Derived classes can add to that. For instance, Logistic regression (https://github.com/biolab/orange3/blob/master/Orange/widgets/model/owlogisticregression.py#L15) adds an additional output and warning, by deriving its output from the inherited class with outputs.

I still don't see why would you need meta classes.

But if you do, you can do two things.

You can add arguments to new, just differently. You can define your meta class exactly as you did above, and then use it like this

class MyWidget(OWBaseWidget, metaclass=ESRFWidgetMetaClass, esrftaskclass=..., input_names=...):
    ...

The arguments to the meta class are given as the arguments to the widget class definition. Incidentally, this is how openclass is implemented in Orange. See https://github.com/biolab/orange-widget-base/blob/master/orangewidget/widget.py#L96: openclass is an extra argument to __new__ and it's value (which is False be default) is overridden in, for example, the code I linked to above. https://github.com/biolab/orange3/blob/master/Orange/widgets/utils/owlearnerwidget.py#L54.

The other way is totally fancy. Parametrizing meta classes via closures.

def my_metaclass(esrftaskclass=None, input_names=None, output_names=None):
    class ESRFWidgetMetaClass(WidgetMetaClass):
        def __new__(metacls, name, bases):
            ...

    return ESRFWidgetMetaClass

class MyWidget(OWBaseWidget, metaclass=my_metaclass(my_specific_task_class, my_input_names)):
    ....

In this way, every widget will have his specific meta class.

[janezd]

Ok perfect, I can do what I want now by using WidgetMetaClass and subclassing with openclass=True. Thanks!

No, no, don't. Just pass openclass=True to the widget from which you plan to derive other widgets. Check this: https://github.com/biolab/orange3/blob/master/Orange/widgets/data/owpreprocess.py#L1066. This is the widget that preprocessing widgets are derived from.

class OWPreprocess(widget.OWWidget, openclass=True):
    ...

    class Inputs:
        data = Input("Data", Orange.data.Table)

    class Outputs:
        preprocessor = Output("Preprocessor", preprocess.preprocess.Preprocess, dynamic=False)
        preprocessed_data = Output("Preprocessed Data", Orange.data.Table)

    storedsettings = Setting({})
    autocommit = Setting(True)
    PREPROCESSORS = PREPROCESS_ACTIONS
    CONTROLLER = Controller

    ...

[woutdenolf]

Yes I got that. I have one common base widget from which I will be deriving my normal widgets

class OWESRFWidget(OWWidget, metaclass=ESRFWidgetMetaClass, openclass=True):
    ...

class MyWidget1(OWESRFWidget, esrftaskclass=MyClass1):
    ...

class MyWidget2(OWESRFWidget, esrftaskclass=MyClass2):
    ...

ESRFWidgetMetaClass adds inputs, outputs and settings based on the esrftaskclass argument. The classes MyClass1, MyClass2, ... do the actual computations (they also know what the inputs and outputs are). This is how we want to decouple GUI from computation so we can execute task graphs without needing the canvas (or Orange for that matter) by different task schedulers (luigi, dask, ... depending on what kind of distribution we need).