Create MagicUI based UI using on a JSON or similiar

[sebi06]

Hi,

imagine you have a JSON file, that is used to define the input elements for an UI (example below). What would be the best approach to get started to create a magicgui-based interface using this as an input? or in other words, is there an example etc., where can I get an idea how to "dynamically" create an UI based on a JSON or dict etc?

The idea is to (maybe) create a Napari plugin that can run our APEER modules, which come with such an Input UI specification. In our ZEN software and on www.apeer.com we already use such specs to create the UI dynamically and I wonder if this would work using magicgui (inside Napari) as well?

We already a commandline-based APEER module executor, which is completely docker-based and can in principle run on Windows on Linux.

Any ideas are appreciated?

{
    "spec": {
        "inputs": {
            "input_image": {
                "type:file": {}
            },
            "red": {
                "type:number": {
                    "lower_inclusive": 0.0,
                    "upper_inclusive": 1.0
                }
            },
            "green": {
                "type:number": {
                    "lower_inclusive": 0.0,
                    "upper_inclusive": 1.0
                }
            },
            "blue": {
                "type:number": {
                    "lower_inclusive": 0.0,
                    "upper_inclusive": 1.0
                }
            }
        },
        "outputs": {
            "tinted_image": {
                "type:file": {}
            }
        }
    },
    "ui": {
        "inputs": {
            "red": {
                "index": 51,
                "label": "Red channel",
                "widget:slider": {
                    "step": 0.1
                }
            },
            "green": {
                "index": 52,
                "label": "Green channel",
                "widget:slider": {
                    "step": 0.1
                }
            },
            "blue": {
                "index": 53,
                "label": "Blue channel",
                "widget:slider": {
                    "step": 0.1
                }
            }
        },
        "outputs": {}
    }
}

This renders in our SW like this:

[tlambert03]

Hey @sebi06. This is definitely something we want to have first-class support for. I'm realizing we don't have an open issue yet, so thanks for opening! We're not there yet, but it's considered very high priority.

The central question of course is the format of that JSON. When I think about this, I've generally planned to start by supporting JSON Schema, which is quite easy to convert into a python model using pydantic... and at that point, we're very close to being able to render a GUI with magicgui (though we will likely need to add a couple more widget types).

Is there a schema somewhere for that apeer UI JSON?

[sebi06]

Hi @tlambert03

that was quick :-). This is not really an issue but rather some kind of feature request or a search for ideas on how to get started.

With the APEER UI JSON you mean this below. I will have a look inro pydantic ...

May basic idea would be:

• user creates account on APEER
• user downloads module from APEER (or maybe even browse inside Napari one day? - we alreayd have tokens & API-Keys etc.)
• module UI specs is used to render the UI of the napari plugin (as it is in ZEN)
• run module with open image in Napari --> Docker takes care of the rest (pull the module) and the APEER module executor (also a docker container) does the execution
• output image (or table) is shown in Napari

Does this idea makes sense to you?

{
  "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
  "$id": "http://127.0.0.1/apeer_module_inputs.schema.json",
  "title": "Input value schema for apeer module executions",
  "description": "A schema defining the input values for an apeer module execution.",
  "self": {
    "vendor": "com.zeiss",
    "name": "apeer-module-inputs",
    "format": "jsonschema",
    "version": "1-0-0"
  },
  "type": "array",
  "items": {
    "anyOf": [
      {
        "$ref": "#/definitions/boolean"
      },
      {
        "$ref": "#/definitions/string"
      },
      {
        "$ref": "#/definitions/number"
      },
      {
        "$ref": "#/definitions/stringList"
      },
      {
        "$ref": "#/definitions/decimalInterval"
      }
    ]
  },
  "definitions": {
    "boolean": {
      "type": "object",
      "patternProperties": {
          "[A-Za-z:0-9]": { "type": "boolean" }
      },
      "minProperties": 1,
      "maxProperties": 1
    },
    "string": {
      "type": "object",
      "patternProperties": {
          "[A-Za-z:0-9]": { "type": "string" }
      },
      "minProperties": 1,
      "maxProperties": 1
    },
    "number": {
      "type": "object",
      "patternProperties": {
          "[A-Za-z:0-9]": { "type": "number" }
      },
      "minProperties": 1,
      "maxProperties": 1
    },
    "stringList": {
      "type": "object",
      "patternProperties": {
          "[A-Za-z:0-9]": {
              "type": "array",
              "items": {
                  "type": "string"
            }
          }
      },
      "minProperties": 1,
      "maxProperties": 1
    },
    "decimalInterval": {
      "type": "object",
      "patternProperties": {
          "[A-Za-z:0-9]": {
              "type": "object",
              "required": [
                "min",
                "max"
              ],
              "additionalProperties": false,
              "properties": {
                "min": {
                  "type": "number",
                  "description": "The interval's minimum value."
                },
                "max": {
                  "type": "number",
                  "description": "The interval's maximum value."
                }
              }
            }
          },
      "minProperties": 1,
      "maxProperties": 1
    }
  }
}

[tlambert03]

yep, all makes sense. would be an awesome use case. I haven't looked into apeer or the module system, but provided that you can get that JSON string in your first post, it would be feasible to generate a UI for it (some more thoughts on that below). As for napari-->docker APEER-->napari, that also seems feasible, but of course, much more involved than the GUI generation... so let's leave that discussion for a future date :)

If you want to get started playing with this now, the challenge is in converting the schema from your ui json to python types that magicgui can recognize. So basically, you need to convert this:

{
            "red": {
                "type:number": {
                    "lower_inclusive": 0.0,
                    "upper_inclusive": 1.0
                }
            },

            "red": {
                "index": 51,
                "label": "Red channel",
                "widget:slider": {
                    "step": 0.1
                }
            }
}

into this:

from magicgui import widgets

red_slider = widgets.FloatSlider(min=0, max=1, step=0.1, label='Red Channel')

or, probably more useful in this case is the magicgui.widgets.create_widget factory function:

red_slider = widgets.create_widget(
    annotation=float,
    widget_type='FloatSlider',
    label='Red Channel',
    options={'min': 0, 'max': 1, 'step': 0.1}
)

to get things playing nicely in napari, you can use a type of napari.types.ImageData for your input_image field (which, once the widget is in a napari viewer, will render as a dropdown menu showing the available image layers).

from napari.types import ImageData

image_dropdown = create_widget(annotation=ImageData, label = 'Input Image')

you can then combine these subwidgets into a container:

container = widgets.Container(widgets=[red_slider, image_dropdown, ...])

# and add that to napari:

viewer.window.add_dock_widget(container)

This doesn't cover what events happen when the user interacts with your GUI... but it's a start. See https://napari.org/guides/stable/magicgui.html for more details on napari-specific types that you can use in magicgui (like ImageData), and see the magicgui widget overview and associated docs for more details on the fields accepted by magicgui.

But again: all of this should be eventually done automatically for you!

[sebi06]

Hi @tlambert03

cool. I will play around a bit. You are right, this is what I have in mind and at some point this should work automatically :-).

With respect to the idea of running APEER modules as Napari plugins, whom should I talk to as well? @sofroniewn maybe?

If you are interested, I can give a quick intro into the APEER module system and how this works. Just drop me a message.

[tlambert03]

With respect to the idea of running APEER modules as Napari plugins, whom should I talk to as well? @sofroniewn maybe?

we're all the right people to talk to there. but that should go in the https://github.com/napari/napari issues. The big challenge here is interprocess communication. It's something we need for a lot of things, both related to plugins and internally. But there's a lot of different ways to consider doing it, and it will just take a lot of discussion before we reach a consensus "how a plugin should start and communicate data with a subprocess".

[sebi06]

@tlambert03

in the first take APEER modules take files as an input and produce files as an output as well. So maybe we can "workaround" subprocesses in the beginning ... :-)

But of course you are completly right. This will be a tricky topic. I will give it some more thoughts and start a discussion on the napari forum. Maybe we can come up with some "small" steps towards this idea first and maybe one needs to try out different things before one can make an educated decision.

Especially for me lot of this will be one (or two) complexity levels above what I typically do but I think the idea is worth discussing it.

[tlambert03]

in the first take APEER modules take files as an input and produce files as an output as well. So maybe we can "workaround" subprocesses in the beginning ... :-)

ok, that does simplify things to start with! so, a basic first implementation might look something like this:

1. manually start up an apeer docker server (assuming there's something like a server?)
2. in another process, start up napari.
3. in your napari widget, you'd have some interactive element (such as that generated by magicgui) that would either ping your server notifying it of some workflow, or, I guess, drop a file into a folder that the server is watching?
4. apeer does its business
5. then you need something back in napari to get notified when a result is ready. This could be something like folder "watcher" running in another thread (something like watchdog, or we can roll our own type thing). And then that watcher needs to emit an event "file ready" that is connected to some callback in your widget that reads the file and adds it to the viewer.

there's obviously a lot of important details left out there :joy: but that's what a basic two-process file-based communication scheme could start out as