a "recipe system" for GUIs

[piever]

This is an issue to discuss the design of a "recipe" system for GUIs. What is necessary is a small light dependency package such that each package can define complex widgets to simplify usage (for example. GLM may add a way to select variables from a table with a dropdown, a modelling package could allow choosing some parameters with sliders / togglebuttons etc.). This implies (unless we recommend some other solution like Requires.jl) having a minimal package that is sufficient to define these complex widgets and is 100% reliable and solid dependency, tentatively called UIRecipesBase.

Option 1 (package of placeholders):

• write placeholders for all the available widget functions (widget, input, slider, checkbox, etc...)
• write placeholders for the helper functions to work with Vue (kwargs2vueprops, props2string...)
• write placeholders for macro dom_str, Node, Observable, observe... (though actually Observables is so lightweight that maybe it can be included in the dependencies)
• define the Widget type? this is actually a bit tricky, as the Widget type requires Node and Scope to type the fields, so it may be better to have a function that returns a Widget type and this function would have a placeholder in UIRecipesBase

I'm not very fond of this option as all JuliaGizmos packages would have to depend and extend this thing, which seems somewhat awkward.

Option 2 (overloading):

This is quite interesting, and would correspond to transforming all widgets into something that can be obtained as widget(type,...). We already have a function widget that allows to access most widgets (colorpicker, date picker, input, etc... The action plan would be as follows:

• instead of slider, dropdown, textbox etc... define dispatches for widget(::Val{:slider}, ...) and then define slider(args....; kwargs...) = widget(::Val{:slider}, args..., kwargs...). This could be done by a macro:

@recipe function slider(args...; kwargs...)
# write function body here
end

that defines the dispatch and the shorthand slider. Though actually in InteractBase there's a place where it's very easy to define all of this, which is where I define the method without the first type argument (for the backend) for all widgets, so I could simply add it there.

• the custom widget will then be a set of widget with some layout. The layout part is tricky: I would propose some sort of placeholder approach where we add a function node end placeholder (that WebIO extends to return Node) that users can use to create layout with flexboxes. Otherwise, we could add some placeholders for CSSUtil functions (hbox, vbox, hskip, vskip, hpad, etc...)

For this macro to work, the user would have to return a tuple of two elements: a node and an observable (or three elements if they want a primary scope) and then the macro would take care of defining the function that actually returns a widget object.

Defining type recipes would simply involve overloading widget with a custom type.

I'm personally in favor of option 2, but happy to discuss alternatives.

cc: @shashi

[piever]

After playing a bit with it locally, a minimal version of option 2 would mainly require:

1. Defining widget, observe and node placeholders in UIRecipesBase
2. Define widget and observe here to extend the definition in UIRecipesBase
3. Define node here extending the definition in UIRecipesBase with node(args...; kwargs...) = Node(args...; kwargs...)
4. Define widget(::Val{:slider}, args...; kwargs...) = slider(args...; kwargs...) and so on for all widgets here
5. Add a macro in UIRecipesBase that simplifies doing point 4.

The recommended way to define the layout in a recipe would then be with node and using CSS Flexbox i.e. node(:div, w1, w2, style = Dict("display"=>"flex", ...))

[shashi]

Nice issue! You're definitely thinking about the right questions here!! Have you thought about what the DSL would look like? This seems to be one of those cases where I feel not using macros is a missed opportunity.

Something about the placeholders doesn't seem very satisfying. While I think this package being "light weight" is nice, I feel like the more important focus should be on making quick interactive GUIs easy and general. We should be able to recommend that users use packages from the whole WebIO ecosystem. One should be able to use anything that updates an observable as input to a UI module. InteractBase probably doesn't need to get any special treatment at least at the moment.

To keep this package light weight, we could go the @manipulate approach of figuring out the right widgets for inputs of common types, try to have a sensible default layout, but defer to the ecosystem of packages for anything more special...

[shashi]

One more thing to think about is chaining / composing -- can we take one recipe and fit it with other recipes to build a pipeline of interactions? For example,

1. load image -> apply certain filters with certain parameters (selected by UI of course) -> plot a histogram
2. load image -> apply certain filters -> apply certain filters (self-chaining)

[piever]

I really think a key example here is how recipes work for plots, where the function plot is called recursively. In this design all widgets would be accessible using widget(Val{wdgname}(), args...; kwargs...). There could also be widgets of custom (non Val) types, where of course you'd need to have access to the type to use them. For example, Sputnik could be adding a dispatch widget(t::NextTable).

Concerning layout, my idea (to be polished, it's more of an intuition right now) is to have widget call itself recursively for vectors and tuples:

widget(t::Tuple) = hbox(map(widget, t)...) # primary_obs would be tuple of the various primary_obs
widget(t::Array) = vbox(map(widget, t)...) # primary_obs would be vector of the various primary_obs

In this way the following would create a column of checkboxes next to a column of sliders.

widget((["Checkbox Column", false,  false],
             ["Slider Column", 1:150, 0:0.1:1])

It only gets a bit awkward when a widget requires more than one argument (for example, how does one use checbox(true, "My Label") ?). This is probably where we need to use macros for syntactic sugar. Otherwise, a way to accomodate this would be to have row vectors (meaning 1xN) do hbox and use tuples of normal elements and pairs for this design to work (bonus advantage of using row and column arrays for layout: one can use vcat and hcat for the layout), meaning:

widget((false, "Click here", "style" => "color:red;")) == widget(false, true, style="color:red")

so in the above for custom style one could use:

widget([["Checkbox Column", (false, "c", "style"=>"color:red;"),  false] ["Slider Column", 1:150, 0:0.1:1]])

I think it's important to get a solid, macro free, verbose design first and then put syntactic sugar on top, so that users who do not want to use macros have an alternative: the nice feature of the above is that it works without macro magic but we can still think how to use macros to simplify it.

Inspiration from Makie

I've looked at Makie recipes and (just like Plots recipes I suppose) they have this interesting idea of a "argument transformation pipeline". This means we could have a function called transform_argument that is called recursively (I guess it can be some fallback widget(args...) = widget(transform_argument(args...)). It could be interesting to have something similar for widgets of custom types (meaning, if I want to do widget of say a DataFrame I only need to teach it how to translate the DataFrame into agruments widget can understand.

(I'm writing in stream of consciousness mode but I'm still in the process of clarifying my thinking on this design)

[piever]

Another consideration is how to make observables accessible for complex widgets: if the ui widget is the combination of many widgets in some layout, what should its primary_obs be and how should one access the "inner" observables? Maybe worth to have some composite widget type which has a collection of widgets, an easy way of accessing their observables and information about how to display all widgets (layout).

[shashi]

It seems OK to have some placeholders like that. :+1: But it's good to remember this is special extra stuff we're doing to reduce dependencies, and distrust that.

In this way the following would create a column of checkboxes next to a column of sliders.

So a Tuple is hbox?

widget([["Checkbox Column", (false, "c", "style"=>"color:red;"), false] ["Slider Column", 1:150, 0:0.1:1]])

If so, in this case, the Tuple is ambiguous with layout. Why not just use widget here? widget([["Checkbox Column", widget(false, "c", "style"=>"color:red;"), false] ["Slider Column", 1:150, 0:0.1:1]])

We'd just need to forward 2nd argument and rest to the checkbox widget.

At first I thought you meant "Checkbox Column" to become a textbox widget, but then I realized you want it as a label. So that's ambiguous...

Another consideration is how to make observables accessible for complex widgets

I can see 2 ways:

1. An Observable of dictionaries (or a Dict of Observables)
2. A set of Observables identified by name.

But obviously if in the non-macro API, the user would also need to annotate which observable comes from which widget. In which case I doubt something like

widget([["Checkbox Column", widget(false, "c", "style"=>"color:red;"), false] ["Slider Column", 1:150, 0:0.1:1]])

would suffice

We'd need:

widget([["Checkbox Column",
    widget(false, "c", "style"=>"color:red;", value=obs1), 
    widget(false, value=obs2)
] ["Slider Column", widget(1:150, value=obs3), slider(0:0.1:1, value=obs4)]])

Zooming out to a higher level of abstraction, I'm still thinking about this:

when I suggested:

UIBase.@defmodule ridgelinear begin
    input::Table     # this will call a generic function such as widget(::Type{Table}) which creates a table input widget (similar to how we implement @manipulate)
    field::Symbol in colnames(input) # this will call widget(Symbol, colnames(input))  
    param1::Int in 1:100 # this will call widget(Int, 1:100)
    plot_title::String="Title" # maybe widget(String, "Title")

    function output() # can use any of the parameters above
    end # the result of this function is passed to WebIO.render

    function render() # Define WebIO.render for result of output
    end # use full blown WebIO ecosystem to compose UI 
end

I guess this still requires some kind of an auto-layout of the input widgets.

Maybe we could allow an optional function layout() to be defined inside the macro where we have the variable names already assigned to the respective widgets. i.e. You could now just do hbox(vbox("Field", field), vbox("Param1", param1), vbox("Title", plot_title)). etc. In fact we can use the same convention of setting the identifiers before hand in render and output as well. Or the functions could take a single Recipe{:ridgelinear} object which maps fields to widgets/observables/values

I think the implementation of the macro would go something like this:

First create a Recipe{:ridgelinear} type which contains a dictionary of widgets with a corresponding dictionary of their observables they update. For this type which we expect layout, output and render to work (either specified by user in the macro or the defaults, it would be required for the user to define output though)

We use layout to show the input widgets (after having created the widgets automatically), output to process the input to produce the output, render to render the output.

I don't think I could get the full picture of your current design from the discussion above. Would you care to summarize the one design you are thinking of implementing?

Thanks!

[piever]

In this way the following would create a column of checkboxes next to a column of sliders.

So a Tuple is hbox?

Thinking more carefully, I think I'd prefer to have a Nx1 vector do hbox, meaning widget([col1 col2]) would be equal to hbox(widget(col1), widget(col2)). This has two advantages: dispatch on Tuple can be used for something else (like what I proposed above widget(t::Tuple) = widget(t...) which would prevent us from having to type widget too many times when designing a recipe. Plus I think in the horizontal array version we can probably do some tricks with broadcast, though I still have to think carefully about that.

To summarize the design, what I have in mind is the following:

• Combining widgets in a complex layout is done by using horizontal and vertical arrays and calling widget recursively, for example widget([1:100, 1:100]) would give us two sliders one on top of the other.
• A widget also needs to have some id field to identify it: ui = widget([(1:100, :id=> :s1), (1:100, :id => :s2)])
• A widget also needs to have a OrderedDict of children, mapping id symbol to widget.
• Once this is done, ui[:s1] would give us the first slider and ui[:s2] the second slider (in nested ui it would first look for :s1 in the children and then continue recursively)
• Concerning the output part, once we have ui be the composite input widget, we can create the output normally using observe(ui[:s1]) for example as a signal

In terms of macros / syntactic sugar, one thing that it makes sense to have is certainly some @map macro which would be more or less like the @map macro in JuliaDBMeta: it would replace symbols with the corresponding observable. Something like:

@map ui plot(1:10, sin(1:10+:s1)) would expand to:

map(t-> plot(1:10, sin(1:10+t)), observe(ui[:s1]))

so it would be quite easy to reproduce the @manipulate style plots but with more control. In particular, I think all functions from Observables should have a macro version that takes a UI (meaning a composite widget) and an expression, for example @on ui println("New value is $(:s2)").

[piever]

The main differences that I see between our two designs are:

• We choose a different syntax: widget(1:100, id = :x) versus x in 1:100. I think the syntaxes can coexist: the first one to be "macro free" and to be used as implementation, the second as syntactic sugar for common use cases

• you propose a Recipe type, whereas I think it's better to enrich Widget type so that it can store more widgets inside of it recursively. In practice it probably doesn't make much difference but I like the idea of having only one type Widget

To reconcile our views on layout, I think Widget can also have a field layout that defaults to vbox (or hbox if widgets are passed as a horizontal array) but that can also be customized.

[piever]

A final consideration: it seems like dispatch is clearly not enough to distinguish what widget the user want (is [1, 2, 3, 4, 5] togglebuttons, radiobuttons, a slider or 5 widgets organized vertically?) I think widget should take a first symbol argument that specifies which widget it is (for example (:slider, [1,2,3,4]) or (:togglebuttons, [1,2,3,4])).

[piever]

In practice, I think I'm going to try and port parts of Sputnik to this new recipe system on branches (one recipe branch here, one in Sputnik and a new UIRecipesBase package for the recipe infrastructure) and see what are the pain points in practice: I think Sputnik is sufficiently rich to be a good test for the new framework.

[shashi]

I'm more and more leaning towards verbosity when it comes to layout. I think layout with arrays has certain problems:

julia> [[1,2] [2]]
ERROR: DimensionMismatch("vectors must have same lengths")
Stacktrace:
 [1] hcat(::Array{Int64,1}, ::Array{Int64,1}) at ./array.jl:1153
 [2] macro expansion at /home/shashi/.julia/v0.6/Revise/src/Revise.jl:775 [inlined]
 [3] (::Revise.##17#18{Base.REPL.REPLBackend})() at ./event.jl:73

I thought this would work but it doesn't:

julia> Any[[1,2] [2]]
ERROR: ArgumentError: number of rows of each array must match (got (2, 1))
Stacktrace:
 [1] typed_hcat(::Type{Any}, ::Array{Int64,1}, ::Array{Int64,1}) at ./abstractarray.jl:1106
 [2] macro expansion at /home/shashi/.julia/v0.6/Revise/src/Revise.jl:775 [inlined]
 [3] (::Revise.##17#18{Base.REPL.REPLBackend})() at ./event.jl:73

I think hbox(vbox(1,2), 2) is much better in this case.

Also, if the goal is to have a non-macro version, I think we shouldn't try to add syntactic sugar like using tuples just to avoid typing widget. But I'm OK with whatever you choose.

I like the idea of unifying this in Widget, but feel that it's taking the role of Scope more and more. E.g.

s = Scope()
s.dom = vbox(widget1, widget2)
s["val1"] = observe(widget1)
s["val2"] = observe(widget2)
s

Concerning the output part, once we have ui be the composite input widget, we can create the output normally using observe(ui[:s1]) for example as a signal

Fair enough, but I think I have been misunderstanding what you mean by "recipe" here ... :-) Correct me if I'm wrong, but I think you're suggesting a recipe system which is intended to create "input widgets", the rest works by just using our existing WebIO features. (e.g. display of output). I'm more interested in creating self-contained UIs complete with output using this recipe system.

[shashi]

(just using this syntax because by now you know what I mean by it)

UIBase.@defmodule ridgelinear begin
    input::Table
    field::Symbol in colnames(input) 
    ...
end

Wanted to point out here that the second field actually depends on the value of the first field (table) at any given time. This would have to be done using an Observable of widgets which write to the same Observable.

[piever]

Fair enough, but I think I have been misunderstanding what you mean by "recipe" here ... :-) Correct me if I'm wrong, but I think you're suggesting a recipe system which is intended to create "input widgets", the rest works by just using our existing WebIO features. (e.g. display of output). I'm more interested in creating self-contained UIs complete with output using this recipe system.

I think our confusion stems from two very different use cases:

1. the average Julia dev has a package doing some modelling and wants to add a way to visualize their model with a couple of widgets to alter parameters without adding heavy dependency to their package

2. the web inclined Julia dev has found an interesting javascript library they want to add to the WebIO ecosystem: they are happy to take WebIO dependencies but want a high degree of customization

I've mainly been thinking about case 1 (which would be pretty much like the Plots ecosystem example) whereas you seem to be more focused on case 2.

However I think my design also addresses case 2, as every widget also has a symbol parameter you could dispatch on to customize rendering. Meaning, if your "output widget" in your GUI is of type Widget{:image} you can simply add a dispatch to WebIO.render for Widget{:image}.

[shashi]

I've been talking about 1) as well. 2) is figured out by WebIO.

What I meant by "output" and "input" is this:

I thought Widget{:image} would be a way to input an image: either via a file or camera while Image itself has a WebIO.render which we can obviously easily show.

The "output" part comes when you want to make a widget to say, select an image and apply gaussian blur to it. Then you need a generic way to apply a transformation to the input parameters.

[shashi]

BTW, my view of WebIO is packages like JuliaDB will depend on it to provide WebIO.render -- as if it's just a better Base.show. WebIO itself is not heavy weight.

In my opinion, a recipe should

1. Do automatic layout of widgets
2. Allow users to use the more powerful tools in the ecosystem to create layouts if the automatic ones don't suit them. If they don't want to depend on these packages, then all they can get is the default layout. That said, I'm ok with having the placeholders for the most common functions that get "filled up" when actually displaying the UI.

What I'm thinking of is a lot like @manipulate with a layout function and a way of composing.

[piever]

Closed by the Widgets package