Support for references inside reactive values

[PGimenez]

I'd like to be able to use a reactive variable in another's definition, and have them linked. Something like this, where the handler for var2is triggered when var1changes.

using GenieFramework
@genietools
@app begin
    @in var1 = 0
    @in var2 = [1,var1,3]

    @onchange var1 begin
        @show var1
    end
    @onchange var2 begin
        @show var2
    end
end

ui() = slider(1:1:10, :var1)
@page("/", ui)

Currently, one way to implement this would be

using GenieFramework, Observables
@genietools
@app begin
    @in var1 = 0
    @in var2 = [1,2,3]
    @onchange isready begin
        var2[2] = var1
    end
    @onchange var1 begin
        @show var1
        model = @init
        notify(model.var2)
    end
    @onchange var2 begin
        @show var2
    end
end

ui() = slider(1:1:10, :var1)
@page("/", ui)

This would entail modifying the way @app instantiates the variables. It could also add a performance penalty, as we'd have to do a search for dependencies between variables.

Alternatively, we could have a @notifymacro that gets the model and calls notify

[essenciary]

What doesn't work with the first example? var1 is not updated?

[PGimenez]

The first example does not work as you get "ERROR: LoadError: UndefVarError: var1 not defined" upon execution

[essenciary]

This requires some deep digging, but I think it's more complicated than this. First question is, how is var2 serialized to be sent to the client? And what does the data payload coming from the client look like? If it's all values, we need to change the serialization to actually use the variable in JS - and send back a symbol for that variable in Julia. Then we need to extend the the serialization/deserialization logic to support variables on both server and client. A potential issue being that at the point where var2 is deserialized on the server, var1 might not be in scope (can't remember).

---

If I'm making myself clear: var2 is serialized as just data. When it comes back from JS as data/values, how do I know that var2[2] was actually a reference to a var (what var, that data is lost -- and is this var in scope)? That information is lost.

[essenciary]

Using variables/references in serialized data payloads goes against the core principles of Stipple: only data is sent over the wire. Julia objects are converted to their JS counterpart using their type's render method and sent over the wire. And on the Julia side, the objects are reinstantiated using the data payload.

[essenciary]

The first example does not work as you get "ERROR: LoadError: UndefVarError: var1 not defined" upon execution

You would need to expand the macro to see what is generated. All the reactive vars are actually kept in the reactive model object that looks like __model__.var1 = ... and the macros add syntactic sugar to avoid having to use the reactive model explicitly. However, this involves parsing Julia expressions. So in this case it's probably because var1 inside the array is not parsed and is not converted to __model__.var1 (effectively the macros rewrite all vars to the corresponding reactive model fields).

However, even if that works, see above: serializing/deserializing references would not work: var2 would be sent as the array of the corresponding values.

[essenciary]

It gets complicated fast. See this:

using GenieFramework
@genietools

mutable struct W{T}
       value::T
end

Stipple.render(w::W) = w.value
Stipple.stipple_parse(::Type{W}, i::Int) = W(i)

@app begin
                @in var1 = W(2)
                @in var2 = [1,W(2),3]

                @onchange var1 begin
                    @show var1, var2
                end
            end

ui() = slider(1:1:10, :var1) * textfield("", :var1) * textfield("", :var2) 
@page("/", ui)

Results: a) @var1 reactive handler is no longer triggered -- this is weird, I expected it would work. b) we don't do deep serialization of structures (ie we serialize var2 as a unit, vs serializing each individual element), which results in W(2) not being correctly serialized in var2 (though it is correctly serialized in var1).

I remember talking to @hhaensel about deep serialization of collections, I think we decided it would be a performance issue. However, now I think we could do it by introducing a supertype of Array, ex Serializable <: AbstractVector and do Serializable([1, W(2), 3]) so users can opt-in to have deep serialization.

---

[PGimenez]

I'm still reading through your replies but just to answer why I need this. This came up when in the exploratory data analysis app I had a single data source for multiple plots. After changing the data, I had to manually update each plot. It'd be nice if I could just change the data and all dependent variables would be updated. This is what Pluto does if I'm not mistaken.

Example, having something like this work:

@app begin
    @in N = 100
    @out data = randn(100)
    @out hist = [histogram(x=data)]
    @out sct = [scatter(x=data, y=data)]
    @onchange N begin
        data = randn(N)
    end
end

ui() = [slider(1:1:10, :var1), plotly(:hist), plotly(sct)]

Now I have to update the plots inside the handler

using GenieFramework, PlotlyBase
@genietools

@app begin
    @in N = 100
    @out data = []
    @out hist = []
    @out sct = []
    @onchange isready, N begin
        data = randn(N)
        hist = [histogram(x=data)]
        sct = [scatter(x=data, y=data)]
    end
end

ui() = [slider(1:1:10, :N), plot(:hist), plot(:sct)]
@page("/", ui)

Imagine having multiple filters for the data this would get repetitive. Although in this case you could define a handler for updating the plots and trigger it from other handlers, like

@onchange N begin
    data = randn(N)
    notify(update)
end
@onchange update begin
    #update plots
    end

[essenciary]

Yes, sorry, I kind of digressed while digging into it. I do understand the use case and that could work if the container with the reference is not a reactive var itself or if it's just @out. If it is a reactive var whose value is overwritten from the client with serialized data, then things get very complicated, cause serialization is pure data, and does not include references.

In addition, the references would have to be objects to be referenced, as value types are passed by value (copied). It could be as simple as a Ref type of object. Which means that first we need to figure out why in my examples, the objects W are not reactive.

And also we need to solve the initial issue you reported, with the var not being found.

[essenciary]

Here is a small example that shows that references would work if we solve the various issues:

ref = Ref{Int}(0)

@app begin
       @in var1 = 1
       var2 = [ref, 1,2]

      @onchange var1 begin
           ref[] = var1
           @show var2
       end
end

ui() = slider(1:1:10, :var1) * textfield("", :var1)

Output:

var2 = Any[Base.RefValue{Int64}(6), 1, 2]
var2 = Any[Base.RefValue{Int64}(3), 1, 2]
var2 = Any[Base.RefValue{Int64}(2), 1, 2]
var2 = Any[Base.RefValue{Int64}(7), 1, 2]
julia>

When we update the slider, the ref gets updated as expected.

[hhaensel]

I'd probably rather re-look at the task you are going to achieve. If you want to share the memory of data for different plots, why not setting up js-expressions that share the memory on the client side?

@app begin
    @in N = 100
    @out data = []
    @out hist = []
    @out sct = []
    @onchange isready, N begin
        data = randn(N)
        # hist = [histogram(x=data)]
        # sct = [scatter(x=1:N, y=data)]
    end
end

# fix an issue in rendering JSONText in plots
# (we should be fixing this in a future release of StipplePlotly ...)
import StipplePlotly.Charts.jsonrender
jsonrender(plot::GenericTrace) = jsonrender(Dict(plot))
jsonrender(plots::Vector{<:GenericTrace}) = jsonrender(Dict.(plots))

# render the trace expressions based on data, `js"data"` is equivalent to `JSONText("data")`
hist = [histogram(x=js"data")] |> jsonrender
sct = [scatter(x=js"Array.from({length: data.length}, (x, i) => i)", y=js"data")] |> jsonrender

ui() = [slider(1:100, :N), plot(hist), plot(sct)]
@page("/", ui)
up()

[hhaensel]

jsonrender is part of StipplePlotly and was built to set up configuration expressions. But we had several cases already, where we would have needed it elsewhere as well. Maybe we move it to Stipple and import it in StipplePlotly ... Meanwhile the above code should work properly.

Instead of writing the somehow ugly js array generator, I could as well have included a field xdata that is set as 1:N, then the sct definition would have become a bit cleaner:

@app begin
    @in N = 100
    @out ydata = []
    @out xdata = []
    @onchange isready, N begin
        xdata = collect(1:N)
        data = randn(N)
    end
end

hist = [histogram(x=js"ydata")] |> jsonrender
sct = [scatter(x=js"xdata", y=js"ydata")] |> jsonrender

[hhaensel]

Just patched StipplePlotly so that the above snippet runs without patching jsonrender.

[PGimenez]

Thanks for the workaround @hhaensel, now any time the data variables change the plots are updated. Still, I'm not sure how this would work for something that is not a plot.

Let's say I'd like to have this code work:

using GenieFramework
@genietools
@app begin
    @in var1 = 1
    @in var2 = [var1,2,3]
    @onchange var1 begin
        @show var1, var2
    end
end

ui() = slider(1:1:10, :var1) * "{{var2}}"
@page("/", ui)
up()

It doesn't work as var2 references var1. To make it work, I'd have to manually substitute var2 with its definition in the uifunction:

using GenieFramework
@genietools
@app begin
    @in var1 = 1
    @onchange var1 begin
        @show var1, var2
    end
end

ui() = slider(1:1:10, :var1) * "{{[var1, 2, 3]}}"
@page("/", ui)
up()

which is not ideal. Also, to avoid confusion, I'd rather have people define as few vars outside @app as possible. We've been telling them that variables appearing in the UI Ulike hist) should be reactive.

I started this discussion to ask whether this was doable, but it seems like a difficult task also from what Adrian said.

[hhaensel]

I digged a bit and I found out, that it is actually doable. The only thing you have to take care of is putting square brackets after the variable name:

@app begin
    @in var1 = 11
    @in var2 = [var1[], 2, 3]
    @onchange var1 begin
        @show var1, var2
    end
end

model = @init

model.var1[], model.var2[]
# (11, [11, 2, 3])

This also has some (negative?) consequences if you want to use a variable for initialisation with the same name as the model. Have a look at the following example:

var1 = 11
@app begin
    @in var1 = 2 * var1
    @in var2 = [var1, 2, 3]
    @onchange var1 begin
        @show var1, var2
    end
end

Here you overwrite var1 with the Observable of var1, and any other occurence of var1 after the line @in var1 = 2 * var1 will always reference the model's Observable. Nevertheless, var2[] expects a Vector{Int} upon initialisation and will fail to convert Reactive{Int64} to Int64.

[hhaensel]

If you use a different name for model variable and init value, then you are also safe:

var1_init = 11
@app begin
    @in var1 = 2 * var1_init
    @in var2 = [var1_init, 2, 3]
    @onchange var1 begin
        @show var1, var2
    end
end

[hhaensel]

Do we really need a more sophisticated API? Most probably, reasonable docs would solve the issue...

One last thing, if you don't want to waste your name space, you can define your init_value within the macro:

@app begin
    var1_init = 12
    @in var1 = 2 * var1_init
    @in var2 = [var1_init, 2, 3]
    @onchange var1 begin
        @show var1, var2
    end
end

model = @init

model.var1[], model.var2[]
# (24, [12, 2, 3])

[hhaensel]

I need to add on this, that the above solution is not a permanent connection between the variables. But again, if you would like to bind the array [var1, 2, 3] to a field, you can

julia> using Stipple, StippleUI

julia> select(:selector, options = R"[var1, 2, 3]")
"<q-select v-model=\"selector\" :options=\"[var1, 2, 3]\"></q-select>"

because bindings are always evaluated as js-expressions in the context of the model.

[hhaensel]

One more more thing. I am not sure what your main goal in this discussion is.

• faster reaction time
• less data transmission
• data storage

In the past I had developed a mechanism to update elements of vectors. This died in a very unfortunate way, but we could revive the topic if there is a need. That would be a way of lowering the data transmission footprint if you are handling large amounts of data with a high single-element update-rate, e.g. online data acquisition.