deferred data loading into components / notebooks

[havok2063]

Is there currently any mechanism to defer data loading, either into an ipyvuetify or bqplot component, or on the voila-embed side of things? I need to be able to embed components from a notebook that don't necessarily have hardcoded data loading in the notebook. For example, pass a target source into a notebook, which looks up and access the data content, then creates the embeddable widgets. This is related to Use Case 1 in #5.

Here is my current example notebook, https://github.com/havok2063/voila-embed/blob/master/spec_viewers.ipynb. This example starts with hardcoding some data that's loaded, at the top of cell 3, which I'm trying to build off of. I tried replacing lines

jwst_sources = [227, 482, 546, 1186]
selected_src = jwst_sources[np.random.randint(0,3)]

with a new cell

src_field = v.TextField(_metadata={'mount_id': 'data'}, disabled=True, hide_details=True, label='target',  v_model='target', single_line=True)

@out.capture()
def update_source(widget, event, data):
        print('updating source', data)
        src_field.value = data

src_field.on_event('change', update_source)

@out.capture()
def show_data(data):
    print('data target ', data.value, data.label)

show_data(src_field)

selected_src = src_field.value

which should run first each time the notebook is run. Inside my Vue instance on the front-end, I have

        created() {
            // access the site url
            let href = window.location.href;
            let target_id = null;

           // extract the object identifier
            if (href.includes('?')) {
                // get the target source id
                [url, params] = href.split('?');
                target_id = parseInt(params.split('=')[1]);
            } else {
                // generate a random target if one not passed
                target_id = this.items[Math.floor(Math.random() * this.items.length)]
            }

           // update the widget text value
            this.target = target_id;
            requestWidget({
                voilaUrl: 'http://localhost:8000',
                notebook: 'spec_viewers.ipynb',
                mountId: 'data',
            }).then(sevent => {
                sevent.send({event: 'change', data: target_id})
            });

However when this runs with href, http://localhost:9000/protospec?target=1186 , I get the debug output

data target  None target
updating source 1186

and the notebook crashes because selected_src is None and none of the components can be generated.

It seems Voila runs the entire notebook before embedding any components into the front-end. It also does this before the Vue instance runs anything in its created lifecycle hook. This seems to mean the notebook will always fail to run unless data is loaded in a hardcoded manner.

Is it possible to generate a full set of components without data, but lazy load it once it is available, or is there something wrong with my example that could be fixed?

Alternatively, I also tried leaving selected_src = jwst_sources[np.random.randint(0,3)] intact so some data was loaded originally in the notebook but in the created hook, I instead run

        created() {
           // update the widget text value
            this.target = target_id;
            requestWidget({
                voilaUrl: 'http://localhost:8000',
                notebook: 'spec_viewers.ipynb',
                mountId: 'protospec',
            }).then(sevent => {
                sevent.send({event: 'change', data: target_id})
            });

which should run the update_data method attached to my main v.Row component, and reload the data with the updated object source. This successfully refreshes the data but not other chart elements, e.g. the image title.

[mariobuikhuizen]

It seems Voila runs the entire notebook before embedding any components into the front-end. It also does this before the Vue instance runs anything in its created lifecycle hook.

Yes, all cells are executed by Voila before the Vue instance is started. Any initialization dependent on parameters supplied by the front-end must be done on an event from the front-end. You could show a widget with some loading status and replace its children with the widget that is initialized in the front-end event.

In your example you could do this:

def update_data(selected_src):
    hdu1d, hdu, cut = prep_data(selected_src)

    # update spec1d data
    wave, flux = get_xy(hdu1d[1].data)
    plot.y = flux
    plot.x = wave

    spec2d, heat = create_spec2d_heatmap(hdu[1].data)
    fig, plot = create_spec1d(wave, flux)
    print('selected source', selected_src)

    img, image = create_cutout(cut[0].data, target=selected_src)

    print('new image title', img.title)
    conainer.children = [
        v.Row(_metadata={'mount_id': 'protospec'}, dense=True, row=True, wrap=True, align_center=True, children=[

            # load the histogram and slider content
            v.Col(xs12=True, lg6=True, xl4=True, children=[
                img
            ]),

            # load the line plot content
            v.Col(xs12=True, xl4=True, children=[
                spec2d
            ]),
            # load the line plot content
            v.Col(xs12=True, xl4=True, children=[
                fig
            ]),       
        ])
    ]

out.capture()
def update_data_handler(widget, event, data):
    # get new data from selected target
    selected_src = int(data)
    update_data(selected_event)

container = v.conainer(_metadata={'mount_id': 'protospec'}, children=['Loading..'])

# attach a change event so the Layout to update the data of all three components
container.on_event('change', update_data_handler)

[maartenbreddels]

Parametrized notebooks is something we have been discussing at voila, but didn't reach consensus yet: https://github.com/voila-dashboards/voila/pull/414 https://github.com/voila-dashboards/voila/pull/218#issuecomment-553653123

The workaround by mario will have to do til them.

[havok2063]

Interesting! Thanks Mario and Maarten for the help and links. ASB will almost certainly need to utilize parametrized notebooks and/or deferred data loading in every case. Reading through those comments, seems like prelaunch notebook hooks could be the answer. We'll primarily be building notebooks generic to multiple kinds of data, with out first cell or prelaunch cell being some kind of data access call. If these hooks make it easier for us to build and maintain these notebooks, we might want to push to get this implemented. I don't think we need this for our prototype deliverable in March but we may need this feature for production. Let's keep this on the radar!

[timkpaine]

@havok2063 we use the prelaunch hook extensively for auth and logging, and we inject some variables about the accessing user into the kernel (e.g. ip, other info). We also use the papermill parameterization (which works like 90% the same way as the prelaunch hook) to generate urls with preconfigured views

[havok2063]

@timkpaine this is very interesting and something I'll be looking into for sure. Do either of these solutions allow for extracting parameters that were sent via a POST request as opposed to GET query parameters in the url?