Data stream loading

[dyuliu]

When there are dozens/hundreds of signals contained in one experiment, it's impossible to present the data until finishing loading all of them.

Let's consider using stream loading method to loading and plotting the data simultanously.

[sergiu-ojoc]

Data Stream Loading: https://github.com/HDI-Project/MTV/issues/113

General: Client cannot display any data until API call ends. Server should provide data in chunks. Data can be cached afterwards to prevent repetitive API calls. I suppose we need a better server side grouping for each datarun (prediction and raw).

1. The simplest implementation of this task will be to use pagination on the server side and the make use load on scroll for displaying data in client. In the overview wrapper we will display 5-6 dataruns in the beginning and when the users scroll down, we’ll load more. As an additional improvement here will be to cache the API results and prevent repetitive API calls the user select a previously selected experiment.

Unfortunately, this feature is not implemented in backend yet. Sever exposes one single endpoint with api.add_resource(ctrl.datarun.Dataruns, current_api_version + 'dataruns/')

and within the Dataruns classname, datarun gets aggregated with all the required props which are very very larg: prediction – data field (11487 in one field record) and raw – also data field.

# get prediction
    prediction_doc = model.Prediction.find_one(datarun=datarun_doc.id)
    datarun['prediction'] = {
        'names': prediction_doc.names,
        'data': prediction_doc.data
    }

    # get raw
    raw_docs = model.Raw.find(datarun=datarun_doc.id).order_by('+year')
    for raw_doc in raw_docs:
        datarun['raw'].append({
            'timestamp': raw_doc.timestamp,
            'year': raw_doc.year,
            'data': raw_doc.data
        })

2. Load all ‘dataruns’ and make additional API calls for loading ‘predictions’ and ‘raw’ data for the first 5-6 dataruns. When user scrolls loaders will be displayed for each datarun row showing that graphic data is loading.

3. Web-sockets. For this one I need to document myself how to do it and how to implement it in React, but the main idea here (like I get it) – client subscribes to a service from the server and server updates the client about the changes.