BlueNaas: use it as a service via websocket

[lbologna]

Aspect	Detail
Summary	BlueNaas will be used as a service, through remote requests via websocket
Expert	@genric
Deadline
Dependencies	python, neuron

List of features

Currently the BlueNaas is used via web application for running the simulation and accept models from the database or uploaded by the users (provided they fulfill a precise format). With this new feature, the developers will not need to access the web application but will be able to use a websocket for a bidirectional communication between a remote server and the BlueNaas. The BlueNaas will exchange messages with the remote server (whitelisted) for fetching the model from a repository (whitelisted), starting and stopping the simulation and for parameter setting.

Tasks

• [x] Implementation of proof of concept (@genric)
• [x] Proof of concept test (@lbologna, @ElisabettaGiacalone )
• [x] Definition of a format for the models/tools (python) (@appukuttan-shailesh )
• [x] Implementation of the "aas" feature on the BlueNaas (@genric)
• [x] Integration of the new feature in Live Paper (@lbologna, @ElisabettaGiacalone)

Acceptance Criteria

The developers can establish a communication with the BlueNaas and control the simulation (start and stop) and simulation parameters via websocket

Performance

The performances will be comparable with the actual ones for the BlueNaas

[lbologna]

Hello @genric,

I inserted the script.py you sent me in a django view of a project that I am running at the following url: bspg.pa.ibf.cnr.it

but I get the following error:

WebSocketBadStatusException: Handshake status 403 Forbidden

I attach herewith the function in a .txt file. Any idea of why it is not working?

Thanks

view.txt

[lbologna]

@genric: we discussed with @ElisabettaGiacalone and @mmigliore about the possible scenarios the BlueNaas may encounter for running user's models through the websocket. We believe it is better to discuss this via skype call so that we can have a handle on it with real time examples and discussion.

If this is fine with you, could you please fill the following doodle for a call later this week: https://doodle.com/poll/sg48bcqfxps9gbfh ?

Thank you.

[lbologna]

Genrich thanks for filling the doodle.

@genric, @ElisabettaGiacalone let's fix the call for Friday 19 at 11.00 a.m. then.

Genrich please let us know how you would like to connect. Thanks

[appukuttan-shailesh]

@lbologna : I am also interested in joining in on this call to get a better idea of this feature. Will join you on Friday at 11 AM.

[appukuttan-shailesh]

As discussed on the call today, I was proposing we could mediate the interaction between the models and BlueNaaS via the "sciunit" framework. https://github.com/scidash/sciunit (Related: https://github.com/scidash/neuronunit)

The framework is primarily directed towards validations, but at the core it uses the concept of "capabilities". These are basically "model agnostic" interfaces which allow interaction with models. As an example, these are used in 'HippoUnit' for loading models developed by BluePyOpt by implementing the required interface.

[appukuttan-shailesh]

I have made some progress on this. We had two sample models: 1) neuronservice_eg_llb_dev 2) neuronservice_ss_dev

I have been working on the first model (neuronservice_eg_llb_dev) as it was much smaller, but also didn't employ the concept of 'templates', which we were accustomed to handling for BluePyOpt etc. I will next take up the other model.

The reworked version (v2) of the first model along with all the assisting files can be found here: https://github.com/appukuttan-shailesh/testData/raw/master/neuronservice_eg_llb_dev_v2.tar.gz

Following is an overview of the changes:

• themodel_eg.py and the mod files have not been altered at all; this ensures that the source code of the models need not be altered for this method to work
• neuronservice_eg_port7890.py has been heavily updated and saved as neuronservice_eg_port7890_changed.py This is the file that needs to be executed to open the web socket. But the file itself contains very minimal code, basically providing 5 input parameters. The file creates a Python class to instantiate the model, and also inherits from a sciunit Capability named _BlueNaaS_PythonModel.
• bluenaas_capabilities.py defines the sciunit capability _BlueNaaS_PythonModel and provides all the 'helper' and 'common' functions required for instantiating, modifying and running a model, and also those required for the creation and management of the web socket. It checks if the 5 input parameters have been provided in neuronservice_eg_port7890_changed.py. If not it raises a NotImplementedError() for these parameters. If all are specified, it goes about doing all the housekeeping activities.
• neuronservice.js has been considerably simplified by removal of many unused lines of code, and also by simplifying the workflow to encourage reuse of existing code for different simulation calls.
• index.html has been slightly modified to align with the changes in neuronservice.js

The five input parameters to be provided (along with sample values for model 1) and specified in neuronservice_eg_port7890_changed.py are:

# 1. provide main model python file path
model_path = "./themodel_eg.py"

# 2. path to mod files; can be 'None' if mod files in base directory of `model_path`
mod_files_path = None

# 3. Set default values for model parameters (used to reset model to initial state)
# Note: should use same name as target parameter name; so e.g. not gkm for gbar_kmb
default_parameters = {'soma' : {'gbar_kmb':  0.005, 'gcanbar_can': 0.6},
                      'stim[0]' : {'amp' : 0.03}}

# 4. Parameters to be set for NEURON simulator
config = {'tstop' : 600, 'v_init' : -80}

# 5. Specify port number for socket
port_number = 7890

The websocket now basically takes only two messages:

• setfields This is used to update parameter values of the model
• run_simulation This is used for running the simulation with specified parameters, retrieving the data, and displaying the same on the graphs

The file attached above is the first working prototype. We might need to alter and update this based on other requirements, and probably also to align with constraints that might come up when working on the second model.

In summary, bluenaas_capabilities.py will contain all the code necessary for the interaction with the websocket, and this file shall remain the same for all models (we could make this pip installable and importable). A supporting python file with very minimal code (e.g. neuronservice_eg_port7890_changed.py) will need to be created for every model in order to fulfill the capability requirements.

[genric]

Thanks @appukuttan-shailesh! I went a bit further and simplified it even more. I'll mail you the version which I can update the backend for. Let me know if you want to try it or we need to amend the design. Thanks!

[appukuttan-shailesh]

Apart from the two models that Luca and Elisabetta provided, I created two dummy models for further testing and generalization of the format - one using templates and the other without. So four models have now been tested and implement the same interface, but with the capacity to handle different requirements. These examples can be found here: https://github.com/appukuttan-shailesh/testData/raw/master/Testing_Models.tar.gz

Each model directory contains a python file named neuronservice_*.py which is to be executed for opening the websocket locally. This is the file that needs to be created for each models wanting to use this service. It basically accepts 5 parameters (slightly changed from previous post on GitHub) which is very easy to specify. It contains several comments that should help the user identify how to input the values. Example:

import sciunit
import bluenaas_capabilities as cap

# 1. provide main model python file path
model_path = "./themodel_without_template.py"

# 2. path to mod files; can be 'None' if mod files in base directory of `model_path`
mod_files_path = None

# 3. Set default values for parameters (used to reset model to initial state)
# Note: should use same name as target parameter name (e.g. nBPAP and NOT nbpap) with outer key specifying section/object
# Note: parameters to be set for NEURON simulator also included here as key:value pair
# Note: function calls for handling parameter initialization can be specified as a list using 'FUNCTIONS' key
# Note: function calls would be executed after all specified parameter values have been assigned
default_parameters = {'soma' : {'gnabar_hh':  0.12, 'gkbar_hh': 0.036},
                      'stim' : {'delay': 5, 'dur': 1, 'amp': 0.1},
                      'tstop' : 25.0}

# 4. Specify the vector objects that are being recorded during simulation
# Note: format is {vector_object_name : display_label}
# Note: time vector should use value of 'TIME' and is mandatory input
recorded_vectors = [{'t_vec' : 'TIME'},
                    {'v_vec' : 'soma(0.5)'},
                    {'dend_v_vec' : 'dend(0.5)'}]

# 5. Specify port number for socket
port_number = 7892

class model_loader(sciunit.Model,
                   cap.BlueNaaS_Python_Model):
    def get_model_path(self):
        return model_path, mod_files_path

    def get_default_parameters(self):
        return default_parameters

    def get_recorded_vectors(self):
        return recorded_vectors

    def get_socket_port(self):
        return port_number

model = model_loader()
model.initialize()
model.setup_socket()

Each model directory contains an identical copy of the file named bluenaas_capabilities.py. We could make this a pip installable package, so that users don't require to copy-paste the file, and could simply import this into their model files. Let me know if you think this would be useful, and I can set it up. In that case, the above file could simple be reduced to something such as:

from blue_naas_capability import model_loader

# 1. provide main model python file path
model_path = "./themodel_without_template.py"

# 2. path to mod files; can be 'None' if mod files in base directory of `model_path`
mod_files_path = None

# 3. Set default values for parameters (used to reset model to initial state)
default_parameters = {'soma' : {'gnabar_hh':  0.12, 'gkbar_hh': 0.036},
                      'stim' : {'delay': 5, 'dur': 1, 'amp': 0.1},
                      'tstop' : 25.0}

# 4. Specify the vector objects that are being recorded during simulation
recorded_vectors = [{'t_vec' : 'TIME'},
                    {'v_vec' : 'soma(0.5)'},
                    {'dend_v_vec' : 'dend(0.5)'}]

# 5. Specify port number for socket
port_number = 7892

model = model_loader(model_path=model_path, mod_files_path=mod_files_path, default_parameters=default_parameters, recorded_vectors=recorded_vectors, port_number=port_number)
model.initialize()
model.setup_socket()

If required, a more "user-friendly" web app could be created to auto-generate the above file based on entries inside a web form.

If the model requires to invoke method(s) for setting model parameters (e.g. example model neuronservice_ss_dev), this can be specified with any required arguments using the FUNCTIONS key. e.g.:

default_parameters = {'panel' : {'nBPAP':  1, 'nstim': 70, 'FUNCTIONS':['set_pulse()']},
                      'tstop' : 250}

The methods would be invoked after all the specified parameters have been assigned.

For the recorded vectors, it is mandatory to specify the vector corresponding to the simulation time series. This is to be indicated via a dictionary with key as the name of the recorded time vector and value as TIME. e.g.

recorded_vectors = [{'t' : 'TIME'},
                    {'v' : 'soma(0.5)'}]

Note on the front-end: Some browsers apparently have problems handling . (dots) in the ID tag. So instead: the HTML page should name the input fields with IDs corresponding to their model parameter name, with __ (double underscore) used to indicate . (dot) operator. (We can change this to have the dot operator, if no issues are found across browsers). So,

Python : soma.gnabar_hh = HTML : soma__gnabar_hh Python : cell.soma.gnabar_hh = HTML : cell__soma__gnabar_hh Python : stim.amp = HTML : stim__amp

But for NEURON parameters, you can directly name the parameters, e.g. Python : h.tstop = HTML : tstop

Known limitations (add more as and when identified): 1) Currently presumes all plots will have time (simulation timestamps) on the x-axis. Fine for now to plot voltage and Calcium temporal profiles.

[appukuttan-shailesh]

@genric : In terms of communication via the websockets, as per my current implementation, the following commands are being sent:

Model to Frond-End 1) setfields 2) setgraph

Frond-End to Model 1) setfields 2) run_simulation

setfields corresponds to set_params in the e.g. that you emailed.

[genric]

I'm a bit confused how BlueNaaS fits into the picture with these type of models. Because they are not models in my understanding but the full blown web applications. I thought the idea was to have pure, model related code and use it in as-a-service infrastructure. So common things such as compiling mod files or handling serialization/deserialization of websocket messages can be provided by as-a-service infrastructure. And then it's up to the web developer build the UI to interact with that model. @appukuttan-shailesh: I think we can agree on the conventions for the messages, no problem with that. There just needs to be a minimal set of constrains on how to structure your model. Otherwise we cannot generalize and using BlueNaaS doesn't make sense. I think then we would probably be better off just running those model apps in a VM as you tried before.

[appukuttan-shailesh]

I think @lbologna would be better placed to answer this.

Currently, as you have pointed out, they have been tailored to suit the model apps that Luca had provided along with the models. The examples provided above were only to illustrate how to streamline different modeling approaches to offer a standard interface (as implemented in bluenaas_capabilities.py) whereby BlueNaaS or other services (e.g. the validation framework) may interact with the model without bothering about the model's internal implementation.

[lbologna]

Hello, @appukuttan-shailesh we tested all the models and they work fine except for some small "tweaks" to be done on the labels/edits dimensions on the html pages (we can send you and email with some screenshots of how it appears on our screen if you want) @genric after Shailesh's improvement about the generalization of the interface, we basically have what we had in mind at the beginning: 1) a model in python which is developed, basically, without any constraints 2) a common interface 3) a launch script in which the user must only set the parameters to be changed and data to be returned

In this scenario, provided all python packages are installed (e.g. sciunit) BlueNaas enters "as a service" in that it will: 1) download the model, the .mod and the launch script (including the bluenaas_capabilities.py) from some allowed remote repository 2) compile the .mod files 3) open the websocket The front-end will be run somewhere else (e.g. github) and its implementation will be transparent for BlueNaas, in that the latter will only communicate through websocket (with the frontend) on a specific port established at handshake time. This will provide a single entrypoint (BlueNaas + port) and exploits the BlueNaas implementation you mentioned earlier, which allows dynamic instantiation of multiple containers. I think this will prevent some redundancy that will probably will be faced in the instantiation of one vm per "full stack model". What's your ideas on this?

We believe that the operation that are done in the launch script that you sent us a couple of days ago are also done in the bluenaas_capabilities.py in Shailesh's code (except maybe for the explicit setting of the domain, currently set to localhost. @appukuttan-shailesh could you confirm this?). So probably the underlying principles are the same.

Are we missing some steps in the way we imagined this workflow?

Thank you

[genric]

In my opinion if we still want to arrive at some sort of as-a-Service infrastructure we need to clearly decouple what is a model and what should be provided by the as-a-Service infrastructure. Currently everything is just bundled together. @lbologna if we go through the list of actions(which you suggested to be performed by the BlueNaaS) the only one which is not performed by the model app is the model web app download. Mod file compilation is performed by the code from the app, websocket is opened and handled by the code from the app. So having BlueNaaS in order just to download the model web app is a bit of an overkill IMHO. As a potential solution on how to structure the model which could fit into the as-a-Service infrastructure please have a look at the zip file I've sent. It requires for the mod files to be present in a well known location and to have entry point python file which would define the model together with simulation trigger function. As there are still open questions I suggest we have a quick call to agree on the solution faster.

[appukuttan-shailesh]

@genric , @lbologna : I am around till tomorrow afternoon for a call, before I leave for vacations. I should be reachable via email afterwards, though I might not be able to work actively on this.

[lbologna]

@genric, @appukuttan-shailesh, @ElisabettaGiacalone is it fine to have a skype call this morning (27 August) at 12?

[appukuttan-shailesh]

Sure, it works for me.

[ElisabettaGiacalone]

Ok, it's fine for me.

[appukuttan-shailesh]

Based on our call earlier today...

I have created a public CSCS container named bluenaas_models. This can be accessed at: https://object.cscs.ch/v1/AUTH_c0a333ecf7c045809321ce9d9ecdfdea/bluenaas_models

It currently has two models: 1) ballandstick_with_template.zip https://object.cscs.ch/v1/AUTH_c0a333ecf7c045809321ce9d9ecdfdea/bluenaas_models/ballandstick_with_template.zip 2) ballandstick_without_template.zip https://object.cscs.ch/v1/AUTH_c0a333ecf7c045809321ce9d9ecdfdea/bluenaas_models/ballandstick_without_template.zip

Each of the models have a file named neuron_service.py which basically specifies the 4 required parameters (as discussed in post above; parameter for web socket has been removed).

The generic file bluenaas_capabilities.py has been made pip installable via: pip install bluenaas_capabilities

This can be utilized in a Python script, e.g.:

import sciunit
import bluenaas_capabilities as cap

class model_loader(sciunit.Model,
                   cap.BlueNaaS_Python_Model):
    def get_model_path(self):
        return model_path, mod_files_path

    def get_default_parameters(self):
        return default_parameters

    def get_recorded_vectors(self):
        return recorded_vectors

model = model_loader()
model.initialize()

A run_simulation() method has now been added to the model loader, and this will return the necessary info to be provided to the front-end.

Note: I need to convert the output of run_simulation() to return a dict instead of a list. I shall do this tomorrow morning, as I will be away for a bit now. In the meantime could you test and see what other things require changes.

[genric]

Great! Thanks @appukuttan-shailesh!

[appukuttan-shailesh]

Made some updates this morning.

Changes to bluenaas_capabilities and released on PyPI as version = 0.0.3: 1) Compilation and loading of mod files has been made optional by use of parameter: model.initialize(compile=False) Default is True, i.e. it will attempt to compile and load file. If compiled, initialize() return the path of the loaded libnrnmech.so.0 file, or else return None. 2) get_data() has been updated to return a "list of dicts", with the keys corresponding to the plot labels. I haven't used just a "dict" as the labels might be duplicated, e.g. "soma(0.5)" being used as labels in an example for both the voltage and calcium concentration at the soma.

Other changes and comments:

• recorded_vectors parameter in neuronservice.py has been reformatted to use the display labels as keys. This is more intuitive. I have updated this for all the models and their neuronservice.py files.
• I have tested the neuronservice_eg_llb_dev model with changed mod file location. I moved all the mod files to a directory named mechanisms and then updated its neuronservice.py to have: mod_files_path = "./mechanisms" and it works! So the location of mod files can be decided by the user, and specified as above.

The bluenaas_models CSCS container now has 5 models:

1. ballandstick_with_template.zip (with updated recorded_vectors) https://object.cscs.ch/v1/AUTH_c0a333ecf7c045809321ce9d9ecdfdea/bluenaas_models/ballandstick_with_template.zip
2. ballandstick_without_template.zip (with updated recorded_vectors) https://object.cscs.ch/v1/AUTH_c0a333ecf7c045809321ce9d9ecdfdea/bluenaas_models/ballandstick_without_template.zip
3. neuronservice_eg_llb_dev https://object.cscs.ch/v1/AUTH_c0a333ecf7c045809321ce9d9ecdfdea/bluenaas_models/neuronservice_eg_llb_dev.zip
4. neuronservice_eg_llb_dev_changed_mod_path (with mod files moved) https://object.cscs.ch/v1/AUTH_c0a333ecf7c045809321ce9d9ecdfdea/bluenaas_models/neuronservice_eg_llb_dev_changed_mod_path.zip
5. neuronservice_ss_dev https://object.cscs.ch/v1/AUTH_c0a333ecf7c045809321ce9d9ecdfdea/bluenaas_models/neuronservice_ss_dev.zip

[lbologna]

Hello,

we tested the service with the python scripts that @genric sent us a couple of weeks ago and the communication works. Thank you. We are now implementing it on the frontend html/javascript page of one the live papers that will integrate the service. At the moment we can establish the connection (websocket) and we are working on getting the reply.

[lbologna]

Hello, we tested the websocket communication from a frontend/html page and everything works!

@genric we noticed that the "default_parameters" and "recorded_vectors" are not present in the current version of the model (i.e. neuronservice_eg_llb_dev_changed_mod_path.zip), neither in the "the_model...py" nor in the "neuronservice...py" files. Shouldn't the config file where they are set be placed in the same .zip of the model, so that the developers can provide this information together with the model itself? Also, it seems the BlueNaas server is down or intermittently working at the moment (both the websocket and GUI). Is it in maintenance? Thank you.

[alex4200]

@lbologna The BlueNaas server should be up now, please try again.

[lbologna]

Yes it works now, both the GUI and the websocket. Thanks!

[genric]

Hi @lbologna I sort of assumed that params will always be explicitly set with the first set_params message, so there was no point to initialize them before that message arrives. But if you want to avoid sending that message then yes we can make some init method in the model to initialize the params. Or you can just initialize the model as the model python script is executed. Regarding the recorded vector: it is part of the run_simulation message data payload. Which is a dictionary of {label -> var_name} var_name is the name of the recorded variable present in the model which will be converted as_numpy() for the response.

[lbologna]

Hi @genric, the way it is implemented now also works for us. Thank you.

I thought you had a config file on the backend that the parameters sent from the frontend should match. But if they are set with the first message that is fine.

By the way, the service (i.e. bluenaas websocket) is not working anymore since about 30 minutes. Could you please let me know when it is up again? Thank you.

[genric]

@lbologna the service is a bit unstable... I'm trying to figure out why and fix it...

[genric]

@lbologna I've made some updates to the service, it should work better now.

[lbologna]

@genric the problem seems to be still there for me. I get the following errors:

Firefox can’t establish a connection to the server at wss://blue-naas-svc.humanbrainproject.eu/ws.

The connection to wss://blue-naas-svc.humanbrainproject.eu/ws was interrupted while the page was loading.

[genric]

That is weird... could you please describe exactly what you are doing? Maybe you have your webapp code shared somewhere? Can I access the actual web app to debug?

[lbologna]

Hello @genric,

I tested again after emptying cache and some cleaning and know it works for me as well and it seems to be stable. Thank you!

I attach the code that I am using on the frontend of the Live Paper herewith. I put it in a .html test page that should work out of the box (just change the extension to .html, open in browser and click on "Run WebSocket").

testWS.txt

[genric]

@lbologna looks good to me. The only comment I have is that you can get potentially http~500 error from the server then you need to fallback for some seconds and retry, eventually you should get a connection. The same way that you see in BlueNaaS in the upper-right corner spinning connection status indicator.

[lbologna]

Hello,

the code for the communication through websocket has been integrated in a work-in-progress live paper (the text is still to be inserted, but the simulation panel is completed in its functionality - minor changes to the layout will be done later).

Here's the link: https://cnr-ibf-pa.github.io/hbp-bsp-live-papers-dev/2019/martinello_et_al_2019/martinello_et_al_2019.html

@alex4200 could you please have a look? After opening the page, you will have to click on the "ModelDB link and test simulations" section and play around with the buttons. Thank you.

@genric as briefly mentioned, the connection seems much more stable now and, if an error occurs, the user is asked to retry after a few seconds. Thank you.

[alex4200]

@lbologna I had a look at the app, and it seems to work so far, and I like how you can do several simulations to be shown on top of each other (for direct comparison).

However, some questions/suggestions:

• When using very high values for conductances (like 100000000000000000000000000) or negative values (-.0005) the app seems to crash. But it seems to 'crash' only on the frontend, not in the backend. You are also allowed to insert values like 0.00.33.22 or something.
• Now sure if this is easy to do, but maybe it would be even better to have legends that incorporate the selected values for the two conductances directly (i.e. to have trace_0.0005_0.7 instead of trace 2)?

[alex4200]

Stefania is working on some details

[lbologna]

@alex4200 we implemented the final points you suggested, moved the model/data to a new container Shailesh just created for the martinello_et_al_2019 lp and inserted it in the index page. Could you please have a final look? If everything works fine I will put it in PROD and we will be able to close the ticket. Here's the DEV link: https://cnr-ibf-pa.github.io/hbp-bsp-live-papers-dev/index.html Thank you.

[alex4200]

Tests of this livepaper and the BlueNaaS integration seem to work very nicely.

From my side this ticket can be closed.

[lbologna]

Thanks Alex! Moved to production. Closing the ticket.