New Live Paper Masoli et al 2019

[lbologna]

New Live Paper Masoli et al 2019

Aspect	Detail
Summary	Inserted a new Live Paper in the Live Paper page
Expert	@SteMasoli @lbologna @appukuttan-shailesh
Dependencies	None

List of additional/changed features

A new Live Paper has been added to the BSP Live Paper "dev" page and linked to the ad-hoc created collab. The live paper collab is private at the moment and contains a link to the dev live paper. Once everything will be tested the prod links will be inserted and the collab made public. Here's the link to the dev page: https://cnr-ibf-pa.github.io/hbp-bsp-live-papers-dev/

Tasks

• [x] create new live paper page
• [x] integrate in the main page
• [ ] reduce time for loading traces

Acceptance Criteria

The Live Paper is correctly visualized. All links work. The traces are visualized in a reasonable amount of time.

Extra Requirements

Performance

The trace visualizer should take a few seconds to visualize the electrophysiological activity and stimuli. At the moment, and in particular for this paper whose experimental data size goes up to 20MB, the waiting time is too long (up to minutes, depending on the internet connection)

[lbologna]

Hello,

with @SteMasoli we inserted a new Live Paper in dev as described in the issue.

@alex4200 could you please have a look and see whether everything is fine? Thank you @appukuttan-shailesh as described before, the trace visualization takes too long (especially on connections slower than academic). I believe this depends on the sampling frequency because the files have big size (~20MB) but the recording duration is not long. Would it be possible to downsample the data fetched from the visualizer to accelerate the process, via a parameter or by choosing a default (i.e. 10kHz, or even 5kHz if the traces are still well visible)? Also, I think it will help the user's experience if a "waiting wheel" (or something similar) be put somewhere in the panel so as to convey the message the system is working. Thank you

[apdavison]

@lbologna I can answer for the trace visualization. The "waiting wheel" has been developed (cf https://github.com/NeuralEnsemble/neo-viewer/issues/5) but is not yet in production. We are currently working on downsampling.

[lbologna]

Thanks @apdavison

[alex4200]

@apdavison For me the trace visualization is not working at all (even with 'academic' internet connection). I get a 504 Gateway timeout.

[lbologna]

@SteMasoli, @alex4200 I inserted the direct download of the electrophysiological traces as discussed today. Here's the link to the dev page: https://cnr-ibf-pa.github.io/hbp-bsp-live-papers-dev/2019/masoli_et_al_2019/masoli_et_al_2019.html Could you please check? If everything is ok, I will move it to production.

Thanks

[alex4200]

@SteMasoli @lbologna I took a look, and it seems to work fine. I suggest to move the live paper to production and then close the ticket.

[SteMasoli]

Thank you. @lbologna @alex4200

The paper can go live but there are two point to be added:

1) I have found a way to reduce the dimensions of the experimental traces. The experimentalist will check the traces to see if they are not ruined too much.

2) When this usecase will be deemed ok, it will be linked in the first panel of the live paper. https://github.com/cnr-ibf-pa/hbp-bsp-issues/issues/435

[lbologna]

Ok, thanks. I put it in prod as the live paper is complete now, but leave the issue open until an exploratory test with the reduced traces is done.

[alex4200]

@lbologna Are the reduced traces already implemented? Or is it some work to be done?

[lbologna]

@alex4200 as far as I understand @SteMasoli reduced the dimensions of the trace files with a software for electrophys recording (pClamp or similar). If the traces look good, these files can be put directly in the container and the in the live paper can be restored. This shoudn't take much. @SteMasoli could you please confirm or correct me if I am wrong? Thank you

[SteMasoli]

@lbologna Exactly as you say.

I'm currently waiting to receive words about the quality of the reduced version.

[SteMasoli]

@lbologna @alex4200 The new traces are online

[alex4200]

@SteMasoli Where can I find the original traces? (Just to compare them)

[SteMasoli]

@alex4200

https://object.cscs.ch:443/v1/AUTH_c0a333ecf7c045809321ce9d9ecdfdea/cerebellum_circuits/BlueNaaS_models/live_paper_grc/trace/original_traces/101018_0012%20CC%20step.abf

https://object.cscs.ch:443/v1/AUTH_c0a333ecf7c045809321ce9d9ecdfdea/cerebellum_circuits/BlueNaaS_models/live_paper_grc/trace/original_traces/111018_0001%20CC%20step.abf

https://object.cscs.ch:443/v1/AUTH_c0a333ecf7c045809321ce9d9ecdfdea/cerebellum_circuits/BlueNaaS_models/live_paper_grc/trace/original_traces/200918_0001%20CC%20step.abf

[lbologna]

@alex4200, @SteMasoli I restored the visualizer. The quality of the images seems good to me. The page does not freeze anymore but loading the traces and hovering above them still lag a bit on my pc.

Here's the link: https://cnr-ibf-pa.github.io/hbp-bsp-live-papers-dev/2019/masoli_et_al_2019/masoli_et_al_2019.html

[SteMasoli]

@lbologna @alex4200

The page does not freeze anymore but loading the traces and hovering above them still lag a bit on my pc.

The loading part is ok, for me, even on my slow connection but, there is indeed some lag in the visualization and in changing from one trace to another.

[alex4200]

@SteMasoli I am wondering: Why do I see two lines for each trace?

[SteMasoli]

@alex4200 The first one is the time voltage trace. The second is the leakage current.

[apdavison]

@alex4200 the upper line is the trace, the lower line is a zoom control: if you click on it, you will be able to drag handles to zoom in on the upper trace (although it might be very slow, as it seems even the reduced files are quite large*).

[*] Note that we plan to evaluate alternative Javascript charting libraries, to try to improve the responsiveness for large files.

[SteMasoli]

The compressed version were incomplete. I have uploaded the right one on the cscs container.

[alex4200]

@SteMasoli I am a bit confused right now.

Compressed? On the live paper page here I can choose the segment and the signal to be displayed, and for the trace 111018_0001_CC_step for example it is downloading 1.6 MB for each segment. There are 16 segments in total, summing up to about 24 MB in total. The total trace is about 24 MB. I do not see that the downloaded segment of 1.6 MB is compressed/downsampled in any way.

Performance Also, using the EPFL network with a download speed of 900 Mbit/s it takes about 6 seconds to download the 1.6 MB, which gives an actual download speed of about 0.2-0.3 MB/s. That is not very fast. Actually it the TTFB (Time to First Byte) time that is about 6 seconds, and that is a clear indication of a slow server! (The actual data is sent with a speed of about 10 MB/s).

Together with the javascript code and the the time it takes to create the plot on the frontend it is about 15 seconds for the final image to appear. That is quite a long time. For a commercial application this would be absolutely insufficient (but maybe thats ok for science work?)

Summary:

• The server takes 6 seconds to start sending the data back.
• The frontend requires about 9 seconds to start the request, handle the response, and create the plot.

[SteMasoli]

@alex4200 @lbologna

The total trace is about 24 MB.

File size taken from the CSCS container and reported when downloaded too.

18gen16_0005 CC step.abf Size = 381KiB

101018_0012 CC step.abf Size = 8.8MiB.

111018_0001 CC step.abf Size = 7.8MiB.

200918_0001 CC step.abf Size = 7.8MiB

I checked my internet connection data transfer and one single downloaded "block" is, as you say, about 1.6MB. There is some overhead somewhere because, if you do the same with the trace of the regular granule cell "18gen16_0005 CC step.abf" it download again 1.6MB from a file that is only 381KiB in total.

[apdavison]

The trace viewer is currently running on a small, demonstration server. I am in the process of writing an application for ICEI VM resources to provide a server specifically for SP6 use cases and live papers. This should improve the responsiveness.

We are also working on several performance improvements. Feel free to create a ticket at https://github.com/NeuralEnsemble/neo-viewer/issues if you have further issues.

[appukuttan-shailesh]

neo-viewer now has a new feature whereby it can automatically downsample signals by a specified factor. E.g. usage syntax:

<visualizer-view source="https://object.cscs.ch/v1/AUTH_c0a333ecf7c045809321ce9d9ecdfdea/cerebellum_circuits/BlueNaaS_models/live_paper_grc/trace/200918_0000%20IV%20-70.abf" height=300 downsamplefactor=4></visualizer-view>

downsamplefactor can take any positive integer value as input.

For basic instructions on using the neo-viewer, see here: https://neo-viewer.brainsimulation.eu/ (this page doesn't currently have info on downsamplefactor)

[alex4200]

@appukuttan-shailesh I had a look at this feature, and it seems to work fine. Please let me know when this has been implemented on the live paper.

[appukuttan-shailesh]

@alex4200 .... this feature is already available for anyone who wishes to use it. So @lbologna and/or @SteMasoli can already employ it in the live papers by deciding on an appropriate downsampling factor.

[lbologna]

@appukuttan-shailesh, @apdavison thanks for the implementation of the downsampling factor

@alex4200, @SteMasoli I applied a factor (of either 2 or 3) to all the traces and the result seems to me a good compromise between image quality and loading time. No lag occurs anymore. Could you please have a look and see if this latest version is fine with you? If yes, I will move the live paper to production. Here's the dev link: https://cnr-ibf-pa.github.io/hbp-bsp-live-papers-dev/2019/masoli_et_al_2019/masoli_et_al_2019.html

Thanks.

[SteMasoli]

@lbologna @alex4200 I have randomly chosen the traces and i see some lag here and there. Maybe is a problem of Firefox 68 engine. The tab, with a single active trace, takes over 550MB of memory.

[SteMasoli]

If the lag problem is limited to me, than the paper can go live.

[lbologna]

Hello,

@SteMasoli I set all the downsample factors at 4 for all the traces and checked on a few of them. They still look fine to me (only a few are a bit "spiky" if you strictly zoom on an individual action potential). No lag is present anymore, even when loading two traces at the same time. If they look fine for you as well and the lag is better I can upload to production.

[SteMasoli]

Hi.

@lbologna I still have some slowness, firefox 68 related at this point, but it is better than before. For me is ok for production.

[lbologna]

@alex4200 I moved the paper to production: https://humanbrainproject.github.io/hbp-bsp-live-papers/2019/masoli_et_al_2019/masoli_et_al_2019.html

Could you please check whether everything is ok? If it is fine with you, the ticket can be closed. Thank you.

[alex4200]

I checked it and it seems to be ok.