search

cnr-ibf-pa / hbp-bsp-issues

Ticketing system for developers/testers and power users of the Brain Simulation Platform of the Human Brain Project
4 stars 0 forks source link

Spectrum analysis should be added #154

Closed antonelepfl closed 5 years ago

antonelepfl commented 6 years ago

Use Case Update

Aspect Detail
Summary The idea is to have an analysis of the Local Field Potential (LFP) as part of the SimulationUI analysis. (Michele, 9.10.2018). This LFP is a time-series of voltages.
Expert @antonelepfl @romani79
Deadline
Dependencies Juelich systems

List of additional/changed features

Tasks

Acceptance Criteria

Extra Requirements

System

Performance

antonelepfl commented 6 years ago

Armando has the functions to transform (PSTH + Raster) to signal and from there add this Spectrum analysis.

antonelepfl commented 6 years ago

I have asked:

How this spectrum analysis is obtained? Do we have the software to generate them available in Jureca?

Michele answered:

No, I do not think that the software is on jureca. We (Armando and ELisabetta) use the bbp library with jupyter notebooks on bbp5. Iasked to have it installed also on jureca but no feedback so far. You may ask Jean-Denis.

Armando confirmed what Michele wanted is spectrum analysis on the LFP.

apdavison commented 6 years ago

Are we using Elephant for these analyses? Elephant is being developed in SP5 for electrophysiology data analysis, and I think it would make sense to use it (and/or contribute to it) rather than develop independent analysis tools within SP6.

Currently available spectral analysis tools in Elephant: https://elephant.readthedocs.io/en/latest/reference/spectral.html

mmigliore commented 6 years ago

I fully agree. If there is a tool already open and available we should use it. And Elephant is one of these. Could you coordinate with Jean-Denis and Stefano to add it to the use case?

Andrew Davison notifications@github.com ha scritto:

Are we using Elephant for these analyses? Elephant is being
developed in SP5 for electrophysiology data analysis, and I think it
would make sense to use it (and/or contribute to it) rather than
develop independent analysis tools within SP6.

Currently available spectral analysis tools in Elephant:
https://elephant.readthedocs.io/en/latest/reference/spectral.html

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/cnr-ibf-pa/hbp-bsp-issues/issues/154#issuecomment-431792338

jdcourcol commented 6 years ago

@romani79 @mmigliore : please provide the script to be added and the needed dependencies, and we will integrate it. if that script uses Elephant, the better.

romani79 commented 6 years ago

I tried to use Elephant. Anyway, python already provides a complete library for the signal processing, while Elephant only provides very few functions. Furthermore, the LFP signal, stored as a two-columns matrix - times and voltage, should be processed to be understood by Elephant. In conclusion, during my initial attempt to use Elephant, I could find it beneficial.

Regarding the spectrum analysis, consider that this is useless if the user cannot compute the LFP in the BSP. Is this feature available?

jdcourcol commented 5 years ago

@romani79 : "compute the LFP" you are already doing that no ?

mmigliore commented 5 years ago

No, it can be done only by connecting with the bp5 host and using one of Armando jupyter notebooks. It cannot be done from the collab.

https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon Virus-free. www.avast.com https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

On Mon, Nov 5, 2018 at 2:37 AM jdcourcol notifications@github.com wrote:

@romani79 https://github.com/romani79 : "compute the LFP" you are already doing that no ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cnr-ibf-pa/hbp-bsp-issues/issues/154#issuecomment-435829184, or mute the thread https://github.com/notifications/unsubscribe-auth/AbNH79eQn92CHS1HfoBCGUw4PeW2bn0Eks5usBTwgaJpZM4Rmfpx .

alex4200 commented 5 years ago

@mmigliore @jdcourcol @romani79 @antonelepfl : So what are the next actions to take? Or can this ticket not be realized at all?

jdcourcol commented 5 years ago

@romani79 : if we want to activate this use case, we need the code so that we can run it through Unicore. Even if it is in a notebook, we can get the code and run it like that.

romani79 commented 5 years ago

As we clarified, the spectrum analysis should be done on the LFP signal. Anyway, LFP cannot be computed in BSP. I would first make the LFP calculation available on BSP and then make the relative analyses like the (power) spectrum.

Regarding Elephant, I agree we should use this tool when possible. Anyway, it requires some time to make it works with our simulation outputs. If someone likes to take this task, I can provide support.

jdcourcol commented 5 years ago

Anyway, LFP cannot be computed in BSP.

what is the reason for that ?

alex4200 commented 5 years ago

@jdcourcol @romani79 @mmigliore @antonelepfl

I have significantly updated this ticket. Please see the top level description for a much better breakdown of this task.

@jdcourcol The reason that the LFP might not be calculated in the BSP (inside a notebook) is that you have to loop over (Number Neuron)*(Number compartments) elements. But this is part of the investigation (see checkboxes).

jdcourcol commented 5 years ago

Why should LFP be executed in a notebook ? the code can be run as a python script through a unicore job like any other right ?

mmigliore commented 5 years ago

Sure, provided that all the relevant parameters are available in the GUI (target, time window, electrode positions, etc) there is no probem in having it as a webapp.

On Thu, Dec 6, 2018 at 12:07 PM jdcourcol notifications@github.com wrote:

Why should LFP be executed in a notebook ? the code can be run as a python script through a unicore job like any other right ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cnr-ibf-pa/hbp-bsp-issues/issues/154#issuecomment-444835594, or mute the thread https://github.com/notifications/unsubscribe-auth/AbNH72uhdAUAz7eelSbhMq4ebN4cQTkIks5u2PqCgaJpZM4Rmfpx .

alex4200 commented 5 years ago

There is already a C++ tool to calculate the LFP, but it is not open-sourced (from Grigori Chevtchenko, 6.12.2018):

You can find the source code here: https://bbpcode.epfl.ch/code/#/admin/projects/viz/EMSim. As you can see this code is not open source so I think you cannot send it to not bbp people. It's also not possible to deploy it on Juelich because of that.

We will ask Felix to open source this code but it can take some time.

I see the following options to continue this task:

  1. Talk to Grigori to create a python script to just calculate the LFP to be run on Juelich.
  2. Talk to Grigori to create a python script to just calculate the LFP to be run within a notebook.
  3. Look into using a python package https://github.com/LFPy/LFPy which has been used at NMBU/Norway (Torbjørn V Ness torbjorn.ness@nmbu.no) with the existing plan to use this on Juelich.
  4. Wait until the C++ tool is opened sourced. This can take a while

My suggestion is to focus on option 3.

jdcourcol commented 5 years ago

The code can be deployed on Juelich. Being open source or not does not prevent the deployment. Gregory should make his package deployable as a nix module. May be it is already.

mmigliore commented 5 years ago

Please deploy it also on cscs.

Il giorno Ven 7 Dic 2018, 12:01 jdcourcol notifications@github.com ha scritto:

The code can be deployed on Juelich. Being open source or not does not prevent the deployment. Gregory should make his package deployable as a nix module. May be it is already.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cnr-ibf-pa/hbp-bsp-issues/issues/154#issuecomment-445212092, or mute the thread https://github.com/notifications/unsubscribe-auth/AbNH76SWWtgSIn_ie7PGfLQCi1MLkl42ks5u2lh_gaJpZM4Rmfpx .

alex4200 commented 5 years ago

@mmigliore This tool (the C++ module from Grigori Chevtchenko) is already installed in BB5

jdcourcol commented 5 years ago

@alex4200 I think @mmigliore means Piz Daint

mmigliore commented 5 years ago

Yes, the deployment should be done on all HPC systems where the model circuits are installed (Piz Daint, Jureca, and Marconi)

https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon Virus-free. www.avast.com https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

On Mon, Dec 10, 2018 at 8:04 AM jdcourcol notifications@github.com wrote:

@alex4200 https://github.com/alex4200 I think @mmigliore https://github.com/mmigliore means Piz Daint

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cnr-ibf-pa/hbp-bsp-issues/issues/154#issuecomment-445725261, or mute the thread https://github.com/notifications/unsubscribe-auth/AbNH78KKJexYuVwa0voNk6odZw9y1gstks5u3hWngaJpZM4Rmfpx .

romani79 commented 5 years ago

I will summarize the status of this task.

  1. First of all we should make the LFP calculation available on BSP and then think about all the analyses.
  2. After all, the analyses can be done also by open source python libraries with few lines of code. Anyway, we are trying to use Elephant for that. So, this is not the real problem.

Regarding 1, we have two options: A) use BBP tool, but as far as I know there is no plan to make it open source. B) use Torbjørn Ness' tool (Norwegian University of life Sciences).

I asked Torbjørn if he is willing to make the tool available on the BSP. If this is the case, we have to test if his tool and the BBP tool produce similar results given a BBP simulation. For that, the BBP simulation should be converted in sonata formata and I am working on that as well.

alex4200 commented 5 years ago

We had a skype call with @romani79, @apdavison and Michael Denker about Elephant and its probable use in relation to the analysis in this task (see also previous comment).

There are issues regarding the input data for elephant, and possibilities for LFP specific analyses that can be done. In order to proceed we agreed on the following tasks:

romani79 commented 5 years ago

I will share an update on this issue. The analyse of LFP is not a problem. We have custom python code or Elephant. The problem remains always the tool to calculate the LFP. I asked Samuel if we can make the viz tool available, and he asked Felix. Unfortunately, I am still waiting a reply. Regarding Torbjørn Ness' tool, we invested some time to make use of it but it still requires some additional efforts to make our simulation output readable by him.

romani79 commented 5 years ago

To advance on this issue, we have to:

  1. Use SimUI to launch a simulation with currents reported from all comportaments
  2. Use another UI to calculate LFP
  3. Implement whatever analyses on the LFP

Is point 1 already available? Do you need the correct setting to launch this simulation? What about point 2?

antonelepfl commented 5 years ago

For the point 1 we need to have the Target AllCompartments on the Microcircuit (that is not the case today on the O1 20181114). For the point 2 wee need to create it from scratch. In here you can give your inputs about what do you have in mind.

romani79 commented 5 years ago

for the LFP UI, you should check with the owners to see what kind of parameters we can set. I can give you an example of command I launch with explanation of the parameters.

emsim -i BlueConfig --target Mosaic --report AllCompartmentsMembrane --voxel-size 1,1,1 --start-time 0 --end-time 3000 --output=central --sample-point 360.0,0.0,630.0

target is the set of cells to use to compute LFP report is the report to use. It can be set to AllCompartmentsMembrane as default voxel-size I am not really change this. It should be the spatial resolution. 1,1,1 can be used as default start and end time. I would put start and end time of the simulation as default output is the filename output sample-point is the x,y,z coordinates of the point where we want to calculate the LFP. Normally, user should be able to pass n points. For small n, we can insert this manually, but for higher n it is better to upload a txt file containing a list of points.

romani79 commented 5 years ago

To calculate LFP, we need specific simulations and changes in the user.target. Here what you would need. user.target has to contain the following entry: Target Compartment LFPCompartments { Mosaic }

The BlueConfig should have the following block: Report AllCompartmentsMembrane { Target LFPCompartments Type Summation ReportOn i_membrane IClamp Unit nA Format Bin Dt 0.5 StartTime 0 EndTime 5000 }

An example of simulation, LFP calculation, and LFP analysis is available at: /gpfs/bbp.cscs.ch/project/proj42/simulations/O1.20181114/20181214

antonelepfl commented 5 years ago

@romani79 Can you provide me the scripts to run the Spectrum and the Traces analysis?

antonelepfl commented 5 years ago

I need emsim deployed in Julich to continue testing this internal ticket

clupascu commented 5 years ago

this is already done. Take a look here https://github.com/cnr-ibf-pa/hbp-bsp-issues/issues/375