search

cnr-ibf-pa / hbp-bsp-issues

Ticketing system for developers/testers and power users of the Brain Simulation Platform of the Human Brain Project
4 stars 0 forks source link

LFP computation on HPC systems #375

Open ElisabettaGiacalone opened 5 years ago

ElisabettaGiacalone commented 5 years ago

Use Case Update

Aspect Detail
Summary LFP computation on HPC systems (Cineca ,Juelich, CSCS) in addition to BBP system
Expert @pramodk
Deadline TBD
Dependencies HPC

Remove anything that does not apply

List of additional/changed features

The new feature will allow to run the following code on the above mentioned HPC systems:

module purge module load nix/viz/emsim emsim -i /gpfs/bbp.cscs.ch/project/proj42/home/giacalon/w1syn01/BlueConfig --target sliceb2 --report AllCompartmentsMembrane --voxel-size 1,1,1 --start-time 200 --end-time 1500 --output=sliceb2_9 \ --sample-point 2494.5,5725.4,5976.8

Tasks

Acceptance Criteria

Extra Requirements

System

Performance

pramodk commented 5 years ago

@ppodhajski : From simulation side we are deploying all stack via Spack. As this package is only with nix, could you have a look at this? If viz would like to have this deployed via Nix, @tristan0x can help you . Or, if you can package inside spack, it can be deployed with simulation stack.

ppodhajski commented 5 years ago

Yes, I will have a look,. In the team we had some discussion on Spack, so maybe it is a good opportunity to test if Spack works for viz team.

jdcourcol commented 5 years ago

@ppodhajski Please deploy with nix first. I think Spack investigation can take a long time and I would avoid to have that in the critical path.

alex4200 commented 5 years ago

@ElisabettaGiacalone Please see related ticket #154

jdcourcol commented 5 years ago

@alex4200 can we have @ppodhajski assigned to that one instead ? did he accept the invite ?

alex4200 commented 5 years ago

Hi,

I am still awaiting for @ppodhajskihttps://github.com/ppodhajski and Pablo Serna to accept the invitation. Should I contact them about that, or can you do that?https://g

Alex

From: jdcourcol notifications@github.com Sent: Tuesday, January 29, 2019 12:59:27 PM To: cnr-ibf-pa/hbp-bsp-issues Cc: Dietz Alexander; Mention Subject: Re: [cnr-ibf-pa/hbp-bsp-issues] LFP computation on HPC systems (#375)

@alex4200https://github.com/alex4200 can we have @ppodhajskihttps://github.com/ppodhajski assigned to that one instead ? did he accept the invite ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/cnr-ibf-pa/hbp-bsp-issues/issues/375#issuecomment-458514465, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AMI7y46jQ-smSk8rR4zZ6V0wdYfOZh3Zks5vIDefgaJpZM4aPFW-.

ppodhajski commented 5 years ago

accepted

alex4200 commented 5 years ago

@ppodhajski Is there any news regarding this issue?

ppodhajski commented 5 years ago

with help of Tristan I installed nix on jurecavis (via spack), as it had to be reinstalled after introducing new usage model at JSC. Currently trying to fix the issue when building with nix.

The problem encountered: patching /p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/nix/store/aa8kd6z12mj69yx89m8km66ccs7cdli6-bootstrap-tools/bin/[ stat: Value too large for defined data type

ppodhajski commented 5 years ago

@ElisabettaGiacalone please try executing: /p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/AppRun

ElisabettaGiacalone commented 5 years ago

@ppodhajski I get "permission denied" to access to emsim folder

ppodhajski commented 5 years ago

@ElisabettaGiacalone please retry

ElisabettaGiacalone commented 5 years ago

@ppodhajski Now I can access to emsim folder, but when I execute /p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/AppRun I get: /p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/AppRun: line 7: /p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/bin/emsim: Permission denied

ppodhajski commented 5 years ago

should be good now

wt., 19 lut 2019 o 13:51 ElisabettaGiacalone notifications@github.com napisał(a):

@ppodhajski https://github.com/ppodhajski Now I can access to emsim folder, but when I execute /p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/AppRun I get: /p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/AppRun: line 7: /p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/bin/emsim: Permission denied

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cnr-ibf-pa/hbp-bsp-issues/issues/375#issuecomment-465116545, or mute the thread https://github.com/notifications/unsubscribe-auth/AS6xo__Fng_HmwwP5QAYnc_FdBiy2ww0ks5vO_NVgaJpZM4aPFW- .

clupascu commented 5 years ago

I get now /p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/bin/emsim: error while loading shared libraries: libboost_program_options.so.1.54.0: cannot open shared object file: No such file or directory

ppodhajski commented 5 years ago

it is strange, I added some debug print, please try again and report LD_LIBRARY_PATH (it will print its content).

wt., 19 lut 2019 o 13:56 clupascu notifications@github.com napisał(a):

I get now /p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/bin/emsim: error while loading shared libraries: libboost_program_options.so.1.54.0: cannot open shared object file: No such file or directory

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cnr-ibf-pa/hbp-bsp-issues/issues/375#issuecomment-465118062, or mute the thread https://github.com/notifications/unsubscribe-auth/AS6xo29dngFtHbmtJnC8ZIUBfI0zqI07ks5vO_SHgaJpZM4aPFW- .

clupascu commented 5 years ago

/p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/AppRun LD_LIBRARY_PATH to be reported: /p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/lib:/usr/local/software/jureca/Stages/2018b/software/binutils/2.31.1-GCCcore-7.3.0/lib:/usr/local/software/jureca/Stages/2018b/software/GCCcore/7.3.0/lib/gcc/x86_64-pc-linux-gnu/7.3.0:/usr/local/software/jureca/Stages/2018b/software/GCCcore/7.3.0/lib64:/usr/local/software/jureca/Stages/2018b/software/GCCcore/7.3.0/lib Error: the option '--input' is required but missing

-h [ --help ] Print this help message.

-i [ --input ] arg Path to Blueconfig file. -o [ --output ] arg Path for the output file. --target arg The circuit's target. --report arg The name of the report. --start-time arg The start time --end-time arg The end time --fraction arg Specify the fraction [0.0 1.0] of gids to be used during the computation. Default is 1.0. --export-volume Will export a floating point volume for each time steps.

--voxel-size arg The size in each dimension of a voxel in circuit units. Default is 4.0,4.0,4.0. Must be written in the form: --voxel-size rx,ry,rz --volume-extent arg Specify an additional 3d extent for the volume in micrometers. Default is 0.0,0.0,0.0. Must be written in the form: --volume-extent ex,ey,ez --sample-point arg The x y z positions of a sample point. Must be written in the form: --sample-point x,y,z

ppodhajski commented 5 years ago

I modified the permissions in the meantime on library folder and it seems to work now, right?

wt., 19 lut 2019 o 14:09 clupascu notifications@github.com napisał(a):

/p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/AppRun LD_LIBRARY_PATH to be reported: /p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/lib:/usr/local/software/jureca/Stages/2018b/software/binutils/2.31.1-GCCcore-7.3.0/lib:/usr/local/software/jureca/Stages/2018b/software/GCCcore/7.3.0/lib/gcc/x86_64-pc-linux-gnu/7.3.0:/usr/local/software/jureca/Stages/2018b/software/GCCcore/7.3.0/lib64:/usr/local/software/jureca/Stages/2018b/software/GCCcore/7.3.0/lib Error: the option '--input' is required but missing

-h [ --help ] Print this help message.

-i [ --input ] arg Path to Blueconfig file. -o [ --output ] arg Path for the output file. --target arg The circuit's target. --report arg The name of the report. --start-time arg The start time --end-time arg The end time --fraction arg Specify the fraction [0.0 1.0] of gids to be used during the computation. Default is 1.0. --export-volume Will export a floating point volume for each time steps.

--voxel-size arg The size in each dimension of a voxel in circuit units. Default is 4.0,4.0,4.0. Must be written in the form: --voxel-size rx,ry,rz --volume-extent arg Specify an additional 3d extent for the volume in micrometers. Default is 0.0,0.0,0.0. Must be written in the form: --volume-extent ex,ey,ez --sample-point arg The x y z positions of a sample point. Must be written in the form: --sample-point x,y,z

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cnr-ibf-pa/hbp-bsp-issues/issues/375#issuecomment-465121927, or mute the thread https://github.com/notifications/unsubscribe-auth/AS6xowxhpNkE3kvE5jRTz47wiqohFi9Wks5vO_eMgaJpZM4aPFW- .

ppodhajski commented 5 years ago

I am not emsim developer but this seems to be application stdout or stderr, which means you need to specify the --input parameter. I think you need to specify --output and --report as well.

wt., 19 lut 2019 o 14:13 clupascu notifications@github.com napisał(a):

it is not working

/p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/AppRun Error: the option '--input' is required but missing

-h [ --help ] Print this help message.

-i [ --input ] arg Path to Blueconfig file. -o [ --output ] arg Path for the output file. --target arg The circuit's target. --report arg The name of the report. --start-time arg The start time --end-time arg The end time --fraction arg Specify the fraction [0.0 1.0] of gids to be used during the computation. Default is 1.0. --export-volume Will export a floating point volume for each time steps.

--voxel-size arg The size in each dimension of a voxel in circuit units. Default is 4.0,4.0,4.0. Must be written in the form: --voxel-size rx,ry,rz --volume-extent arg Specify an additional 3d extent for the volume in micrometers. Default is 0.0,0.0,0.0. Must be written in the form: --volume-extent ex,ey,ez --sample-point arg The x y z positions of a sample point. Must be written in the form: --sample-point x,y,z

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cnr-ibf-pa/hbp-bsp-issues/issues/375#issuecomment-465122993, or mute the thread https://github.com/notifications/unsubscribe-auth/AS6xo7Meqsz1Id-5tepqxgT8KpzYHfLbks5vO_hsgaJpZM4aPFW- .

ElisabettaGiacalone commented 5 years ago

Yes, I get the same output, but I suppose it is the output with all options of the function. I am going to try with all the required data and I will give you a feedback.

ElisabettaGiacalone commented 5 years ago

Hi @ppodhajski, I tried with a test simulation output, and it works.

/p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/AppRun -i /p/scratch/cvsk25/vsk2505/test-190219/BlueConfig --target slice0 --report AllCompartmentsMembrane --voxel-size 1,1,1 --start-time 0 --end-time 300 --output=slice0 --sample-point 3740.8,2398.6,5724.4

ElisabettaGiacalone commented 5 years ago

@ppodhajski Would it be be possible to do the same on Cineca and CSCS systems?

ppodhajski commented 5 years ago

@ElisabettaGiacalone, yes I will work on that and let you know when it is in place

ppodhajski commented 5 years ago

@pramodk Could you please let me know the path (gpfs) where I can deploy it on Cineca?

pramodk commented 5 years ago

In the past Adrien was deploying stuff on Marconi in /marconi_work/HBP_CDP2_it/nix.

I am not sure if project (HBP_CDP2_it) is changed. @clupascu or @ElisabettaGiacalone should confirm the project directory under /macroni_work.

clupascu commented 5 years ago

The project is HBP_CDP2_it_0

ppodhajski commented 5 years ago

@ElisabettaGiacalone: I put Emsim here: /marconi_work/HBP_CDP2_it_0/bbp-visualisation/emsim/emsim

alex4200 commented 5 years ago

Is there any update on this issue?

@romani79: For your information

ElisabettaGiacalone commented 5 years ago

I have not tested yet on Marconi with data from simulation because I want to test it on the new full scale Hippocampus model, but it seems to work.

@ppodhajski, Is there any update on the installation on CSCS ?

ppodhajski commented 5 years ago

@ElisabettaGiacalone I am waiting for access rights to CSCS. Will update the ticket then.

alex4200 commented 5 years ago

@ppodhajski No news on this item?

ppodhajski commented 5 years ago

@ppodhajski, Is there any update on the installation on CSCS ?

@ElisabettaGiacalone what path on CSCS should it be deployed?

clupascu commented 5 years ago

I think it can be deployed here /apps/hbp/ich002/hbp-spack-deployments/

ppodhajski commented 5 years ago

@clupascu @elisamarenzi please see it under /apps/hbp/ich002/hbp-visualisation-deployements/emsim/emsim

ElisabettaGiacalone commented 5 years ago

@ppodhajski I have just tested that on CSCS with this code:

/apps/hbp/ich002/hbp-visualisation-deployements/emsim/emsim -i /scratch/snx3000/bp000139/w1syn03_testLFP/BlueConfig --target slice2 --report AllCompartmentsMembrane --voxel-size 1,1,1 --start-time 200 --end-time 600 --output=slice2 --sample-point 3740.8,2398.6,5724.4

But I get this error (despite the input is specified) :

Error: the option '--input' is required but missing

-h [ --help ] Print this help message.

-i [ --input ] arg Path to Blueconfig file. -o [ --output ] arg Path for the output file. --target arg The circuit's target. --report arg The name of the report. --start-time arg The start time --end-time arg The end time --fraction arg Specify the fraction [0.0 1.0] of gids to be used during the computation. Default is 1.0. --export-volume Will export a floating point volume for each time steps.

--voxel-size arg The size in each dimension of a voxel in circuit units. Default is 4.0,4.0,4.0. Must be written in the form: --voxel-size rx,ry,rz --volume-extent arg Specify an additional 3d extent for the volume in micrometers. Default is 0.0,0.0,0.0. Must be written in the form: --volume-extent ex,ey,ez --sample-point arg The x y z positions of a sample point. Must be written in the form: --sample-point x,y,z

ppodhajski commented 5 years ago

Hi @ElisabettaGiacalone, I am back to the office after my holidays, please try it now; there was a small permission issue

ElisabettaGiacalone commented 5 years ago

Hi, I just tried on CSCS and it works now. Thanks.

I am going to test it on Marconi.

ElisabettaGiacalone commented 5 years ago

@ppodhajski I have just tried on Marconi, but I think I have the same permission problem as before on CSCS:

[mmiglior@r000u07l02 test-31aug]$ /marconi_work/HBP_CDP2_it_0/bbp-visualisation/emsim/emsim -i /marconi_work/HBP_CDP2_it_0/hippo-mm/test-31aug/BlueConfig --target slice2 --report AllCompartmentsMembrane --voxel-size 1,1,1 --start-time 0 --end-time 300 --output=slice2 --sample-point 3740.8,2398.6,5724.4 Error: the option '--input' is required but missing

-h [ --help ] Print this help message.

-i [ --input ] arg Path to Blueconfig file. -o [ --output ] arg Path for the output file. --target arg The circuit's target. --report arg The name of the report. --start-time arg The start time --end-time arg The end time --fraction arg Specify the fraction [0.0 1.0] of gids to be used during the computation. Default is 1.0. --export-volume Will export a floating point volume for each time steps.

--voxel-size arg The size in each dimension of a voxel in circuit units. Default is 4.0,4.0,4.0. Must be written in the form: --voxel-size rx,ry,rz --volume-extent arg Specify an additional 3d extent for the volume in micrometers. Default is 0.0,0.0,0.0. Must be written in the form: --volume-extent ex,ey,ez --sample-point arg The x y z positions of a sample point. Must be written in the form: --sample-point x,y,z

ppodhajski commented 5 years ago

@ElisabettaGiacalone it is fixed now!

ElisabettaGiacalone commented 5 years ago

Thanks, it works now. We can close the issue.

antonelepfl commented 4 years ago

Hi @ppodhajski, I would need this fix deployed on PizDaint and Jureca.

ppodhajski commented 4 years ago

hi @antonelepfl, I am trying to deploy it via spack on piz daint but ran into some compilation issues. will keep you posted.

antonelepfl commented 4 years ago

Do you have any update on this?

ppodhajski commented 4 years ago

please try binaries in the folder below: /apps/hbp/ich002/hbp-visualisation-deployements/spack/softwares/28-04-2020/install/install/cray-cnl7-haswell/intel-19.0.1.144/emsim-1.0.0-vbdt6g/bin

or built with gcc: /apps/hbp/ich002/hbp-visualisation-deployements/spack/softwares/28-04-2020/install/install/cray-cnl7-haswell/gcc-8.3.0/emsim-1.0.0-lewa5k/bin

These are built with brion version 3.1.0 (https://github.com/BlueBrain/Brion/releases)

ElisabettaGiacalone commented 4 years ago

Hi @ppodhajski, I am trying emsim on Jureca for the first time with the new model, but I get this error:

[giacalone1@jrl12 type_l_test]$ /p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/emsim -i BlueConfig --target slice30 --report AllCompartmentsMembrane --voxel-size 1,1,1 --start-time 0 --end-time 300 --output=slice0 --sample-point 3740.8,2398.6,5724.4 WARNING: Time range is clamped to the minimum bound. INFO: Time range is: [300 300] with DT: 0.25 Normalized current sum: -3.43139e-09 INFO: Compartments count: 8300167 terminate called after throwing an instance of 'std::runtime_error' what(): Could not open morphology file /p/project/cvsk25/vsk2516/hpc/entities/morphologies/20180417//011127HP1_-_Clone_0.h5: Unable to open file /p/project/cvsk25/vsk2516/hpc/entities/morphologies/20180417//011127HP1_-_Clone_0.h5 (File accessibilty) Unable to open file /p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/emsim: line 11: 31321 Aborted LC_NUMERIC=C LD_LIBRARY_PATH=${HERE}/lib:$LD_LIBRARY_PATH ${HERE}/bin/emsim "$@"

ppodhajski commented 4 years ago

Hi, new version is deployed on piz daint only. please try with the binaries I posted above

wt., 28 kwi 2020 o 18:34 ElisabettaGiacalone notifications@github.com napisał(a):

Hi @ppodhajski https://github.com/ppodhajski, I am trying emsim on Jureca for the first time with the new model, but I get this error:

[giacalone1@jrl12 type_l_test]$ /p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/emsim -i BlueConfig --target slice30 --report AllCompartmentsMembrane --voxel-size 1,1,1 --start-time 0 --end-time 300 --output=slice0 --sample-point 3740.8,2398.6,5724.4 WARNING: Time range is clamped to the minimum bound. INFO: Time range is: [300 300] with DT: 0.25 Normalized current sum: -3.43139e-09 INFO: Compartments count: 8300167 terminate called after throwing an instance of 'std::runtimeerror' what(): Could not open morphology file /p/project/cvsk25/vsk2516/hpc/entities/morphologies/20180417//011127HP1-_Clone0.h5: Unable to open file /p/project/cvsk25/vsk2516/hpc/entities/morphologies/20180417//011127HP1-_Clone_0.h5 (File accessibilty) Unable to open file /p/project/cvsk25/bbp-viz-deployment/HBP/jurecavis/appimage/emsim/emsim: line 11: 31321 Aborted LC_NUMERIC=C LD_LIBRARY_PATH=${HERE}/lib:$LD_LIBRARY_PATH ${HERE}/bin/emsim "$@"

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cnr-ibf-pa/hbp-bsp-issues/issues/375#issuecomment-620718622, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEXLDI7JH54G4I4BQDRMU6LRO4AQBANCNFSM4GR4KW7A .

ElisabettaGiacalone commented 4 years ago

@ppodhajski please, could you deploy it also on Jureca?

antonelepfl commented 4 years ago

Your deployment on PizDaint works correctly. I tried both paths and they work. Which one do you recommend to use?

ppodhajski commented 4 years ago

I cannot download some sources on jureca:

[podhajski1@jrc1387 spack]$ wget https://sourceware.org/
--2020-04-29 13:28:49--  https://sourceware.org/
Resolving sourceware.org (sourceware.org)... 209.132.180.131
Connecting to sourceware.org (sourceware.org)|209.132.180.131|:443...

Do you know if I there is some proxy I could use? In the meantime I created a ticket for this issue.

ppodhajski commented 4 years ago

Which one do you recommend to use?

Both are fine to use.