BluePyOpt related usecases - Full update to python3 (frontend, backend, HPC, optimized models)

[alex4200]

Full update to python3

With the start of the year 2020 python2 has been deprecated. No more updates or fixes will be created for python2 anymore, which means an update to python3 in every aspect of coding and analysis is envisaged.

This also applies to the Brain Simulation Usecases and will affect the following components in the Brain Simulation Platform:

• Frond end code, i.e. code that is running inside the jupyter notebooks (Collab)
• Backend code, i.e. code that is running on the HPC centers and on web app backends (e.g. HH-Neuron-Builder)
• Code that is part of an optimized neuron model

The recommended action is to move everything to python3 in our usecases, which might however break people's optimizations (e.g. when downloading a model containing python3 code from the cscs container and launching the optimization on a python2-based bluepyopt installation on hpc). But since these optimizations can be rerun (with python3), as bluepyopt modules based on python3 are available on PizDaint and NSG, this should not a major drawback.

@jdcourcol @mmigliore The decision has to be made if we go the recommended path with the python3 update, or if you see any issues with that action.

[clupascu]

@appukuttan-shailesh and @antonelepfl the hippocampus optimizations models list created by @antonelepfl based on queries on the model catalog contains 83 optimizations. 39 from the hippocampus_optimization container and 44 from the Migliore_2018_CA1 container. @mmigliore should we continue working in the BSP with all 83 models? So we need to convert to Python3 all 83 models? Or should we continue using in the BSP usecases only with the 39 new models and the 44 from the paper should continue existing only for the Live Paper purposes? And hence we have to convert to Python3 only 39 models...

[mmigliore]

Let's continue using in the BSP usecases only with the 39 new models. The 44 from the paper can continue to exist only for the Live Paper purposes.

On Mon, Feb 17, 2020 at 3:25 PM clupascu notifications@github.com wrote:

@appukuttan-shailesh https://github.com/appukuttan-shailesh and @antonelepfl https://github.com/antonelepfl the hippocampus optimizations models list created by @antonelepfl https://github.com/antonelepfl based on queries on the model catalog contains 83 optimizations. 39 from the hippocampus_optimization container and 44 from the Migliore_2018_CA1 container. @mmigliore https://github.com/mmigliore should we continue working in the BSP with all 83 models? So we need to convert to Python3 all 83 models? Or should we continue using in the BSP usecases only with the 39 new models and the 44 from the paper should continue existing only for the Live Paper purposes?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cnr-ibf-pa/hbp-bsp-issues/issues/526?email_source=notifications&email_token=AGZUP335VBMQUQHBFE2FZZ3RDKM4FA5CNFSM4KWQT5Q2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL6S7IY#issuecomment-587018147, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGZUP3ZHGVTVJ7OVQ2GLKBLRDKM4FANCNFSM4KWQT5QQ .

[mmigliore]

Let's do a full update of all use cases to python3.

On Mon, Feb 17, 2020 at 1:25 PM alex4200 notifications@github.com wrote:

Full update to python3?

With the start of the year 2020 python2 has been deprecated. No more updates or fixes will be created for python2 anymore, which means an update to python3 in every aspect of coding and analysis is envisaged.

This also applies to the Brain Simulation Usecases and will affect the following components in the Brain Simulation Platform:

• Frond end code, i.e. code that is running inside the jupyter notebooks (Collab)
• Backend code, i.e. code that is running on the HPC centers and on web app backends (e.g. HH-Neuron-Builder)
• Code that is part of a neuron model
• Code that is part of a neuron optimization

The recommended action is to move everything to python3 in our usecases, which might however break people's optimizations (e.g. when downloading a model containing python3 code for analysis, and launching the optimization on a python2-based bluepyopt installation on hpc). But since these optimizations can be rerun (with python3) this should not a major drawback.

@jdcourcol https://github.com/jdcourcol @mmigliore https://github.com/mmigliore The decision has to be made if we go the recommended path with the python3 update, or if you see any issues with that action.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cnr-ibf-pa/hbp-bsp-issues/issues/526?email_source=notifications&email_token=AGZUP36X7IG2IKLFARREKU3RDJ63LA5CNFSM4KWQT5Q2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IOARE2A, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGZUP34TK75S4WHTBKSD7STRDJ63LANCNFSM4KWQT5QQ .

[clupascu]

@antonelepfl I think we can fetch from the model catalog only the 39 new models (the ones from hippocampus_optimization/rat/CA1/v4.0.5) by adding to the query collab id=12027. I will update the models with the Python3 versions and we will probably have to update the download location in the model catalog and to test if they still work with BlueNaaS. @appukuttan-shailesh and @genric do you agree?

[appukuttan-shailesh]

@clupascu : Yes, I agree. The REST API to be used for retrieving relevant hippocampus models info for the use cases should then be changed to: https://validation-v1.brainsimulation.eu/models/?brain_region=hippocampus&organization=HBP-SP6&model_scope=single%20cell&species=Rattus%20norvegicus&collab_id=12027 (related to discussion in #527 )

On the model catalog, we will simply add a new "model instance" to all the existing 39 model entries. In effect, we will end up two model instances for each model - one being a Py2 variant (currently named "1.0") and the other being a Py3 variant.

So infact we do not update the download location of the existing entries, but rather add new model instances with new URLs. You can send me the list of new URLs with the corresponding model names, and I can take care of this. I believe, these new instances should automatically load up in BlueNaaS (as per currently implemented integration of model catalog and BlueNaaS).

[clupascu]

@appukuttan-shailesh the Py3 variant of the existing 39 model entries may be found here

https://object.cscs.ch/v1/AUTH_c0a333ecf7c045809321ce9d9ecdfdea/hippocampus_optimization/rat/CA1/v4.0.5/optimizations_Python3/

so in the new URLs you just have to replace 'optimizations' with 'optimizations_Python3'. Please let me know when you added the new model instances with the new URLs in the model catalog. Thanks.

[appukuttan-shailesh]

@clupascu : I have added a new instance titled "2.0" which is Py3 compatible. I have updated the descriptions of both the instances to indicate that "1.0" is "Python 2 compatible" and that "2.0" is "Python 3 compatible".

Example entry: https://collab.humanbrainproject.eu/#/collab/8123/nav/395557?state=model.055557a5-1bc6-4f7e-9816-981813596f9c

I have tested that the model download works, and also its loading in BlueNaaS.

[clupascu]

@appukuttan-shailesh Thank you very much. How the REST API to be used for retrieving relevant hippocampus models info for the use cases should then be changed in order to fetch directly the instance titled "2.0"?

[appukuttan-shailesh]

@clupascu : Welcome. As example of code changes, see: https://github.com/cnr-ibf-pa/hbp-bsp-issues/issues/527#issuecomment-588246822

Alternatively, also see: https://hbp-validation-framework-api.readthedocs.io/en/latest/model_instances.html#get-model-instance-description-using-model-uuid-and-version

[clupascu]

@wvangeit and @pramodk do you know if the bluepyopt module available in the collab is python2 or python3? While making my BluePyOpt related usecases Python3 compatible I am compiling, but when I try to run the analysis I get this error Traceback (most recent call last): File "opt_neuron.py", line 77, in evaluator = model.evaluator.create('bACnoljp', cvode_active=True, dt=0.025) File "/home/jovyan/BPOPTANALYSIS/output_2020-02-19T14:28:46+0100/output/CA1_int_bAC_011017HP2_20190329115307/model/evaluator.py", line 172, in create cvode_minstep=cvode_minstep) File "/opt/conda/lib/python3.6/site-packages/bluepyopt/ephys/simulators.py", line 32, in init self.neuron.h.load_file('stdrun.hoc') File "/opt/conda/lib/python3.6/site-packages/bluepyopt/ephys/simulators.py", line 88, in neuron NrnSimulator._nrn_disable_banner() File "/opt/conda/lib/python3.6/site-packages/bluepyopt/ephys/simulators.py", line 74, in _nrn_disable_banner nrnpy_path) Exception: hoc shared library not found in /home/jovyan/.local/nrn/lib/python/neuron

Any suggestion?

[pramodk]

@clupascu : I don’t know how collab image is built with NEURON. May be you can ask HBP support team?

[clupascu]

The bluepyopt default version in the collab is 1.6.11. Installing version 1.6.42 solves my problem.