search

cnr-ibf-pa / hbp-bsp-issues

Ticketing system for developers/testers and power users of the Brain Simulation Platform of the Human Brain Project
4 stars 0 forks source link

Code error in "Multicompartmental simulation of the cerebellum" usecase #544

Closed alex4200 closed 4 years ago

alex4200 commented 4 years ago

Expected behavior

No error in the notebook

Actual Behavior (please include screenshot if possible)

Code error in code cell below simulation run:

building swapped
building swapped
...
Node 0 created 1382 cells
Cell creation on node 0 took 42.99 seconds
Source is a relay; Skipping connection model glomerulus_to_granule transmitters
Source is a relay; Skipping connection model mossy_to_glomerulus transmitters
Node 0 created 1365 transmitters
Node 0 created 2403 transmitters
Node 0 created 5505 transmitters
Node 0 created 9167 transmitters
Transmitter creation on node 0 took 5.22 seconds
Indexing relays
Intermediate resolved to 328 targets
Intermediate resolved to 1160 targets
Intermediate resolved to 754 targets
Intermediate resolved to 1258 targets
Intermediate resolved to 1468 targets
I need to receive from 117 relays
Receiver creation on node 0 took 1.8 seconds
Creating spike generator patterns for 'mossy_fiber_input'
Pattern [96.31750386310384] for 5.0
Pattern [59.347257706326204, 85.62130611796684] for 6.0
Pattern [80.51546330894682] for 7.0
Pattern [79.86354952095301, 86.48543488077758] for 8.0
Pattern [66.94797465644716, 81.84188062447068, 88.02790168807945, 99.59564098970681] for 9.0
Targetting representatives [323, 0, 3, 21, 12]
Traceback (most recent call last):
  File "/opt/conda/bin/scaffold", line 11, in <module>
    sys.exit(scaffold_cli())
  File "<frozen scaffold.cli>", line 26, in scaffold_cli
  File "<frozen scaffold.cli>", line 237, in start_cli
  File "<frozen scaffold.core>", line 395, in run_simulation
  File "<frozen scaffold.simulators.neuron.adapter>", line 285, in simulate
  File "<frozen scaffold.simulation>", line 100, in progress
NameError: name 'types' is not defined

Steps to reproduce the problem

Run the notebook cell - by-cell.

But still I can continue with the following code cells, maybe it is not a critical error?

Optional infrastructural data (user, platform, browser, environment, ...)

PROD

Helveg commented 4 years ago

There seems to be an import types missing in scaffold/simulation.py. But it is indeed only to visually report progress of the simulation, and not an essential part of the simulation.

Where should I fix this error, just in my original notebook or is some copy of it made during deployment?

alex4200 commented 4 years ago

Hi @Helveg ,

the best place to fix this error is the DEV location of the notebook which is under "file" in the usecases.json:

"file": "9a820c03-fe7f-41f2-aede-ab8eeae16f5e",

which in turn points to this location:

https://collab.humanbrainproject.eu/#/collab/79605/Cerebellar scaffold multicompartmental simulation.ipynb

which in turn might be one of the notebooks in that collab (I seem to not access).

Let me know when you fixed this notebook.

Thanks Alex

Helveg commented 4 years ago

@alex4200 I believe I fixed the problem locally but the notebook seems to be down. Maybe related to the recent cyber attack?

alex4200 commented 4 years ago

@Helveg Thanks for the update, and yes the notebooks are unavailable at the moment because CSCS is down as a result of the cyberattack.

alex4200 commented 4 years ago

@Helveg The notebooks seem to be back running.

alex4200 commented 4 years ago

@Helveg I did a rerun of the use case, but I ran into problems in the third code cell (notebook 79605/Cerebellar scaffold multicompartmental simulation.ipynb) with the following error:

---------------------------------------------------------------------------
NeuronError                               Traceback (most recent call last)
/opt/conda/lib/python3.6/site-packages/glia/__init__.py in <module>()
     40         else:
---> 41             manager.start(load_dll=not bool(os.getenv("GLIA_NO_AUTOLOAD_DLL")))
     42 except GliaError as e:

/opt/conda/lib/python3.6/site-packages/glia/glia.py in start(self, load_dll)
     34         self.compile(check_cache=True)
---> 35         self.init_neuron(load_dll=load_dll)
     36 

/opt/conda/lib/python3.6/site-packages/glia/glia.py in init_neuron(self, load_dll)
     43         if load_dll:
---> 44             self.load_neuron_dll()
     45 

/opt/conda/lib/python3.6/site-packages/glia/glia.py in load_neuron_dll(self)
    319             if not self._dll_loaded:
--> 320                 raise NeuronError("Library could not be loaded into NEURON.")
    321         else:

NeuronError: Library could not be loaded into NEURON.

During handling of the above exception, another exception occurred:

NameError                                 Traceback (most recent call last)
<ipython-input-1-50093042d933> in <module>()
      5 get_ipython().system('{sys.executable} -m pip install rtree mpi4py')
      6 get_ipython().system('{sys.executable} -m pip install pynb-sim==0.2.0 dbbs-scaffold==3.2.11rc0 dbbs_models dbbs_mod_collection')
----> 7 import rtree, scaffold, dbbs_models
      8 finalize_packages()

/opt/conda/lib/python3.6/site-packages/dbbs_models/__init__.py in <module>()
      4 if not os.getenv("SETUPTOOLS_INSTALL"):
      5 
----> 6     from .granule_cell_models import GranuleCell
      7     from .stellate_cell_models import StellateCell
      8     from .basket_cell_models import BasketCell

/opt/conda/lib/python3.6/site-packages/dbbs_models/granule_cell_models.py in <module>()
      1 import numpy as np
      2 from patch import p
----> 3 from arborize import NeuronModel
      4 from math import floor
      5 

/opt/conda/lib/python3.6/site-packages/arborize/__init__.py in <module>()
----> 1 from .core import NeuronModel
      2 from .synapse import Synapse
      3 
      4 __version__ = "1.1.0"
      5 

/opt/conda/lib/python3.6/site-packages/arborize/core.py in <module>()
      7     from patch import p
      8     from patch.objects import Section
----> 9     import glia as g
     10     from .synapse import Synapse
     11     p.load_file('stdlib.hoc')

/opt/conda/lib/python3.6/site-packages/glia/__init__.py in <module>()
     42 except GliaError as e:
     43     print("GLIA ERROR", e)
---> 44     exit(1)
     45 
     46 

NameError: name 'exit' is not defined
Helveg commented 4 years ago

I don't think I left my notebook in a runnable state before the crash, I can look at it tonight

alex4200 commented 4 years ago

Ah ok, thanks a lot

Helveg commented 4 years ago

@alex4200 Ok for me my DEV notebook works again, but the neuron.hdf5 file provided in the storage is outdated and I still need to replace it. I can do that tonight.

alex4200 commented 4 years ago

@Helveg Thanks for the update.

Please keep me posted!

Helveg commented 4 years ago

@alex4200 The DEV notebook seems to be fully functional again!

alex4200 commented 4 years ago

Hi @Helveg, I tried the DEV notebook, but in the section "Spike Raster" after the simulation I got the following error:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-9-bbde19d5d963> in <module>()
      1 from scaffold.plotting import hdf5_plot_spike_raster
----> 2 from scaffold.output import latest_results
      3 import notebook_tools
      4 from h5py import File
      5 

ImportError: cannot import name 'latest_results'

Maybe the notebook uses a wrong version of scaffold? I also cannot see that module with pip list.

Helveg commented 4 years ago

Thank you for your extensive testing! The latest_results function was missing from scaffold/output.py, I've added it to notebook_tools.py and adjusted the notebook itself. The 3 bottom plots are now working again. Some deprecation warnings have been introduced because of outdated dependencies but it all works so let's ignore these ;)

PS: The reason why you didn't see it with pip list is because the package name is dbbs-scaffold.

alex4200 commented 4 years ago

@Helveg I got the same error again. Do we talk about the same notebook? The one I am using is located at Collab 79605 with the name Cerebellar scaffold multicompartmental simulation.ipynb.

Helveg commented 4 years ago

I'm not a 100% on how the virtualization of the Collab works, so your notebook might be in a different state than mine. We both access https://collab.humanbrainproject.eu/#/collab/79605/nav/540654 right?

alex4200 commented 4 years ago

@Helveg I have trouble running this notebook. I sometimes get random kernel failures, and then the network compile cell does not respond:

Screenshot 2020-05-27 at 16 39 16

I am not sure if the cell is running, or not running, or what.

I tried again, but I got a different error (not sure what).

So I did a reset of my Collab container to be sure everything is reset to the basis configuration and started over.

Then I got this error:


---------------------------------------------------------------------------
NeuronError                               Traceback (most recent call last)
/opt/conda/lib/python3.6/site-packages/glia/__init__.py in <module>()
     40         else:
---> 41             manager.start(load_dll=not bool(os.getenv("GLIA_NO_AUTOLOAD_DLL")))
     42 except GliaError as e:

/opt/conda/lib/python3.6/site-packages/glia/glia.py in start(self, load_dll)
     34         self.compile(check_cache=True)
---> 35         self.init_neuron(load_dll=load_dll)
     36 

/opt/conda/lib/python3.6/site-packages/glia/glia.py in init_neuron(self, load_dll)
     43         if load_dll:
---> 44             self.load_neuron_dll()
     45 

/opt/conda/lib/python3.6/site-packages/glia/glia.py in load_neuron_dll(self)
    319             if not self._dll_loaded:
--> 320                 raise NeuronError("Library could not be loaded into NEURON.")
    321         else:

NeuronError: Library could not be loaded into NEURON.

During handling of the above exception, another exception occurred:

NameError                                 Traceback (most recent call last)
<ipython-input-1-96b4575a4918> in <module>()
      5 get_ipython().system('{sys.executable} -m pip install rtree mpi4py')
      6 get_ipython().system('{sys.executable} -m pip install pynb-sim==0.2.0 dbbs-scaffold==3.2.12rc0 dbbs_models dbbs_mod_collection')
----> 7 import rtree, scaffold, dbbs_models
      8 finalize_packages()

/opt/conda/lib/python3.6/site-packages/dbbs_models/__init__.py in <module>()
      4 if not os.getenv("SETUPTOOLS_INSTALL"):
      5 
----> 6     from .granule_cell_models import GranuleCell
      7     from .stellate_cell_models import StellateCell
      8     from .basket_cell_models import BasketCell

/opt/conda/lib/python3.6/site-packages/dbbs_models/granule_cell_models.py in <module>()
      1 import numpy as np
      2 from patch import p
----> 3 from arborize import NeuronModel
      4 from math import floor
      5 

/opt/conda/lib/python3.6/site-packages/arborize/__init__.py in <module>()
----> 1 from .core import NeuronModel
      2 from .synapse import Synapse
      3 
      4 __version__ = "1.1.0"
      5 

/opt/conda/lib/python3.6/site-packages/arborize/core.py in <module>()
      7     from patch import p
      8     from patch.objects import Section
----> 9     import glia as g
     10     from .synapse import Synapse
     11     p.load_file('stdlib.hoc')

/opt/conda/lib/python3.6/site-packages/glia/__init__.py in <module>()
     42 except GliaError as e:
     43     print("GLIA ERROR", e)
---> 44     exit(1)
     45 
     46 

NameError: name 'exit' is not defined

Then I tried again and got the same error as above, where a cell is running or not running.

I cannot test your notebook.

I really suggest to either completely rework this notebook and to take out these init() and finalize() methods that no other notebook has s far as I know. Or you create a separate GUI outside of the Collab outside of the notebook.

But in the current form it neither runs nor cannot be tested.

I am open to have a skype/zoom chat about this.

Helveg commented 4 years ago

I am not sure if the cell is running, or not running, or what.

Right above the cell it mentions that the compilation process can take up to 10 minutes, the line below it it mentions that you can switch to another command to show more progress, and next to the cell is an asterisk indicating that the cell is still running, which will turn into a number when it is finished. As with all Jupyter notebooks.

I really suggest to either completely rework this notebook and to take out these init() and finalize() methods that no other notebook has s far as I know. Or you create a separate GUI outside of the Collab outside of the notebook.

The GUI is nothing more than some surface level Javascript being injected using the IPython.display method, it doesn't affect the python kernel.

NeuronError: Library could not be loaded into NEURON.

This suggests that something is going wrong with the compilation of the mod files, maybe because of the interruption of the first run?

I'm quite displeased with the virtualization on the Collab as the whole point of virtualization is to have a reproducible sandbox environment and this system offers none of that as the notebook is fully functional to me.

Helveg commented 4 years ago

These are the init and finalize functions:

def init():
    os.environ["LD_LIBRARY_PATH"] = (os.getenv("LD_LIBARY_PATH") or "") + ":/home/jovyan/bin/lib"
    os.environ["SPATIALINDEX_C_LIBRARY"] = "/home/jovyan/bin/lib/libspatialindex.so"
    sys.path.insert(0, "/home/jovyan/bin/lib")
    clear_output()
    display(HTML('<script src="https://kit.fontawesome.com/aceb3af2d4.js" crossorigin="anonymous"></script><i class="fas fa-spinner fa-spin"></i> Installing tools...'))

def finalize():
    clear_output()
    display(HTML("Environment loaded."))
    display(HTML("<script>Jupyter.notebook.kernel.restart()</script>"))
Helveg commented 4 years ago

Can you tell me how I could also completely reset my container as you did? Maybe there's some state left over from previous things I did.

alex4200 commented 4 years ago

You can write an email to support@humanbrainproject.eu, or way faster, you give me your User ID:

On the Collab on the top right you have a small icon (maybe a letter or a photo) to get to your profile, the on the left you have 'profile', and there you have the User ID

Helveg commented 4 years ago

UserID: 308446

alex4200 commented 4 years ago

@Helveg Your container should have been reset. Now you can try with a Collab 'from scratch'...

Helveg commented 4 years ago

@Helveg I did a rerun of the use case, but I ran into problems in the third code cell (notebook 79605/Cerebellar scaffold multicompartmental simulation.ipynb) with the following error:

I now get the same error after a container reset. I will try to fix it asap. I've made the other changes you suggested. (Kept the spinner ;) but removed the kernel restart)

Helveg commented 4 years ago

@alex4200 For some strange reason a kernel restart is required between installing a package and its dependencies before python's pkg_resources picks up on its installation. Therefore I moved the installation of the dependency to the codecell that is executed before the kernel restart that the user is prompted about.

I've tested all cells and they seem fine now! Give it a go with the latest version and let me know.

alex4200 commented 4 years ago

@Helveg I just finished testing the current DEV notebook, but I had problems with the installed version of numpy. The line from scaffold.core import from_hdf5 created an import error, was requiring numpy 1.13.3.

Maybe you can add that as a requirement in one of the cells before the kernel restart?

Helveg commented 4 years ago

Very strange, it seems to be alike to the previous error. My setup.py script states numpy>=1.16.4 as a dependency, and it gets installed. But like with the previous error on the first run of a container it seems to miss the freshly installed packages...

We would have to move the installation of the following packages into the pre-kernel-restart code cell:

        "h5py>=2.9.0",
        "numpy>=1.16.4",
        "scipy>=1.3.1",
        "scikit-learn>=0.20.3",
        "plotly>=4.1.0",
        "rtree-linux==0.9.4",
        "nrn-patch>=2.0.0",
Helveg commented 4 years ago

@alex4200 I've copied the pip install commands to the code cell before the kernel restart, is the problem solved now?

alex4200 commented 4 years ago

I am still getting the following error (for cell after Full network plot (only cell soma))"

/opt/conda/lib/python3.6/site-packages/scipy/__init__.py:144: UserWarning:

Numpy 1.13.3 or above is required for this version of scipy (detected version 1.11.2)

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
RuntimeError: module compiled against API version 0xb but this version of numpy is 0xa

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-4-a973940dbd8a> in <module>()
----> 1 from scaffold.core import from_hdf5
      2 from scaffold.plotting import plot_network
      3 scaffold = from_hdf5("neuron.hdf5")
      4 plot_network(scaffold, from_memory=False);

/opt/conda/lib/python3.6/site-packages/scaffold/core.py in <module>()

/opt/conda/lib/python3.6/site-packages/scaffold/core.py in <module>()

/opt/conda/lib/python3.6/site-packages/scaffold/plotting.py in <module>()

/opt/conda/lib/python3.6/site-packages/scaffold/plotting.py in <module>()

/opt/conda/lib/python3.6/site-packages/scaffold/morphologies.py in <module>()

/opt/conda/lib/python3.6/site-packages/scaffold/morphologies.py in <module>()

/opt/conda/lib/python3.6/site-packages/scaffold/voxels.py in <module>()

/opt/conda/lib/python3.6/site-packages/scaffold/voxels.py in <module>()

/opt/conda/lib/python3.6/site-packages/scipy/ndimage/__init__.py in <module>()
    151 from __future__ import division, print_function, absolute_import
    152 
--> 153 from .filters import *
    154 from .fourier import *
    155 from .interpolation import *

/opt/conda/lib/python3.6/site-packages/scipy/ndimage/filters.py in <module>()
     35 import operator
     36 from . import _ni_support
---> 37 from . import _nd_image
     38 from . import _ni_docstrings
     39 

ImportError: numpy.core.multiarray failed to import
Helveg commented 4 years ago

I have no idea why it would detect numpy 1.11.2 that seems like a BSP/Collab specific issue. What does pip show numpy tell you? Can you try pip install numpy==1.16.4, restart the kernel and try to run the cell again?

Update: It is for sure environment specific, for me all 3 of these code cells produce their plot. Can me/you try another container reset?

alex4200 commented 4 years ago

After a container restart I made the following observations:

Would it be possible to combine the two setup cells somehow, in order to do the pip installs only once and the kernel restart also only once?

Helveg commented 4 years ago

So the issue is probably that the fresh containers already have some (VERY OLD) python packages installed (and they even get imported when the kernel starts?)

I don't remember exactly why but there was also a strange reason why I seperated the 2 setup cells to begin with and wanted to be so absolutely sure the kernel was restarted in between. Numpy is very standard and might be the only affected "already installed" package, so if we explicitly install numpy 1.18.5 in the first setup cell the issue might already be resolved after the first kernel restart. Could you add pip install numpy 1.18.5 to the first setup cell in the DEV workbook and try?

And if another kernel restart is all it takes to fix these issues I'm sure we can put a note somewhere and just ignore this problem entirely.

alex4200 commented 4 years ago

Hi @Helveg, I might have found a procedure that should work:

  1. You need to add a new code cell between the first and the second code cell with the following content:
import sys
!pip3 install hbp_service_client --upgrade
!{sys.executable} -m pip install numpy==1.18.5

  1. Below that you have to give the hint to restart the kernel.

  2. Take out the 'hbp_service_client' install from the one code cell

  3. Then the user continues normally with the 3rd code cell (the previous 2nd code cell), including a second kernel restart after this cell.

Please see this notebook: https://collab.humanbrainproject.eu/#/collab/52393/nav/567347

But this procedure now seems to work just fine. Please give it a try.

Helveg commented 4 years ago

@alex4200 Done