container method not working.

[Sachuriga]

Hi,

The code are running well till today, but after I updaeted to 0.97.1version suddenly apears the error below, does anyone know what cause the problem?

Sampling frequency (Hz): 30000.0 Singularity: pulling image spikeinterface/kilosort3-compiled-base singularity pull --name kilosort3-compiled-base.sif docker://spikeinterface/kilosort3-compiled-base WARNING: DEPRECATED USAGE: Environment variable SINGULARITY_PULLFOLDER will not be supported in the future, use APPTAINER_PULLFOLDER instead INFO: Using cached SIF image WARNING: DEPRECATED USAGE: Environment variable SINGULARITY_PULLFOLDER will not be supported in the future, use APPTAINER_PULLFOLDER instead INFO: Using cached SIF image kilosort3-compiled-base.sif Starting container Installing spikeinterface==0.97.0 in spikeinterface/kilosort3-compiled-base Installing extra requirements: ['neo'] Running kilosort3 sorter inside spikeinterface/kilosort3-compiled-base Stopping container Traceback (most recent call last): File "/cluster/home/sachur/Overlap_project/code/Python/OpenEphysForm/ks3_f_probe_20230508_Backup.py", line 148, in sorting = ss.run_kilosort3( File "/cluster/home/sachur/.local/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py", line 691, in run_kilosort3 return run_sorter('kilosort3', *args, **kwargs) File "/cluster/home/sachur/.local/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py", line 137, in run_sorter return run_sorter_container( File "/cluster/home/sachur/.local/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py", line 578, in run_sorter_container raise SpikeSortingError( spikeinterface.sorters.utils.misc.SpikeSortingError: Spike sorting in singularity failed with the following error: Traceback (most recent call last): File "/cluster/home/sachur/Overlap_project/SpkSorting/RawDate/in_container_sorter_script.py", line 3, in from spikeinterface import load_extractor ModuleNotFoundError: No module named 'spikeinterface'

[alejoe91]

Hi, did it work before?

The problem seems to be that spikeinterface fails to install inside the container.. If you installed version 0.97.1, the same version should be installed in the container, but you can see from the log that version 0.97.0 is attempted to be installed. Not sure why!

Did you also updgrade singularity? Can you share the entire code to reproduice the issue?

[DradeAW]

If I remember correctly, I had the same issue with an older version of Singularity.

It disappeared when I installed the latest version3

[Sachuriga]

hi

Thanks all, I upgraded the singularity, but the problem is still there. Below is my environment. it based on the computer group running on Linux system. And I will phaste the code in other comment.

absl-py @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/abslpy/absl-py-1.4.0 alabaster @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/alabaster/alabaster-0.7.12 appdirs @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/appdirs/appdirs-1.4.4 asciitree==0.3.3 asn1crypto @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/asn1crypto/asn1crypto-1.5.1 astor @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/astor/astor-0.8.1 astunparse @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/astunparse/astunparse-1.6.3 atomicwrites @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/atomicwrites/atomicwrites-1.4.0 attrs @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/attrs/attrs-21.4.0 Babel @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/Babel/Babel-2.10.1 backports.entry-points-selectable @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/backportsentry_points_selectable/backports.entry_points_selectable-1.1.1 backports.functools-lru-cache @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/backportsfunctools_lru_cache/backports.functools_lru_cache-1.6.4 bcrypt @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/bcrypt/bcrypt-3.2.2 beniget @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/beniget/beniget-0.3.0 bitstring @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/bitstring/bitstring-3.1.9 blist @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/blist/blist-1.3.6 boto3==1.26.137 botocore==1.29.137 Bottleneck @ file:///tmp/eb_pavlokh/SciPybundle/2022.05/foss-2022a/Bottleneck/Bottleneck-1.3.4 CacheControl @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/CacheControl/CacheControl-0.12.11 cachetools @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/cachetools/cachetools-5.2.1 cachy @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/cachy/cachy-0.3.0 certifi @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/certifi/certifi-2021.10.8 cffi @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/cffi/cffi-1.15.0 chardet @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/chardet/chardet-4.0.0 charset-normalizer @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/charsetnormalizer/charset-normalizer-2.0.12 cleo @ file:///cluster/apps/eb/sources/p/Python/extensions/cleo-0.8.1-py2.py3-none-any.whl click @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/click/click-8.1.3 clikit @ file:///cluster/apps/eb/sources/p/Python/extensions/clikit-0.6.2-py2.py3-none-any.whl cloudpickle==2.2.1 colorama @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/colorama/colorama-0.4.4 colorlog==6.7.0 contourpy==1.0.7 crashtest @ file:///cluster/apps/eb/sources/p/Python/extensions/crashtest-0.3.1-py3-none-any.whl cryptography @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/cryptography/cryptography-37.0.1 cuda-python==12.1.0 cupy==12.0.0 cycler==0.11.0 Cython @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/Cython/Cython-0.29.28 dask==2023.5.0 deap @ file:///tmp/eb_pavlokh/SciPybundle/2022.05/foss-2022a/deap/deap-1.3.1 decorator @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/decorator/decorator-5.1.1 dill @ file:///tmp/eb_pavlokh/dill/0.3.6/GCCcore-11.3.0/dill-0.3.6 distinctipy==1.2.2 distlib @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/distlib/distlib-0.3.4 docopt @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/docopt/docopt-0.6.2 docutils @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/docutils/docutils-0.17.1 ecdsa @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/ecdsa/ecdsa-0.17.0 editables @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/editables/editables-0.3 entrypoints==0.4 fasteners==0.18 fastrlock==0.8.1 filelock @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/filelock/filelock-3.6.0 flake8==6.0.0 flatbuffers @ file:///tmp/eb_pavlokh/flatbuffers/2.0.7/GCCcore-11.3.0/flatbuffers/flatbuffers-2.0.7 flit @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/flit/flit-3.7.1 flit_core @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/flitcore/flit_core-3.7.1 fonttools==4.39.4 fsspec @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/fsspec/fsspec-2022.3.0 future @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/future/future-0.18.2 gast @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/gast/gast-0.4.0 glob2 @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/glob2/glob2-0.7 google-auth @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/googleauth/google-auth-2.16.0 google-auth-oauthlib @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/googleauthoauthlib/google-auth-oauthlib-0.4.6 google-pasta @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/googlepasta/google-pasta-0.2.0 grpcio @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/grpcio/grpcio-1.51.1 gviz-api @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/gvizapi/gviz_api-1.10.0 h5py @ file:///tmp/eb_pavlokh/h5py/3.7.0/foss-2022a/h5py-3.7.0 herdingspikes==0.3.102 html5lib @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/html5lib/html5lib-1.1 ibl-neuropixel==0.5.3 iblutil==1.6.0 idna @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/idna/idna-3.3 imagesize @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/imagesize/imagesize-1.3.0 importlib-metadata==6.6.0 importlib-resources @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/importlib_resources/importlib_resources-5.7.1 iniconfig @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/iniconfig/iniconfig-1.1.1 intervaltree @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/intervaltree/intervaltree-3.1.0 intreehooks @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/intreehooks/intreehooks-1.0 ipaddress @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/ipaddress/ipaddress-1.0.23 jeepney @ file:///cluster/apps/eb/sources/p/Python/extensions/jeepney-0.8.0-py3-none-any.whl Jinja2 @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/Jinja2/Jinja2-3.1.2 jmespath==1.0.1 joblib==1.2.0 jsonschema @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/jsonschema/jsonschema-4.4.0 keras @ file:///cluster/apps/eb/sources/t/TensorFlow/extensions/keras-2.11.0-py2.py3-none-any.whl keyring @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/keyring/keyring-23.5.0 keyrings.alt @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/keyringsalt/keyrings.alt-4.1.0 kiwisolver==1.4.4 liac-arff @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/liacarff/liac-arff-2.5.0 llvmlite==0.40.0 locket==1.0.0 lockfile @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/lockfile/lockfile-0.12.2 Markdown @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/Markdown/Markdown-3.4.1 MarkupSafe @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/MarkupSafe/MarkupSafe-2.1.1 matplotlib==3.7.1 mccabe==0.7.0 mock @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/mock/mock-4.0.3 more-itertools @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/moreitertools/more-itertools-8.12.0 mpi4py @ file:///tmp/eb_pavlokh/SciPybundle/2022.05/foss-2022a/mpi4py/mpi4py-3.1.3 mpmath @ file:///tmp/eb_pavlokh/SciPybundle/2022.05/foss-2022a/mpmath/mpmath-1.2.1 msgpack @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/msgpack/msgpack-1.0.3 mtscomp==1.0.2 neo==0.12.0 netaddr @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/netaddr/netaddr-0.8.0 netifaces @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/netifaces/netifaces-0.11.0 networkx @ file:///tmp/eb_pavlokh/networkx/2.8.4/foss-2022a/networkx-2.8.4 numba==0.57.0 numcodecs==0.11.0 numexpr @ file:///tmp/eb_pavlokh/SciPybundle/2022.05/foss-2022a/numexpr/numexpr-2.8.1 numpy @ file:///tmp/eb_pavlokh/SciPybundle/2022.05/foss-2022a/numpy/numpy-1.22.3 oauthlib @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/oauthlib/oauthlib-3.2.2 ONE-api==2.0.0 opt-einsum @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/opteinsum/opt_einsum-3.3.0 packaging==23.1 pandas @ file:///tmp/eb_pavlokh/SciPybundle/2022.05/foss-2022a/pandas/pandas-1.4.2 paramiko @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/paramiko/paramiko-2.10.4 partd==1.4.0 pastel @ file:///cluster/apps/eb/sources/p/Python/extensions/pastel-0.2.1-py2.py3-none-any.whl pathlib2 @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/pathlib2/pathlib2-2.3.7.post1 pathspec @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/pathspec/pathspec-0.9.0 pbr @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/pbr/pbr-5.8.1 pexpect @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/pexpect/pexpect-4.8.0 phylib==2.4.3 Pillow==9.5.0 pkginfo @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/pkginfo/pkginfo-1.8.2 platformdirs @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/platformdirs/platformdirs-2.4.1 pluggy @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/pluggy/pluggy-1.0.0 ply @ file:///tmp/eb_pavlokh/SciPybundle/2022.05/foss-2022a/ply/ply-3.11 poetry @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/poetry/poetry-1.1.13 poetry-core @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/poetrycore/poetry-core-1.0.8 portpicker @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/portpicker/portpicker-1.5.2 probeinterface==0.2.16 protobuf @ file:///tmp/eb_pavlokh/protobufpython/3.19.4/GCCcore-11.3.0/protobuf-3.19.4 psutil @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/psutil/psutil-5.9.0 ptyprocess @ file:///cluster/apps/eb/sources/p/Python/extensions/ptyprocess-0.7.0-py2.py3-none-any.whl py @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/py/py-1.11.0 py-expression-eval @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/py_expression_eval/py_expression_eval-0.3.14 pyarrow==12.0.0 pyasn1 @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/pyasn1/pyasn1-0.4.8 pyasn1-modules @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/pyasn1modules/pyasn1-modules-0.2.8 pybind11 @ file:///tmp/eb_hrn/pybind11/2.9.2/GCCcore-11.3.0/pybind11-2.9.2 pycodestyle==2.10.0 pycparser @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/pycparser/pycparser-2.21 pycrypto @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/pycrypto/pycrypto-2.6.1 pydantic==1.10.7 pyflakes==3.0.1 Pygments @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/Pygments/Pygments-2.12.0 pykilosort===ibl-1.4.2 pylev @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/pylev/pylev-1.4.0 PyNaCl @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/PyNaCl/PyNaCl-1.5.0 pypandoc==1.11 pyparsing @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/pyparsing/pyparsing-3.0.8 pyrsistent @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/pyrsistent/pyrsistent-0.18.1 pytest @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/pytest/pytest-7.1.2 python-dateutil @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/pythondateutil/python-dateutil-2.8.2 pythran @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/pythran/pythran-0.9.11 pytoml @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/pytoml/pytoml-0.1.21 pytz @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/pytz/pytz-2022.1 PyYAML==6.0 quantities==0.14.1 regex @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/regex/regex-2022.4.24 requests @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/requests/requests-2.27.1 requests-oauthlib @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/requestsoauthlib/requests-oauthlib-1.3.1 requests-toolbelt @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/requeststoolbelt/requests-toolbelt-0.9.1 retrying==1.3.4 rsa @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/rsa/rsa-4.9 s3transfer==0.6.1 scandir @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/scandir/scandir-1.10.0 scikit-learn==1.2.2 SciPy @ file:///tmp/eb_pavlokh/SciPybundle/2022.05/foss-2022a/scipy/scipy-1.8.1 SecretStorage @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/SecretStorage/SecretStorage-3.3.2 semantic-version @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/semantic_version/semantic_version-2.9.0 setuptools-rust @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/setuptoolsrust/setuptools-rust-1.3.0 setuptools-scm @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/setuptools_scm/setuptools_scm-6.4.2 shellingham @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/shellingham/shellingham-1.4.0 simplegeneric @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/simplegeneric/simplegeneric-0.8.1 simplejson @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/simplejson/simplejson-3.17.6 singularity==3.0.0 six @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/six/six-1.16.0 sklearn==0.0.post5 snowballstemmer @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/snowballstemmer/snowballstemmer-2.2.0 sortedcontainers @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/sortedcontainers/sortedcontainers-2.4.0 Sphinx @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/Sphinx/Sphinx-4.5.0 sphinx-bootstrap-theme @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/sphinxbootstraptheme/sphinx-bootstrap-theme-0.8.1 sphinxcontrib-applehelp @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/sphinxcontribapplehelp/sphinxcontrib-applehelp-1.0.2 sphinxcontrib-devhelp @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/sphinxcontribdevhelp/sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/sphinxcontribhtmlhelp/sphinxcontrib-htmlhelp-2.0.0 sphinxcontrib-jsmath @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/sphinxcontribjsmath/sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/sphinxcontribqthelp/sphinxcontrib-qthelp-1.0.3 sphinxcontrib-serializinghtml @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/sphinxcontribserializinghtml/sphinxcontrib-serializinghtml-1.1.5 sphinxcontrib-websupport @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/sphinxcontribwebsupport/sphinxcontrib-websupport-1.2.4 spikeinterface==0.97.1 spython==0.3.0 tabulate @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/tabulate/tabulate-0.8.9 tblib @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/tblib/tblib-1.7.0 tensorboard @ file:///cluster/apps/eb/sources/t/TensorFlow/extensions/tensorboard-2.11.1-py3-none-any.whl tensorboard-data-server @ file:///cluster/apps/eb/sources/t/TensorFlow/extensions/tensorboard_data_server-0.6.1-py3-none-any.whl tensorboard-plugin-profile @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/tensorboardpluginprofile/tensorboard_plugin_profile-2.11.1 tensorboard-plugin-wit @ file:///cluster/apps/eb/sources/t/TensorFlow/extensions/tensorboard_plugin_wit-1.8.1-py3-none-any.whl tensorflow @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/tensorflow-2.11.0-cp310-cp310-linux_x86_64.whl tensorflow-estimator @ file:///cluster/apps/eb/sources/t/TensorFlow/extensions/tensorflow_estimator-2.11.0-py2.py3-none-any.whl termcolor @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/termcolor/termcolor-1.1.0 threadpoolctl @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/threadpoolctl/threadpoolctl-3.1.0 toml @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/toml/toml-0.10.2 tomli @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/tomli/tomli-2.0.1 tomli_w @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/tomli_w/tomli_w-1.0.0 tomlkit @ file:///cluster/apps/eb/sources/p/Python/extensions/tomlkit-0.10.2-py3-none-any.whl toolz==0.12.0 tqdm==4.65.0 typing_extensions @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/typing_extensions/typing_extensions-4.2.0 tzdata==2023.3 ujson @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/ujson/ujson-5.2.0 urllib3 @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/urllib3/urllib3-1.26.9 virtualenv @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/virtualenv/virtualenv-20.14.1 wcwidth @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/wcwidth/wcwidth-0.2.5 webencodings @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/webencodings/webencodings-0.5.1 Werkzeug @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/Werkzeug/Werkzeug-2.2.2 wrapt @ file:///tmp/eb_pavlokh/TensorFlow/2.11.0/foss-2022a/wrapt/wrapt-1.14.1 xarray==2022.9.0 xlrd @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/xlrd/xlrd-2.0.1 zarr==2.14.2 zipfile36 @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/zipfile36/zipfile36-0.1.3 zipp @ file:///cluster/work/hrn/build/Python/3.10.4/GCCcore-11.3.0/zipp/zipp-3.8.0

[Sachuriga]

And here is the code python code.

import spikeinterface as si import spikeinterface.extractors as se import spikeinterface.preprocessing as spre import spikeinterface.sorters as ss import spikeinterface.postprocessing as spost import spikeinterface.qualitymetrics as sqm import spikeinterface.comparison as sc import spikeinterface.exporters as sexp import spikeinterface.widgets as sw

from spikeinterface.preprocessing import (bandpass_filter, notch_filter, common_reference, remove_artifacts, preprocesser_dict)

import numpy as np import os from pathlib import Path

import warnings warnings.simplefilter("ignore")

def current_path(): print("Current working directory before") print(os.getcwd()) print()

os.chdir('/cluster/home/sachur/Overlap_project/SpkSorting/RawDate') current_path()

num_channels = 64

sampling_frequency = 20000

gain_to_uV = 0.05 offset_to_uV = 0

dtype="int16"

time_axis = 1

base_folder = Path(".") OpenEphys_file_path = "2023-05-11_18-23-06" current_file_name_path = os.path.basename(OpenEphys_file_path) current_file_name = str(current_file_name_path)

stream_name='Record Node 106#Acquisition_Board-100.Rhythm Data' recording = se.read_openephys(OpenEphys_file_path,stream_name=stream_name)

os.chdir('/cluster/home/sachur/Overlap_project/SpkSorting/RawDate')

recording.annotate(is_filtered=False)

channel_ids = recording.get_channel_ids() fs = recording.get_sampling_frequency() num_chan = recording.get_num_channels() num_segments = recording.get_num_segments()

print(f'Channel ids: {channel_ids}') print(f'Sampling frequency: {fs}') print(f'Number of channels: {num_chan}') print(f"Number of segments: {num_segments}") trace_snippet = recording.get_traces(start_frame=int(fs0), end_frame=int(fs2)) print('Traces shape:', trace_snippet.shape)

import probeinterface as pi

from probeinterface import plotting

manufacturer = 'cambridgeneurotech' probe_name = 'ASSY-236-F'

probe = pi.get_probe(manufacturer, probe_name) print(probe)

probe.wiring_to_device('cambridgeneurotech_mini-amp-64')

map channels to device indices

mapping_to_device = [

connector J2 TOP

    41, 39, 38, 37, 35, 34, 33, 32, 29, 30, 28, 26, 25, 24, 22, 20,
    46, 45, 44, 43, 42, 40, 36, 31, 27, 23, 21, 18, 19, 17, 16, 14,
    # connector J1 BOTTOM
    55, 53, 54, 52, 51, 50, 49, 48, 47, 15, 13, 12, 11, 9, 10, 8,
    63, 62, 61, 60, 59, 58, 57, 56, 7, 6, 5, 4, 3, 2, 1, 0

]

probe.set_device_channel_indices(mapping_to_device) probe.to_dataframe(complete=True).loc[:, ["contact_ids", "shank_ids","device_channel_indices"]] probegroup=pi.ProbeGroup() probegroup.add_probe(probe)

pi.write_prb(f"{probe_name}.prb",probegroup,group_mode="by_shank")

recording_prb=recording recording_prb = recording.set_probe(probe, group_mode="by_shank")

print(f'Original channels:{recording.get_channel_ids()}') print(f'Channels after loading the probe file: {recording_prb.get_channel_ids()}') print(f'Channel groups after loading the probe file: {recording_prb.get_channel_groups()}') print("Properties before loading the probe:", list(recording.get_property_keys())) print("Properties after loading the probe:", list(recording_prb.get_property_keys()))

brain_area_property_values = ['CA1']32 + ['CA3']32

print(brain_area_property_values)

recording_prb.set_property(key='brain_area', values=brain_area_property_values)

recording_prb.set_property(key='quality', values=["good"]*(recording_prb.get_num_channels() - 3), ids=recording_prb.get_channel_ids()[:-3])

recording_prb.get_property("quality") recording_prb.annotate(description="Sachuriga_test1") print(recording_prb.get_annotation_keys())

recording_cmr=recording_prb recording_f = bandpass_filter(recording_prb, freq_min=300, freq_max=6000)

recording_cmr = common_reference(recording_f, reference='global', operator='median')

fs = recording_cmr.get_sampling_frequency()

recording_sub = recording_cmr job_kwargs = dict(n_jobs=10, chunk_duration="1s", progress_bar=True)

preprocessed=current_filename + "" + "preprocessed" recording_saved = recording_sub.save(folder=base_folder / preprocessed, **job_kwargs)

print(f'Cached channels ids: {recording_saved.get_channel_ids()}') print(f'Channel groups after caching: {recording_saved.get_channel_groups()}')

recording_loaded = si.load_extractor(base_folder / preprocessed)

print("Properties after adding custom properties:", list(recording_prb.get_property_keys())) print(recording_prb._properties.keys())

print(recording_f) print("Duration (minutes):", recording_f.get_total_duration() / 60) print("Num. channels:", recording_f.get_num_channels()) print("Sampling frequency (Hz):", recording_f.get_sampling_frequency())

sorter_params = job_kwargs kilosort3=current_filename + "" + "kilosort3"

sorter_params = dict(n_jobs=12, chunk_duration="1s", progress_bar=True, minFR=0.1, minfr_goodchannels=0.1 )

sorting=ss.run_sorter(sorter_name="tridesclous", recording=recording_saved, detect_threshold=4)

sorting = ss.run_kilosort3(recording=recording_saved, output_folder=base_folder / kilosort3, verbose=True, singularity_image=True, **sorter_params)

[Sachuriga]

Additionally this should be the original error, I installed cuda then it changed to error above.

Traceback (most recent call last): File "/cluster/home/sachur/Overlap_project/code/Python/OpenEphysForm/ks3_f_probe_20230522_expand_edition.py", line 149, in sorting = ss.run_sorter_by_property('kilosort3', recording=recording_saved, grouping_property='group',working_folder=base_folder / kilosort3, mode_if_folder_exists='keep', singularity_image=True) File "/cluster/apps/eb/software/Python/3.9.5-GCCcore-10.3.0/lib/python3.9/site-packages/spikeinterface/sorters/launcher.py", line 119, in run_sorter_by_property sorting_output = run_sorters([sorter_name], recording_dict, working_folder, File "/cluster/apps/eb/software/Python/3.9.5-GCCcore-10.3.0/lib/python3.9/site-packages/spikeinterface/sorters/launcher.py", line 269, in run_sorters _run_one(task_args) File "/cluster/apps/eb/software/Python/3.9.5-GCCcore-10.3.0/lib/python3.9/site-packages/spikeinterface/sorters/launcher.py", line 38, in _run_one run_sorter(sorter_name, recording, output_folder=output_folder, File "/cluster/apps/eb/software/Python/3.9.5-GCCcore-10.3.0/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py", line 137, in run_sorter return run_sorter_container( File "/cluster/apps/eb/software/Python/3.9.5-GCCcore-10.3.0/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py", line 453, in run_sorter_container assert has_nvidia(), "The container requires a NVIDIA GPU capability, but it is not available" File "/cluster/apps/eb/software/Python/3.9.5-GCCcore-10.3.0/lib/python3.9/site-packages/spikeinterface/sorters/utils/misc.py", line 65, in has_nvidia from cuda import cuda ModuleNotFoundError: No module named 'cuda'

[Sachuriga]

The problem might be due to singularity is not excluding the home lib, thus the all the thing in the container was shaded by that. yet I don't have any solution to solve this. Would somebody have any idea?

[alejoe91]

I see, I have mainly used Docker, but I think that @DradeAW had similar issues. Maybe he could help?

[ghost]

Did anyone make any progress on this issue? I'm getting the exact same thing - trying to run any sorter in a singularity container returns a ModuleNotFoundError

[alejoe91]

Can you paste the full error?

[alejoe91]

@sachuriga283 looking again, it might be fixed with a simple: pip install cuda-python

[DradeAW]

The fix for me was to install the latest version of Go (and not the one recommended in the documentation!) Then install the latest version of singularity Then install singularity in the Python environment. Then installing cuda-python (for cuda-based sorters)

[Sachuriga]

Thanks all, I will try and report back.

[ghost]

Here's my full error:

SpikeSortingError Traceback (most recent call last) Cell In[22], line 1 ----> 1 sorting = si.run_sorter('kilosort3',rec_clean,output_folder='kilosort3',verbose=True,**sorter_params,singularity_image='/public/singularity/containers/spikesorters/kilosort3.sif')

File ~/local/mamba/envs/si_env/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py:137, in run_sorter(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, with_output, sorter_params) 135 else: 136 container_image = singularity_image --> 137 return run_sorter_container( 138 container_image=container_image, 139 mode=mode, 140 common_kwargs, 141 ) 143 return run_sorter_local(**common_kwargs)

File ~/local/mamba/envs/si_env/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py:583, in run_sorter_container(sorter_name, recording, mode, container_image, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, extra_requirements, **sorter_params) 581 if run_error: 582 if raise_error: --> 583 raise SpikeSortingError( 584 f"Spike sorting in {mode} failed with the following error:\n{run_sorter_output}") 585 else: 586 if with_output:

SpikeSortingError: Spike sorting in singularity failed with the following error: Traceback (most recent call last): File "/lmb/home/rbedford/Documents/testsort/in_container_sorter_script.py", line 3, in from spikeinterface import load_extractor ModuleNotFoundError: No module named 'spikeinterface'

[alejoe91]

Thanks, so something is wrong when it attempts to install spikeinterface in the container. What is the output before the error? Also, what version of spikeinterface and singularity are you using? We had some troubles with older versions of singularity

[DradeAW]

Did you set the environment variable SPIKEINTERFACE_DEV_PATH?

I had the same problem until I set that variable to point to the repo :)

[ghost]

output before the error:

Starting container Installing spikeinterface==0.97.1 in /public/singularity/containers/spikesorters/kilosort3.sif Installing extra requirements: ['neo'] Running kilosort3 sorter inside /public/singularity/containers/spikesorters/kilosort3.sif Stopping container

I'm also running spikeinterface==0.97.1 on my local machine in a conda environment

Singularity version: apptainer version 1.1.7-1.el8

I appreciate all the help and the quick replies. I'm in an institution where I can't control the version of singularity being used so if that is the issue I will need to chat to my sysadmin.

[DradeAW]

I think you have the wrong version of Singularity. Here is my output:

$ singularity --version
singularity-ce version 3.11.1

[ghost]

I believe sylabs' singularity and the linux foundation's apptainer are meant to be fully compatible

[ghost]

To follow this up: I've spent quite a while troubleshooting this now and have found a couple sources for the error. If I enter the container myself and run:

'pip install --upgrade --no-input spikeinterface[full]==0.97.1'

I get:

'ERROR: Could not install packages due to an EnvironmentError: [Errno 30] Read-only file system'

So it definitely seems like it's failing to install spikeinterface from within the container. I will chat to my sysadmin about our version of singularity and how to allow the container to install software.

Secondly I previously had an issue where the container was finding my local installs of python in my home directory, and I think it was confusing this with its own python installation. I had some conda environments stored under $HOME/local, and changed this to $HOME/.local, and it doesn't seem to find them under the hidden directory. Thought I'd mention this just so people know having conda environments stored like this could cause the container to not use its own python.

An honest question here, just as I'm fairly new to using containers: my understanding is that containers should include all the necessary dependencies for an application pre-installed and ready to run. Why is it then that all the spikeinterface sorting containers on dockerhub don't include spikeinterface preinstalled? Would it be that tricky to have si_env included in the image? Is this so that the latest updates can be installed without having to reupload the docker image? Forgive me for the newbie question, it just seems to me that having a container ready to go as is is kind of the purpose of using containers in the first place.

[alejoe91]

An honest question here, just as I'm fairly new to using containers: my understanding is that containers should include all the necessary dependencies for an application pre-installed and ready to run. Why is it then that all the spikeinterface sorting containers on dockerhub don't include spikeinterface preinstalled? Would it be that tricky to have si_env included in the image? Is this so that the latest updates can be installed without having to reupload the docker image? Forgive me for the newbie question, it just seems to me that having a container ready to go as is is kind of the purpose of using containers in the first place.

Very honest question! Yes indeed the reason is to decouple the development of individual sorters (each image should "fix" a sorter) to the development of SpikeInterface. SpikeInterface is still slowly changing, so if we installed it directly in the images we would need to update them quite often. Instead we prefer to install the current SI version in the container.

[ghost]

That makes sense thanks for explaining!

Just wanted to add for now I'm using the following workaround: I'm building the container using apptainer with the following .def file (substitute any sorter):

Bootstrap: docker
From: spikeinterface/klusta-base:latest

%post
pip install spikeinterface[full]==0.95.1

This works well and I guess you could just update the definition and rebuild whenever you want a new version of spikeinterface. Hope this helps anyone else who runs into the same issue.

[alejoe91]

Yep that sounds totally reasonable! Note that klusta unfortunately doesn't work with SI>=0.96 because it requires python 3.7! (I see you figured it out, just wanted to explain why ;))

[alejoe91]

Should be fixed by #2053

(Thanks to @mhhennig)