Error running SpykingCircus2 clustering

[Antho2422]

Hi @yger,

I have a question about the clustering process in spyking circus. I noticed that sometimes when I run the sorter on small periods of data or channels with a few peaks I get the following error. I assumed that it was coming from a lack of spikes to perform a clustering but I would like to make sure it is not coming from something else.

Thank you, Anthony

[yger]

If this is a small recording, can you share it with me (alongside with the analysis script)?

[Antho2422]

Of course @yger , here is a link to download the dataset : https://instituteicm-my.sharepoint.com/:f:/g/personal/anthony_pinto_icm-institute_org/EpiPint_OABJiQNFPG2Q-44B7cmBfe_iV1CJOI_RrHLZKg?e=XCZgWL

It is already filtered using a bandpass filter between 300 and 6000Hz with dtype64 so I did not put apply preprocessing in the run_sorter function.

The sorting is integrated in a pipeline so it is difficult to give you the full code but I made the following code that is replicating the process to reproduce the error. Here is the code :

from spikeinterface.sorters import run_sorter
import spikeinterface as si

recording = si.load_extractor(r'C:\Users\anthony.pinto\Documents\dataset_fail_clustering\recording.zarr')
params = si.sorters.get_default_sorter_params('spykingcircus2')
params['waveforms']['max_spikes_per_unit'] = 500
params['detection']['peak_sign'] = 'pos'
params['detection']['detect_threshold'] = 3.5779258099871694
params['apply_preprocessing'] = False
sorting = run_sorter('spykingcircus2', recording, output_folder='test_run_sorting', **params)

The weird value of the detection threshold is determined automatically so don't mind it. I didn't tweak the other parameters appart from those in the above code, maybe I should but I'm not sure of what each parameter is doing so that is why I didn't change a lot of the params.

Thank you, Anthony

[yger]

So on my machine, with the main branch, the code is working fine. This is both a good and a bad news :-( Can you tell me the version of hdbscan that you have?

[Antho2422]

Argh, yes ofc here is my hdbscan version :

Note also that I get this working with both Windows and Linux.

[yger]

When you say "you get this working", you mean that the code is crashing? I have the sam hdbscan version, so maybe send me the full version for the envs (pip list)

[Antho2422]

Yes sorry for the formulation, I mean the code crashes.

Here is my pip list :

Package                   Version      Editable project location
------------------------- ------------ ------------------------------------------
aiohttp                   3.9.1
aiohttp-retry             2.8.3
aiosignal                 1.3.1
altair                    5.0.1
amqp                      5.2.0
annotated-types           0.6.0
antlr4-python3-runtime    4.9.3
antropy                   0.1.5
anyio                     4.1.0
appdirs                   1.4.4
argon2-cffi               23.1.0
argon2-cffi-bindings      21.2.0
arrow                     1.3.0
asciitree                 0.3.3
asttokens                 2.4.1
async-timeout             4.0.3
asyncssh                  2.14.1
atomicwrites              1.4.1
atpublic                  4.0
attrs                     23.1.0
audioread                 3.0.1
autoreject                0.4.3
backcall                  0.2.0
beautifulsoup4            4.12.2
billiard                  4.2.0
black                     23.11.0
bleach                    6.1.0
cachetools                5.3.2
cbor2                     5.4.6
celery                    5.3.6
certifi                   2023.11.17
cffi                      1.16.0
cfgv                      3.4.0
chardet                   5.2.0
charset-normalizer        3.3.2
click                     8.1.7
click-didyoumean          0.3.0
click-plugins             1.1.1
click-repl                0.3.0
cloudpickle               3.0.0
colorama                  0.4.6
colorcet                  3.0.1
coloredlogs               15.0.1
comm                      0.2.0
configobj                 5.0.8
contourpy                 1.2.0
coverage                  7.3.2
cryptography              41.0.7
cycler                    0.12.1
Cython                    0.29.36
dacite                    1.8.1
dask                      2023.11.0
debugpy                   1.8.0
decorator                 5.1.1
defusedxml                0.7.1
dictdiffer                0.9.0
diskcache                 5.6.3
distlib                   0.3.7
distributed               2023.5.1
distro                    1.8.0
dpath                     2.1.6
dulwich                   0.21.6
dvc                       3.33.4
dvc-data                  2.24.0
dvc-http                  2.30.2
dvc-objects               2.0.1
dvc-render                1.0.0
dvc-studio-client         0.18.0
dvc-task                  0.3.0
dvclive                   3.5.1
EDFlib-Python             1.0.7
elephant                  0.12.0
entrypoints               0.4
exceptiongroup            1.2.0
executing                 2.0.1
fasteners                 0.19
fastjsonschema            2.19.0
figurl                    0.2.18
filelock                  3.13.1
flake8                    4.0.1
flakeheaven               3.3.0
flatbuffers               23.5.26
flatten-dict              0.4.2
flufl.lock                7.1.1
fonttools                 4.45.1
fqdn                      1.5.1
frozenlist                1.4.0
fsspec                    2023.10.0
funcy                     2.0
ghp-import                2.1.0
gitdb                     4.0.11
GitPython                 3.1.40
grandalf                  0.8
gto                       1.4.0
h5io                      0.1.9
h5py                      3.10.0
hdbscan                   0.8.33
hdmf                      3.6.1
htmlmin                   0.1.12
humanfriendly             10.0
hydra-core                1.3.2
identify                  2.5.32
idna                      3.6
ImageHash                 4.3.1
importlib-metadata        6.8.0
importlib-resources       6.1.1
iniconfig                 2.0.0
ipykernel                 6.27.1
ipython                   8.18.1
ipython-genutils          0.2.0
ipywidgets                8.1.1
isoduration               20.11.0
iterative-telemetry       0.0.8
jedi                      0.19.1
Jinja2                    3.1.2
joblib                    1.3.2
jsonpointer               2.4
jsonschema                4.20.0
jsonschema-specifications 2023.11.1
jupyter_client            7.4.9
jupyter_core              5.5.0
jupyter-events            0.9.0
jupyter_server            2.11.1
jupyter_server_terminals  0.4.4
jupyterlab_pygments       0.3.0
jupyterlab-widgets        3.0.9
kachery-cloud             0.4.2
kiwisolver                1.4.5
kombu                     5.3.4
lazy_loader               0.3
librosa                   0.10.1
lightgbm                  4.1.0
littleutils               0.2.2
llvmlite                  0.39.1
locket                    1.0.0
lspopt                    1.3.0
Mako                      1.3.0
Markdown                  3.5.1
markdown-it-py            3.0.0
MarkupSafe                2.1.3
mat73                     0.60
matplotlib                3.8.2
matplotlib-inline         0.1.6
mccabe                    0.6.1
mdurl                     0.1.2
mergedeep                 1.3.4
mistune                   3.0.2
mkdocs                    1.5.3
mne                       1.6.0
mne-features              0.2.1
mpmath                    1.3.0
msgpack                   1.0.7
mtscomp                   1.0.2
multidict                 6.0.4
multimethod               1.10
mypy                      1.7.1
mypy-extensions           1.0.0
nanotime                  0.5.2
nbclassic                 1.0.0
nbclient                  0.9.0
nbconvert                 7.11.0
nbformat                  5.9.2
neo                       0.12.0
nest-asyncio              1.5.8
networkx                  3.2.1
nodeenv                   1.8.0
notebook                  6.5.6
notebook_shim             0.2.3
numba                     0.56.4
numcodecs                 0.12.1
numexpr                   2.8.7
numpy                     1.23.5
omegaconf                 2.3.0
onnx                      1.15.0
onnxconverter-common      1.13.0
onnxmltools               1.11.2
onnxruntime               1.16.1
orjson                    3.9.10
outdated                  0.2.2
overrides                 7.4.0
packaging                 23.2
pandas                    2.1.3
pandocfilters             1.5.0
param                     2.0.1
parso                     0.8.3
partd                     1.4.1
pathspec                  0.11.2
patsy                     0.5.3
pdoc3                     0.10.0
phik                      0.12.3
phy                       2.0b5
phylib                    2.4.3
pickleshare               0.7.5
Pillow                    9.5.0
pip                       23.1.2
platformdirs              3.11.0
pluggy                    1.3.0
pooch                     1.8.0
pre-commit                3.5.0
priam                     0.1.0        C:\Users\anthony.pinto\Documents\git\priam
probeinterface            0.2.19
prometheus-client         0.19.0
prompt-toolkit            3.0.41
protobuf                  4.25.1
psutil                    5.9.6
pubnub                    6.3.3
pure-eval                 0.2.2
py                        1.11.0
pyarrow                   11.0.0
pycodestyle               2.8.0
pycparser                 2.21
pycryptodomex             3.18.0
pyct                      0.5.0
pydantic                  2.5.2
pydantic_core             2.14.5
pydot                     1.4.2
pyflakes                  2.4.0
pygit2                    1.13.3
Pygments                  2.17.2
pygtrie                   2.5.0
pymatreader               0.0.32
pynwb                     2.3.2
PyOpenGL                  3.1.1a1
PyOpenGL-accelerate       3.1.7
pyparsing                 3.1.1
pyprep                    0.4.3
pyproject-api             1.6.1
PyQt5                     5.15.10
PyQt5-Qt5                 5.15.2
PyQt5-sip                 12.13.0
PyQtWebEngine             5.15.6
PyQtWebEngine-Qt5         5.15.2
pyreadline3               3.4.1
pyriemann                 0.5
pyrsistent                0.19.3
pytest                    7.4.3
pytest-cov                4.1.0
pytest-flakes             4.0.5
pytest-html               3.2.0
pytest-metadata           3.0.0
python-dateutil           2.8.2
python-json-logger        2.0.7
python-picard             0.7
pytz                      2023.3.post1
PyWavelets                1.5.0
pywin32                   306
pywinpty                  2.0.12
PyYAML                    6.0.1
pyyaml_env_tag            0.1
pyzmq                     24.0.1
qtconsole                 5.5.1
QtPy                      2.4.1
quantities                0.14.1
referencing               0.31.0
requests                  2.31.0
rfc3339-validator         0.1.4
rfc3986-validator         0.1.1
rich                      13.7.0
rpds-py                   0.13.1
ruamel.yaml               0.18.5
ruamel.yaml.clib          0.2.8
scikit-learn              1.2.2
scipy                     1.11.4
scmrepo                   1.6.0
seaborn                   0.13.0
semver                    3.0.2
Send2Trash                1.8.2
setuptools                69.0.2
setuptools-scm            8.0.4
shortuuid                 1.0.11
shtab                     1.6.4
simplejson                3.19.1
six                       1.16.0
skl2onnx                  1.15.0
sleepecg                  0.5.5
smmap                     5.0.1
sniffio                   1.3.0
sortedcontainers          2.4.0
sortingview               0.11.15
soundfile                 0.12.1
soupsieve                 2.5
soxr                      0.3.7
spikeinterface            0.100.0.dev0
sqltrie                   0.9.0
stack-data                0.6.3
statsmodels               0.14.0
stochastic                0.6.0
sympy                     1.12
tabulate                  0.9.0
tangled-up-in-unicode     0.2.0
tblib                     1.7.0
tensorpac                 0.6.5
terminado                 0.18.0
threadpoolctl             3.2.0
tinycss2                  1.2.1
tk                        0.1.0
toml                      0.10.2
tomli                     2.0.1
tomlkit                   0.12.3
toolz                     0.12.0
tornado                   6.3.3
tox                       4.11.4
tqdm                      4.66.1
traitlets                 5.14.0
typeguard                 2.13.3
typer                     0.9.0
types-python-dateutil     2.8.19.14
typing_extensions         4.8.0
tzdata                    2023.3
uri-template              1.3.0
urllib3                   2.1.0
vine                      5.1.0
virtualenv                20.24.7
visions                   0.7.5
viziphant                 0.3.0
voluptuous                0.14.1
watchdog                  3.0.0
wcwidth                   0.2.12
webcolors                 1.13
webencodings              0.5.1
websocket-client          1.6.4
wheel                     0.40.0
widgetsnbextension        4.0.9
wordcloud                 1.9.2
xmltodict                 0.13.0
yarl                      1.9.3
yasa                      0.6.3
ydata-profiling           4.5.1
zarr                      2.16.1
zc.lockfile               3.0.post1
zict                      3.0.0
zipp                      3.17.0

[wliWanru]

Hi Anthony,

I encountered a similar problem with my single-channel data today.

I had installed SpikeInterface several months ago, and in the version I had 0.99.1 directly installed via pip, the following lines of code in sortingcomponents/clustering/random_projections.py

projections = np.random.randn(num_chans, d["nb_projections"])
projections -= projections.mean(0)
projections /= projections.std(0)

resulted in all NaN values in the 'projections' array, which then caused a ValueError. Then I found that this issue has been fixed in the current version of the SpikeInterface repository, as seen in this commit: https://github.com/SpikeInterface/spikeinterface/commit/93fbc55a29d769020b4835156dd1bfe094e8296c

I noticed that you also dealing with single-channel data, and I'm curious if you experienced a similar problem due to the same reason. If so, the main branch version works fine!

[Antho2422]

Hi @wliWanru,

Thanks for this update, I will try to update spikeinterface and see if this works. Thank you

[alejoe91]

Note that version 0.100.0nwas just released today!