Probe missing when using run_sorter()

electro-phys commented 9 months ago

Hi, I hope this is just a user error on my part. I am trying to spike sort some data, the sorters appear to be working. However, even though a probe appears to be attached to the recording, run_sorter() says there is none. I have this issue for other sorters that require a probe map as well (spykingcircuis, kilosort, etc.), but not for mountainsort5 (I'm assuming due to some inherent difference in computation in MS5 not needing the probe).

import matplotlib.pyplot as plt
from pprint import pprint
import spikeinterface.full as si
from pathlib import Path
from probeinterface import get_probe
from probeinterface.plotting import plot_probe
import numpy as np

camb_64x1 = get_probe('cambridgeneurotech','ASSY-77-H3') #64x1 probe catalog number
camb64x1_tdt_map = np.array([12, 11, 4, 8, 16, 64, 1, 2, 13, 9, 10, 14, 6, 7, 5, 3, 50, 49, 52, 
                             51, 54, 53, 56, 55, 15, 63, 62, 61, 59, 60, 57, 58, 23, 28, 21, 22, 20, 
                             19, 18, 17, 42, 24, 29, 25, 30, 26, 31, 27, 46, 45, 40, 34, 37, 32, 33, 
                             41, 44, 47, 48, 43, 35, 38, 39, 36])
camb64x1_tdt_map = camb64x1_tdt_map - 1 # subtract for python indexing
camb_64x1.set_device_channel_indices(camb64x1_tdt_map)

base_folder = Path('E:/PFC_DREADDS/Record-230306-100339/r392_pfc_spon_bl1/') # so we can save binary files and such to the same data folder
tdt_tank = base_folder / 'Record-230306-100339_r392_pfc_spon_bl1.Tbk'
raw_data = si.read_tdt(tdt_tank,stream_name="b'SPKw'")

raw_data = raw_data.set_probe(camb_64x1)
probe_df = raw_data.get_probe().to_dataframe()
print(probe_df)

The output confirming the probe:

c:\Users\admin\anaconda3\envs\si_env\lib\site-packages\neo\rawio\tdtrawio.py:293: UserWarning: Could not identify sev files for channels [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64]. warnings.warn(f"Could not identify sev files for channels {missing_sev_channels}.") x y contact_shapes width height shank_ids contact_ids 0 0.0 880.0 rect 11.0 15.0 7 1 0.0 700.0 rect 11.0 15.0 8 2 0.0 1160.0 rect 11.0 15.0 16 3 0.0 920.0 rect 11.0 15.0 3 4 0.0 800.0 rect 11.0 15.0 15 .. ... ... ... ... ... ... ... 59 0.0 1200.0 rect 11.0 15.0 30 60 0.0 980.0 rect 11.0 15.0 28 61 0.0 1100.0 rect 11.0 15.0 27 62 0.0 1020.0 rect 11.0 15.0 26 63 0.0 680.0 rect 11.0 15.0 6

[64 rows x 7 columns]

Probe not showing up for sorting

sorting = si.run_sorter(recording=raw_data, sorter_name='tridesclous2', output_folder= base_folder / 'tdc_map_test')

Output:

c:\Users\admin\anaconda3\envs\si_env\lib\site-packages\neo\rawio\tdtrawio.py:293: UserWarning: Could not identify sev files for channels [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64]. warnings.warn(f"Could not identify sev files for channels {missing_sev_channels}.") c:\Users\admin\anaconda3\envs\si_env\lib\site-packages\spikeinterface\core\baserecordingsnippets.py:244: UserWarning: There is no Probe attached to this recording. Creating a dummy one with contact positions warn("There is no Probe attached to this recording. Creating a dummy one with contact positions") c:\Users\admin\anaconda3\envs\si_env\lib\site-packages\neo\rawio\tdtrawio.py:293: UserWarning: Could not identify sev files for channels [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64]. warnings.warn(f"Could not identify sev files for channels {missing_sev_channels}.")

Thank you very much for your help and this interface, Walker

alejoe91 commented 9 months ago

Hi Walker, what version are you using? Can you upgrade to the latest version (pip install -U spikeinterface) and check if the problem is still there?

electro-phys commented 9 months ago

That did something. I updated to the latest version (probeinterface 0.2.21 and spikeinterface 0.100.0). If I try to import spikeinterface, import spikeinterface.full as si, I get an error that there is no numcodecs.shuffle module. Running pip install numcodecs says that I have both numcodecs (0.12.1) and numpy (1.26.3) installed currently. Is there a problem with the versions here? I know there were some numpy changes after 1.23.1 that I've had issues with in the past.

current numcodecs error

{ "name": "ModuleNotFoundError", "message": "No module named 'numcodecs._shuffle'", "stack": "--------------------------------------------------------------------------- ModuleNotFoundError Traceback (most recent call last) Cell In[4], line 3 1 import matplotlib.pyplot as plt 2 from pprint import pprint ----> 3 import spikeinterface.full as si 4 from pathlib import Path 5 from probeinterface import get_probe

File c:\Users\admin\anaconda3\envs\si_env\lib\site-packages\spikeinterface\init.py:10 6 import importlib.metadata 8 version = importlib.metadata.version(\"spikeinterface\") ---> 10 from .core import * 12 import warnings 14 warnings.filterwarnings(\"ignore\", message=\"distutils Version classes are deprecated\")

File c:\Users\admin\anaconda3\envs\si_env\lib\site-packages\spikeinterface\core\init.py:19 10 from .npzsortingextractor import NpzSortingExtractor, read_npz_sorting 11 from .numpyextractors import ( 12 NumpyRecording, 13 SharedMemoryRecording, (...) 17 NumpySnippets, 18 ) ---> 19 from .zarrextractors import ZarrRecordingExtractor, ZarrSortingExtractor, read_zarr, get_default_zarr_compressor 20 from .binaryfolder import BinaryFolderRecording, read_binary_folder 21 from .sortingfolder import NumpyFolderSorting, NpzFolderSorting, read_numpy_sorting_folder, read_npz_folder

File c:\Users\admin\anaconda3\envs\si_env\lib\site-packages\spikeinterface\core\zarrextractors.py:5 3 from pathlib import Path 4 import numpy as np ----> 5 import zarr 7 from probeinterface import ProbeGroup 9 from .baserecording import BaseRecording, BaseRecordingSegment

File c:\Users\admin\anaconda3\envs\si_env\lib\site-packages\zarr\init.py:2 1 # flake8: noqa ----> 2 from zarr.codecs import * 3 from zarr.convenience import ( 4 consolidate_metadata, 5 copy, (...) 14 tree, 15 ) 16 from zarr.core import Array

File c:\Users\admin\anaconda3\envs\si_env\lib\site-packages\zarr\codecs.py:2 1 # flake8: noqa ----> 2 from numcodecs import * 3 from numcodecs import get_codec, Blosc, Pickle, Zlib, Delta, AsType, BZ2 4 from numcodecs.registry import codec_registry

File c:\Users\admin\anaconda3\envs\si_env\lib\site-packages\ umcodecs\init.py:91 88 from numcodecs.base64 import Base64 89 register_codec(Base64) ---> 91 from numcodecs.shuffle import Shuffle 92 register_codec(Shuffle) 94 from numcodecs.bitround import BitRound

File c:\Users\admin\anaconda3\envs\si_env\lib\site-packages\ umcodecs\shuffle.py:4 2 from .compat import ensure_contiguous_ndarray 3 from .abc import Codec ----> 4 from ._shuffle import _doShuffle, _doUnshuffle 7 class Shuffle(Codec): 8 \"\"\"Codec providing shuffle 9 10 Parameters (...) 14 15 \"\"\"

ModuleNotFoundError: No module named 'numcodecs._shuffle'" }

alejoe91 commented 9 months ago

Can you try to upgrade zarr?

electro-phys commented 9 months ago

I upgraded zarr. Now I am getting an actual error message for running tridesclous sorting = si.run_sorter(recording=raw_data, sorter_name='tridesclous2', output_folder= sort_folder) , and a different error for mountainsort sorting = si.run_sorter(recording=raw_data, sorter_name='mountainsort5', output_folder= sort_folder). Looks like maybe a windows pathing problem for MS5.

Tridesclous2 Log.json

{
    "sorter_name": "tridesclous2",
    "sorter_version": "2.0",
    "datetime": "2024-02-06T10:24:37.447393",
    "runtime_trace": [],
    "error": true,
    "error_trace": "Traceback (most recent call last):\n  File \"c:\\Users\\admin\\anaconda3\\envs\\si_bare_test\\lib\\site-packages\\spikeinterface\\sorters\\basesorter.py\", line 258, in run_from_folder\n    SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)\n  File \"c:\\Users\\admin\\anaconda3\\envs\\si_bare_test\\lib\\site-packages\\spikeinterface\\sorters\\internal\\tridesclous2.py\", line 123, in _run_from_folder\n    recording = cache_preprocessing(\n  File \"c:\\Users\\admin\\anaconda3\\envs\\si_bare_test\\lib\\site-packages\\spikeinterface\\sortingcomponents\\tools.py\", line 92, in cache_preprocessing\n    recording = recording.save_to_memory(format=\"memory\", shared=True, **job_kwargs)\n  File \"c:\\Users\\admin\\anaconda3\\envs\\si_bare_test\\lib\\site-packages\\spikeinterface\\core\\base.py\", line 853, in save_to_memory\n    cached = self._save(format=\"memory\", sharedmem=sharedmem, **save_kwargs)\n  File \"c:\\Users\\admin\\anaconda3\\envs\\si_bare_test\\lib\\site-packages\\spikeinterface\\core\\baserecording.py\", line 490, in _save\n    cached = SharedMemoryRecording.from_recording(self, **job_kwargs)\n  File \"c:\\Users\\admin\\anaconda3\\envs\\si_bare_test\\lib\\site-packages\\spikeinterface\\core\\numpyextractors.py\", line 215, in from_recording\n    traces_list, shms = write_memory_recording(source_recording, buffer_type=\"sharedmem\", **job_kwargs)\n  File \"c:\\Users\\admin\\anaconda3\\envs\\si_bare_test\\lib\\site-packages\\spikeinterface\\core\\recording_tools.py\", line 322, in write_memory_recording\n    arr, shm = make_shared_array(shape, dtype)\n  File \"c:\\Users\\admin\\anaconda3\\envs\\si_bare_test\\lib\\site-packages\\spikeinterface\\core\\core_tools.py\", line 167, in make_shared_array\n    shm = SharedMemory(name=None, create=True, size=nbytes)\n  File \"c:\\Users\\admin\\anaconda3\\envs\\si_bare_test\\lib\\multiprocessing\\shared_memory.py\", line 76, in __init__\n    raise ValueError(\"'size' must be a positive integer\")\nValueError: 'size' must be a positive integer\n",
    "run_time": null
}

Spike Sorting Error for TDC

{ "name": "SpikeSortingError", "message": "Spike sorting error trace: Traceback (most recent call last): File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\sorters\basesorter.py\", line 258, in run_from_folder SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\sorters\internal\tridesclous2.py\", line 123, in _run_from_folder recording = cache_preprocessing( File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\sortingcomponents\tools.py\", line 92, in cache_preprocessing recording = recording.save_to_memory(format=\"memory\", shared=True, job_kwargs) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\core\base.py\", line 853, in save_to_memory cached = self._save(format=\"memory\", sharedmem=sharedmem, save_kwargs) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\core\baserecording.py\", line 490, in _save cached = SharedMemoryRecording.from_recording(self, job_kwargs) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\core\ umpyextractors.py\", line 215, in from_recording traces_list, shms = write_memory_recording(source_recording, buffer_type=\"sharedmem\", job_kwargs) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\core\recording_tools.py\", line 322, in write_memory_recording arr, shm = make_shared_array(shape, dtype) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\core\core_tools.py\", line 167, in make_shared_array shm = SharedMemory(name=None, create=True, size=nbytes) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\multiprocessing\shared_memory.py\", line 76, in init raise ValueError(\"'size' must be a positive integer\") ValueError: 'size' must be a positive integer

Spike sorting failed. You can inspect the runtime trace in E:\PFC_DREADDS\Record-230306-100339\r392_pfc_spon_bl1\tdc_map_test10/spikeinterface_log.json.", "stack": "--------------------------------------------------------------------------- SpikeSortingError Traceback (most recent call last) Cell In[9], line 2 1 sort_folder = Path('E:/PFC_DREADDS/Record-230306-100339/r392_pfc_spon_bl1/tdc_map_test10') ----> 2 sorting = si.run_sorter(recording=raw_data, sorter_name='tridesclous2', output_folder= sort_folder)

File c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\sorters\runsorter.py:174, in run_sorter(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, delete_container_files, with_output, sorter_params) 167 container_image = singularity_image 168 return run_sorter_container( 169 container_image=container_image, 170 mode=mode, 171 common_kwargs, 172 ) --> 174 return run_sorter_local(**common_kwargs)

File c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\sorters\runsorter.py:224, in run_sorter_local(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, **sorter_params) 222 SorterClass.set_params_to_folder(recording, output_folder, sorter_params, verbose) 223 SorterClass.setup_recording(recording, output_folder, verbose=verbose) --> 224 SorterClass.run_from_folder(output_folder, raise_error, verbose) 225 if with_output: 226 sorting = SorterClass.get_result_from_folder(output_folder, register_recording=True, sorting_info=True)

File c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\sorters\basesorter.py:293, in BaseSorter.run_from_folder(cls, output_folder, raise_error, verbose) 290 print(f\"{sorter_name} run time {run_time:0.2f}s\") 292 if has_error and raise_error: --> 293 raise SpikeSortingError( 294 f\"Spike sorting error trace:\ {log['error_trace']}\ \" 295 f\"Spike sorting failed. You can inspect the runtime trace in {output_folder}/spikeinterface_log.json.\" 296 ) 298 return run_time

SpikeSortingError: Spike sorting error trace: Traceback (most recent call last): File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\sorters\basesorter.py\", line 258, in run_from_folder SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\sorters\internal\tridesclous2.py\", line 123, in _run_from_folder recording = cache_preprocessing( File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\sortingcomponents\tools.py\", line 92, in cache_preprocessing recording = recording.save_to_memory(format=\"memory\", shared=True, job_kwargs) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\core\base.py\", line 853, in save_to_memory cached = self._save(format=\"memory\", sharedmem=sharedmem, save_kwargs) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\core\baserecording.py\", line 490, in _save cached = SharedMemoryRecording.from_recording(self, job_kwargs) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\core\ umpyextractors.py\", line 215, in from_recording traces_list, shms = write_memory_recording(source_recording, buffer_type=\"sharedmem\", job_kwargs) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\core\recording_tools.py\", line 322, in write_memory_recording arr, shm = make_shared_array(shape, dtype) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\core\core_tools.py\", line 167, in make_shared_array shm = SharedMemory(name=None, create=True, size=nbytes) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\multiprocessing\shared_memory.py\", line 76, in init raise ValueError(\"'size' must be a positive integer\") ValueError: 'size' must be a positive integer

Spike sorting failed. You can inspect the runtime trace in E:\PFC_DREADDS\Record-230306-100339\r392_pfc_spon_bl1\tdc_map_test10/spikeinterface_log.json." } MS5 Log.json

{
    "sorter_name": "mountainsort5",
    "sorter_version": "0.5.3",
    "datetime": "2024-02-06T10:27:01.325974",
    "runtime_trace": [],
    "error": true,
    "error_trace": "Traceback (most recent call last):\n  File \"c:\\Users\\admin\\anaconda3\\envs\\si_bare_test\\lib\\shutil.py\", line 627, in _rmtree_unsafe\n    os.unlink(fullname)\nPermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\\\Users\\\\admin\\\\AppData\\\\Local\\\\Temp\\\\tmp_t3_mc0i\\\\recording.dat'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"c:\\Users\\admin\\anaconda3\\envs\\si_bare_test\\lib\\tempfile.py\", line 805, in onerror\n    _os.unlink(path)\nPermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\\\Users\\\\admin\\\\AppData\\\\Local\\\\Temp\\\\tmp_t3_mc0i\\\\recording.dat'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"c:\\Users\\admin\\anaconda3\\envs\\si_bare_test\\lib\\site-packages\\spikeinterface\\sorters\\basesorter.py\", line 258, in run_from_folder\n    SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)\n  File \"c:\\Users\\admin\\anaconda3\\envs\\si_bare_test\\lib\\site-packages\\spikeinterface\\sorters\\external\\mountainsort5.py\", line 203, in _run_from_folder\n    raise ValueError(f\"Invalid scheme: {scheme} given. scheme must be one of '1', '2' or '3'\")\n  File \"c:\\Users\\admin\\anaconda3\\envs\\si_bare_test\\lib\\tempfile.py\", line 830, in __exit__\n    self.cleanup()\n  File \"c:\\Users\\admin\\anaconda3\\envs\\si_bare_test\\lib\\tempfile.py\", line 834, in cleanup\n    self._rmtree(self.name)\n  File \"c:\\Users\\admin\\anaconda3\\envs\\si_bare_test\\lib\\tempfile.py\", line 816, in _rmtree\n    _shutil.rmtree(name, onerror=onerror)\n  File \"c:\\Users\\admin\\anaconda3\\envs\\si_bare_test\\lib\\shutil.py\", line 759, in rmtree\n    return _rmtree_unsafe(path, onerror)\n  File \"c:\\Users\\admin\\anaconda3\\envs\\si_bare_test\\lib\\shutil.py\", line 629, in _rmtree_unsafe\n    onerror(os.unlink, fullname, sys.exc_info())\n  File \"c:\\Users\\admin\\anaconda3\\envs\\si_bare_test\\lib\\tempfile.py\", line 808, in onerror\n    cls._rmtree(path)\n  File \"c:\\Users\\admin\\anaconda3\\envs\\si_bare_test\\lib\\tempfile.py\", line 816, in _rmtree\n    _shutil.rmtree(name, onerror=onerror)\n  File \"c:\\Users\\admin\\anaconda3\\envs\\si_bare_test\\lib\\shutil.py\", line 759, in rmtree\n    return _rmtree_unsafe(path, onerror)\n  File \"c:\\Users\\admin\\anaconda3\\envs\\si_bare_test\\lib\\shutil.py\", line 610, in _rmtree_unsafe\n    onerror(os.scandir, path, sys.exc_info())\n  File \"c:\\Users\\admin\\anaconda3\\envs\\si_bare_test\\lib\\shutil.py\", line 607, in _rmtree_unsafe\n    with os.scandir(path) as scandir_it:\nNotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\\\Users\\\\admin\\\\AppData\\\\Local\\\\Temp\\\\tmp_t3_mc0i\\\\recording.dat'\n",
    "run_time": null
}

Output for MS5

Using training recording of duration 300 sec with the sampling mode uniform MS5 Elapsed time for SCHEME2 get_sampled_recording_for_training: 0.801 seconds Running phase 1 sorting Number of channels: 64 Number of timepoints: 7324218 Sampling frequency: 24414.0625 Hz Channel 0: [ 0. 880.] Channel 1: [ 0. 700.] Channel 2: [ 0. 1160.] Channel 3: [ 0. 920.] Channel 4: [ 0. 800.] Channel 5: [ 0. 820.] Channel 6: [ 0. 760.] Channel 7: [ 0. 660.] Channel 8: [ 0. 720.] Channel 9: [ 0. 840.] Channel 10: [ 0. 640.] Channel 11: [ 0. 940.] Channel 12: [ 0. 860.] Channel 13: [ 0. 740.] Channel 14: [ 0. 1040.] Channel 15: [ 0. 900.] Channel 16: [ 0. 80.] Channel 17: [ 0. 60.] Channel 18: [ 0. 140.] ... Perorming label mapping MS5 Elapsed time for SCHEME2 label mapping: 0.000 seconds Creating sorting object MS5 Elapsed time for SCHEME2 creating sorting object: 0.009 seconds

MS5 Spike Sorting Error

{ "name": "SpikeSortingError", "message": "Spike sorting error trace: Traceback (most recent call last): File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\shutil.py\", line 627, in _rmtree_unsafe os.unlink(fullname) PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\admin\\AppData\\Local\\Temp\\tmp_t3_mc0i\\recording.dat'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\tempfile.py\", line 805, in onerror _os.unlink(path) PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\admin\\AppData\\Local\\Temp\\tmp_t3_mc0i\\recording.dat'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\sorters\basesorter.py\", line 258, in run_from_folder SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\sorters\external\mountainsort5.py\", line 203, in _run_from_folder raise ValueError(f\"Invalid scheme: {scheme} given. scheme must be one of '1', '2' or '3'\") File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\tempfile.py\", line 830, in exit self.cleanup() File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\tempfile.py\", line 834, in cleanup self._rmtree(self.name) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\tempfile.py\", line 816, in _rmtree _shutil.rmtree(name, onerror=onerror) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\shutil.py\", line 759, in rmtree return _rmtree_unsafe(path, onerror) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\shutil.py\", line 629, in _rmtree_unsafe onerror(os.unlink, fullname, sys.exc_info()) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\tempfile.py\", line 808, in onerror cls._rmtree(path) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\tempfile.py\", line 816, in _rmtree _shutil.rmtree(name, onerror=onerror) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\shutil.py\", line 759, in rmtree return _rmtree_unsafe(path, onerror) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\shutil.py\", line 610, in _rmtree_unsafe onerror(os.scandir, path, sys.exc_info()) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\shutil.py\", line 607, in _rmtree_unsafe with os.scandir(path) as scandir_it: NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\admin\\AppData\\Local\\Temp\\tmp_t3_mc0i\\recording.dat'

Spike sorting failed. You can inspect the runtime trace in E:\PFC_DREADDS\Record-230306-100339\r392_pfc_spon_bl1\ms5_map_test10/spikeinterface_log.json.", "stack": "--------------------------------------------------------------------------- SpikeSortingError Traceback (most recent call last) Cell In[11], line 2 1 sort_folder = Path('E:/PFC_DREADDS/Record-230306-100339/r392_pfc_spon_bl1/ms5_map_test10') ----> 2 sorting = si.run_sorter(recording=raw_data, sorter_name='mountainsort5', output_folder= sort_folder)

File c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\sorters\runsorter.py:174, in run_sorter(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, delete_container_files, with_output, sorter_params) 167 container_image = singularity_image 168 return run_sorter_container( 169 container_image=container_image, 170 mode=mode, 171 common_kwargs, 172 ) --> 174 return run_sorter_local(**common_kwargs)

File c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\sorters\runsorter.py:224, in run_sorter_local(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, **sorter_params) 222 SorterClass.set_params_to_folder(recording, output_folder, sorter_params, verbose) 223 SorterClass.setup_recording(recording, output_folder, verbose=verbose) --> 224 SorterClass.run_from_folder(output_folder, raise_error, verbose) 225 if with_output: 226 sorting = SorterClass.get_result_from_folder(output_folder, register_recording=True, sorting_info=True)

File c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\sorters\basesorter.py:293, in BaseSorter.run_from_folder(cls, output_folder, raise_error, verbose) 290 print(f\"{sorter_name} run time {run_time:0.2f}s\") 292 if has_error and raise_error: --> 293 raise SpikeSortingError( 294 f\"Spike sorting error trace:\ {log['error_trace']}\ \" 295 f\"Spike sorting failed. You can inspect the runtime trace in {output_folder}/spikeinterface_log.json.\" 296 ) 298 return run_time

SpikeSortingError: Spike sorting error trace: Traceback (most recent call last): File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\shutil.py\", line 627, in _rmtree_unsafe os.unlink(fullname) PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\admin\\AppData\\Local\\Temp\\tmp_t3_mc0i\\recording.dat'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\tempfile.py\", line 805, in onerror _os.unlink(path) PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\admin\\AppData\\Local\\Temp\\tmp_t3_mc0i\\recording.dat'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\sorters\basesorter.py\", line 258, in run_from_folder SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\sorters\external\mountainsort5.py\", line 203, in _run_from_folder raise ValueError(f\"Invalid scheme: {scheme} given. scheme must be one of '1', '2' or '3'\") File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\tempfile.py\", line 830, in exit self.cleanup() File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\tempfile.py\", line 834, in cleanup self._rmtree(self.name) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\tempfile.py\", line 816, in _rmtree _shutil.rmtree(name, onerror=onerror) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\shutil.py\", line 759, in rmtree return _rmtree_unsafe(path, onerror) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\shutil.py\", line 629, in _rmtree_unsafe onerror(os.unlink, fullname, sys.exc_info()) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\tempfile.py\", line 808, in onerror cls._rmtree(path) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\tempfile.py\", line 816, in _rmtree _shutil.rmtree(name, onerror=onerror) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\shutil.py\", line 759, in rmtree return _rmtree_unsafe(path, onerror) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\shutil.py\", line 610, in _rmtree_unsafe onerror(os.scandir, path, sys.exc_info()) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\shutil.py\", line 607, in _rmtree_unsafe with os.scandir(path) as scandir_it: NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\admin\\AppData\\Local\\Temp\\tmp_t3_mc0i\\recording.dat'

Spike sorting failed. You can inspect the runtime trace in E:\PFC_DREADDS\Record-230306-100339\r392_pfc_spon_bl1\ms5_map_test10/spikeinterface_log.json." }

alejoe91 commented 9 months ago

Pinging @samuelgarcia (tridesclous) and @magland (MS5)

zm711 commented 9 months ago

For MS5 I can say I just ran into this and thought it was a bug, but that error on Windows was occurring because you need the newest version of spikeinterface and newest version of mountainsort5 in order to work together. We might need to rewrite a warning to make sure the versions work together. Could you try pip install -U mountainsort5 (and of course if @magland has better advice I would also love to hear it). But after re-syncing up the versions of everything that exact error went away for me.

electro-phys commented 9 months ago

Thanks for the info. I have updated both mountainsort5 (0.5.3) and spikeinterface (0.100.0), but I'm still receiving the same error. Are these correct/compatible versions? Our lab computers are all windows machines, but I will try on both Mac and Linux (Debian) this evening and see if the same issues occur.

zm711 commented 9 months ago

I'm using 0.5.3 for Mountainsort5 and 0.100.0 for when I tested on Windows. I think at this point we just have to wait for @magland to weigh in. I would try restarting your python (whether in juypter or whatever IDE you're using) if you haven't tried that yet. But other than that just updating worked for me.

magland commented 9 months ago

The error message that stands out to me is:

"Invalid scheme: {scheme} given. scheme must be one of '1', '2' or '3'"

What sorting params are you using?

electro-phys commented 9 months ago

I'm using the default parameters. And I get the same error even if I force the other scheme values.

{'scheme': '2',
 'detect_threshold': 5.5,
 'detect_sign': -1,
 'detect_time_radius_msec': 0.5,
 'snippet_T1': 20,
 'snippet_T2': 20,
 'npca_per_channel': 3,
 'npca_per_subdivision': 10,
 'snippet_mask_radius': 250,
 'scheme1_detect_channel_radius': 150,
 'scheme2_phase1_detect_channel_radius': 200,
 'scheme2_detect_channel_radius': 50,
 'scheme2_max_num_snippets_per_training_batch': 200,
 'scheme2_training_duration_sec': 300,
 'scheme2_training_recording_sampling_mode': 'uniform',
 'scheme3_block_duration_sec': 1800,
 'freq_min': 300,
 'freq_max': 6000,
 'filter': True,
 'whiten': True,
 'temporary_base_dir': None,
 'n_jobs_for_preprocessing': -1}

magland commented 9 months ago

That's strange. The source code is:

https://github.com/SpikeInterface/spikeinterface/blob/a4af39095e055ffa9e7295823a9bb9e9d331bf4e/src/spikeinterface/sorters/external/mountainsort5.py#L195-L203

So if it is calling that ValueError there, then the parameters must not be getting passed through correctly to _run_from_folder

Could you please show exactly how you are calling this? And show once again the error that you are getting?

electro-phys commented 9 months ago

Yep: Code to load in recording and set probe (no preprocessing for sake of brevity)

import matplotlib.pyplot as plt
from pprint import pprint
import spikeinterface.full as si
from pathlib import Path
from probeinterface import get_probe
from probeinterface.plotting import plot_probe
import numpy as np

camb_64x1 = get_probe('cambridgeneurotech','ASSY-77-H3') #64x1 probe catalog number
camb64x1_tdt_map = np.array([12, 11, 4, 8, 16, 64, 1, 2, 13, 9, 10, 14, 6, 7, 5, 3, 50, 49, 52, 
                             51, 54, 53, 56, 55, 15, 63, 62, 61, 59, 60, 57, 58, 23, 28, 21, 22, 20, 
                             19, 18, 17, 42, 24, 29, 25, 30, 26, 31, 27, 46, 45, 40, 34, 37, 32, 33, 
                             41, 44, 47, 48, 43, 35, 38, 39, 36])
camb64x1_tdt_map = camb64x1_tdt_map - 1 # subtract for python indexing
camb_64x1.set_device_channel_indices(camb64x1_tdt_map)

base_folder = Path('C:/Users/admin/spikeinterface_testing/Record-230306-100339/r392_pfc_spon_bl2') # so we can save binary files and such to the same data folder
tdt_tank = base_folder / 'Record-230306-100339_r392_pfc_spon_bl2.Tbk'
raw_data = si.read_tdt(tdt_tank,stream_name="b'SPKw'")

raw_data = raw_data.set_probe(camb_64x1)
probe_df = raw_data.get_probe().to_dataframe()

Running spike sorter:

sorting = si.run_sorter(recording=raw_data, sorter_name='mountainsort5', output_folder= base_folder / 'ms5_test22')

MS5 SpikeSortingError

{ "name": "SpikeSortingError", "message": "Spike sorting error trace: Traceback (most recent call last): File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\shutil.py\", line 627, in _rmtree_unsafe os.unlink(fullname) PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\admin\\AppData\\Local\\Temp\\tmp9x792n15\\recording.dat'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\tempfile.py\", line 805, in onerror _os.unlink(path) PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\admin\\AppData\\Local\\Temp\\tmp9x792n15\\recording.dat'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\sorters\basesorter.py\", line 258, in run_from_folder SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\sorters\external\mountainsort5.py\", line 203, in _run_from_folder raise ValueError(f\"Invalid scheme: {scheme} given. scheme must be one of '1', '2' or '3'\") File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\tempfile.py\", line 830, in exit self.cleanup() File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\tempfile.py\", line 834, in cleanup self._rmtree(self.name) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\tempfile.py\", line 816, in _rmtree _shutil.rmtree(name, onerror=onerror) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\shutil.py\", line 759, in rmtree return _rmtree_unsafe(path, onerror) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\shutil.py\", line 629, in _rmtree_unsafe onerror(os.unlink, fullname, sys.exc_info()) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\tempfile.py\", line 808, in onerror cls._rmtree(path) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\tempfile.py\", line 816, in _rmtree _shutil.rmtree(name, onerror=onerror) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\shutil.py\", line 759, in rmtree return _rmtree_unsafe(path, onerror) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\shutil.py\", line 610, in _rmtree_unsafe onerror(os.scandir, path, sys.exc_info()) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\shutil.py\", line 607, in _rmtree_unsafe with os.scandir(path) as scandir_it: NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\admin\\AppData\\Local\\Temp\\tmp9x792n15\\recording.dat'

Spike sorting failed. You can inspect the runtime trace in C:\Users\admin\spikeinterface_testing\Record-230306-100339\r392_pfc_spon_bl2\ms5_test22/spikeinterface_log.json.", "stack": "--------------------------------------------------------------------------- SpikeSortingError Traceback (most recent call last) Cell In[50], line 2 1 #sort_folder = Path('E:/PFC_DREADDS/Record-230306-100339/r392_pfc_spon_bl1/ms5_map_test16') ----> 2 sorting = si.run_sorter(recording=raw_data, sorter_name='mountainsort5', output_folder= base_folder / 'ms5_test22')

File c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\sorters\runsorter.py:174, in run_sorter(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, delete_container_files, with_output, sorter_params) 167 container_image = singularity_image 168 return run_sorter_container( 169 container_image=container_image, 170 mode=mode, 171 common_kwargs, 172 ) --> 174 return run_sorter_local(**common_kwargs)

File c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\sorters\runsorter.py:224, in run_sorter_local(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, **sorter_params) 222 SorterClass.set_params_to_folder(recording, output_folder, sorter_params, verbose) 223 SorterClass.setup_recording(recording, output_folder, verbose=verbose) --> 224 SorterClass.run_from_folder(output_folder, raise_error, verbose) 225 if with_output: 226 sorting = SorterClass.get_result_from_folder(output_folder, register_recording=True, sorting_info=True)

File c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\sorters\basesorter.py:293, in BaseSorter.run_from_folder(cls, output_folder, raise_error, verbose) 290 print(f\"{sorter_name} run time {run_time:0.2f}s\") 292 if has_error and raise_error: --> 293 raise SpikeSortingError( 294 f\"Spike sorting error trace:\ {log['error_trace']}\ \" 295 f\"Spike sorting failed. You can inspect the runtime trace in {output_folder}/spikeinterface_log.json.\" 296 ) 298 return run_time

SpikeSortingError: Spike sorting error trace: Traceback (most recent call last): File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\shutil.py\", line 627, in _rmtree_unsafe os.unlink(fullname) PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\admin\\AppData\\Local\\Temp\\tmp9x792n15\\recording.dat'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\tempfile.py\", line 805, in onerror _os.unlink(path) PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\admin\\AppData\\Local\\Temp\\tmp9x792n15\\recording.dat'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\sorters\basesorter.py\", line 258, in run_from_folder SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\site-packages\spikeinterface\sorters\external\mountainsort5.py\", line 203, in _run_from_folder raise ValueError(f\"Invalid scheme: {scheme} given. scheme must be one of '1', '2' or '3'\") File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\tempfile.py\", line 830, in exit self.cleanup() File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\tempfile.py\", line 834, in cleanup self._rmtree(self.name) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\tempfile.py\", line 816, in _rmtree _shutil.rmtree(name, onerror=onerror) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\shutil.py\", line 759, in rmtree return _rmtree_unsafe(path, onerror) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\shutil.py\", line 629, in _rmtree_unsafe onerror(os.unlink, fullname, sys.exc_info()) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\tempfile.py\", line 808, in onerror cls._rmtree(path) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\tempfile.py\", line 816, in _rmtree _shutil.rmtree(name, onerror=onerror) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\shutil.py\", line 759, in rmtree return _rmtree_unsafe(path, onerror) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\shutil.py\", line 610, in _rmtree_unsafe onerror(os.scandir, path, sys.exc_info()) File \"c:\Users\admin\anaconda3\envs\si_bare_test\lib\shutil.py\", line 607, in _rmtree_unsafe with os.scandir(path) as scandir_it: NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\admin\\AppData\\Local\\Temp\\tmp9x792n15\\recording.dat'

Spike sorting failed. You can inspect the runtime trace in C:\Users\admin\spikeinterface_testing\Record-230306-100339\r392_pfc_spon_bl2\ms5_test22/spikeinterface_log.json." }

magland commented 9 months ago

Thanks @electro-phys . I'm at a loss because I don't know why scheme would not be equal to '2'. Is there additional info in the spike_interface_log.json?

@zm711 any idea why the scheme parameter wouldn't be getting passed through as the default '2'?

zm711 commented 9 months ago

@magland The way I'm reading this error trace it looks a Win32 Error is occurring here:

File "c:\Users\admin\anaconda3\envs\si_bare_test\lib\shutil.py", line 627, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\admin\\AppData\\Local\\Temp\\tmp9x792n15\\recording.dat'

If you look at the text above the error we are actually able to trigger scheme 2


Using training recording of duration 300 sec with the sampling mode uniform
*** MS5 Elapsed time for SCHEME2 get_sampled_recording_for_training: 0.801 seconds ***
Running phase 1 sorting
Number of channels: 64
Number of timepoints: 7324218
Sampling frequency: 24414.0625 Hz
Channel 0: [ 0. 880.]
Channel 1: [ 0. 700.]
Channel 2: [ 0. 1160.]
Channel 3: [ 0. 920.]
Channel 4: [ 0. 800.]
Channel 5: [ 0. 820.]
Channel 6: [ 0. 760.]
Channel 7: [ 0. 660.]
Channel 8: [ 0. 720.]
Channel 9: [ 0. 840.]
Channel 10: [ 0. 640.]
Channel 11: [ 0. 940.]
Channel 12: [ 0. 860.]
Channel 13: [ 0. 740.]
Channel 14: [ 0. 1040.]
Channel 15: [ 0. 900.]
Channel 16: [ 0. 80.]
Channel 17: [ 0. 60.]
Channel 18: [ 0. 140.]
...
Perorming label mapping
*** MS5 Elapsed time for SCHEME2 label mapping: 0.000 seconds ***
Creating sorting object
*** MS5 Elapsed time for SCHEME2 creating sorting object: 0.009 seconds ***

My guess is that TemporaryDirectory is unable to cleanup on Windows due to the recording not being released and the ValueError is just being displayed due to a bad stack trace.

I can confirm I had the exact same error on Windows, but it was fixed by updating, but maybe this is due to another difference as well. I typically run run_sorter_by_property instead of run_sorter. One other difference is the temp file I made isn't being made in the admin level but at my user level, so maybe that is part of the problem. My IT department has us in user folders with admin privileges, but we don't actually run in an admin account. I don't know Windows OS well enough to know if that could matter (personally a Mac user).

@electro-phys If you are able to share your files I could try to run on my computer and see if this is a spikeinterface/mountainsort5 issue or local to your computer's setup.

magland commented 9 months ago

Okay good point. I guess it's a bad stack trace as you said.

alejoe91 commented 9 months ago

Another option is to try a different temp dir with the temporary_base_dir parameter. It might be an issue due to a limited size of the default tempdir

electro-phys commented 9 months ago

Sorry for the delay. I used a different temp directory with the temporary_base_dir parameter as suggested, and it seems to have fixed the issue. Thanks for the help everyone.

alejoe91 commented 9 months ago

Closing then :)

jesusdpa1 commented 9 months ago

Hi!

I am encountering a similar issue even when running

# code
#%%
recording_loaded = si.load_extractor(drv_dir.joinpath("preprocessed"))

# %%
si.get_default_sorter_params('mountainsort5')

#%%
sorter_params = {'filter': False,
                 'temporary_base_dir': drv_dir.joinpath('tmp').__str__(),
                 'n_jobs_for_preprocessing': 1}
#%%
sorting_ = si.run_sorter(sorter_name='mountainsort5', recording=recording_loaded, remove_existing_folder=True,
                         output_folder=drv_dir.joinpath('results_').__str__(), 
                         **sorter_params,
                         verbose=True)

# params
{
    "sorter_name": "mountainsort5",
    "sorter_params": {
        "scheme": "2",
        "detect_threshold": 5.5,
        "detect_sign": -1,
        "detect_time_radius_msec": 0.5,
        "snippet_T1": 20,
        "snippet_T2": 20,
        "npca_per_channel": 3,
        "npca_per_subdivision": 10,
        "snippet_mask_radius": 250,
        "scheme1_detect_channel_radius": 150,
        "scheme2_phase1_detect_channel_radius": 200,
        "scheme2_detect_channel_radius": 50,
        "scheme2_max_num_snippets_per_training_batch": 200,
        "scheme2_training_duration_sec": 300,
        "scheme2_training_recording_sampling_mode": "uniform",
        "scheme3_block_duration_sec": 1800,
        "freq_min": 300,
        "freq_max": 6000,
        "filter": false,
        "whiten": true,
        "temporary_base_dir": "C:\\Users\\jesus.penalozaa\\Documents\\Data\\SpikeInterface\\02-09-24_8898-1_testSubject-HD EMG electrodes 5th try\\drv\\tmp",
        "n_jobs_for_preprocessing": 1
    }
}

# Error
{
    "sorter_name": "mountainsort5",
    "sorter_version": "0.5.3",
    "datetime": "2024-02-14T15:30:53.649657",
    "runtime_trace": [],
    "error": true,
    "error_trace": "Traceback (most recent call last):\n  File \"c:\\Users\\jesus.penalozaa\\AppData\\Local\\miniconda3\\envs\\roboz\\Lib\\shutil.py\", line 634, in _rmtree_unsafe\n    os.unlink(fullname)\nPermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\\\Users\\\\jesus.penalozaa\\\\Documents\\\\Data\\\\SpikeInterface\\\\02-09-24_8898-1_testSubject-HD EMG electrodes 5th try\\\\drv\\\\tmp\\\\tmpdcq4c5or\\\\recording.dat'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"c:\\Users\\jesus.penalozaa\\AppData\\Local\\miniconda3\\envs\\roboz\\Lib\\site-packages\\spikeinterface\\sorters\\basesorter.py\", line 258, in run_from_folder\n    SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)\n  File \"c:\\Users\\jesus.penalozaa\\AppData\\Local\\miniconda3\\envs\\roboz\\Lib\\site-packages\\spikeinterface\\sorters\\external\\mountainsort5.py\", line 189, in _run_from_folder\n    with TemporaryDirectory(dir=p[\"temporary_base_dir\"]) as tmpdir:\n  File \"c:\\Users\\jesus.penalozaa\\AppData\\Local\\miniconda3\\envs\\roboz\\Lib\\tempfile.py\", line 946, in __exit__\n    self.cleanup()\n  File \"c:\\Users\\jesus.penalozaa\\AppData\\Local\\miniconda3\\envs\\roboz\\Lib\\tempfile.py\", line 950, in cleanup\n    self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)\n  File \"c:\\Users\\jesus.penalozaa\\AppData\\Local\\miniconda3\\envs\\roboz\\Lib\\tempfile.py\", line 930, in _rmtree\n    _shutil.rmtree(name, onexc=onexc)\n  File \"c:\\Users\\jesus.penalozaa\\AppData\\Local\\miniconda3\\envs\\roboz\\Lib\\shutil.py\", line 808, in rmtree\n    return _rmtree_unsafe(path, onexc)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"c:\\Users\\jesus.penalozaa\\AppData\\Local\\miniconda3\\envs\\roboz\\Lib\\shutil.py\", line 636, in _rmtree_unsafe\n    onexc(os.unlink, fullname, err)\n  File \"c:\\Users\\jesus.penalozaa\\AppData\\Local\\miniconda3\\envs\\roboz\\Lib\\tempfile.py\", line 905, in onexc\n    _os.unlink(path)\nPermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\\\Users\\\\jesus.penalozaa\\\\Documents\\\\Data\\\\SpikeInterface\\\\02-09-24_8898-1_testSubject-HD EMG electrodes 5th try\\\\drv\\\\tmp\\\\tmpdcq4c5or\\\\recording.dat'\n",
    "run_time": null
}

zm711 commented 9 months ago

This might be worth just opening another issue @jesusdpa1, but I would try two things:

1) Set a different temporary_base_dir maybe not so nested inside your data folder. This file should get cleaned up so it shouldn't really matter where it gets put. So I would try putting it in a less nested spot. Also make sure your c drive has enough space to contain the .dat file which depending on your recording could be many gigabyes.

SpikeInterface / spikeinterface