Error - Size must be positive integer in shared_memory

[adam-hockley]

I have come across and error that I can't seem to fix. It occurs with SpykingCircus2 and TDC2 in two environments that I've set up.

I'm sorting 64 channel data, from a binary file of about 8gb.

The global job kwargs are set as: global_job_kwargs = dict(n_jobs=-1, chunk_duration="5s", progress_bar=True) si.set_global_job_kwargs(**global_job_kwargs)

Any help much appreciated!!!

---

SpikeSortingError: Spike sorting error trace: Traceback (most recent call last): File "C:\Users\ANL\anaconda3\envs\SI_env_updated\lib\site-packages\spikeinterface\sorters\basesorter.py", line 258, in run_from_folder SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose) File "C:\Users\ANL\anaconda3\envs\SI_env_updated\lib\site-packages\spikeinterface\sorters\internal\spyking_circus2.py", line 126, in _run_from_folder recording_f = cache_preprocessing(recording_f, job_kwargs, params["cache_preprocessing"]) File "C:\Users\ANL\anaconda3\envs\SI_env_updated\lib\site-packages\spikeinterface\sortingcomponents\tools.py", line 92, in cache_preprocessing recording = recording.save_to_memory(format="memory", shared=True, job_kwargs) File "C:\Users\ANL\anaconda3\envs\SI_env_updated\lib\site-packages\spikeinterface\core\base.py", line 853, in save_to_memory cached = self._save(format="memory", sharedmem=sharedmem, save_kwargs) File "C:\Users\ANL\anaconda3\envs\SI_env_updated\lib\site-packages\spikeinterface\core\baserecording.py", line 490, in _save cached = SharedMemoryRecording.from_recording(self, job_kwargs) File "C:\Users\ANL\anaconda3\envs\SI_env_updated\lib\site-packages\spikeinterface\core\numpyextractors.py", line 215, in from_recording traces_list, shms = write_memory_recording(source_recording, buffer_type="sharedmem", job_kwargs) File "C:\Users\ANL\anaconda3\envs\SI_env_updated\lib\site-packages\spikeinterface\core\recording_tools.py", line 322, in write_memory_recording arr, shm = make_shared_array(shape, dtype) File "C:\Users\ANL\anaconda3\envs\SI_env_updated\lib\site-packages\spikeinterface\core\core_tools.py", line 167, in make_shared_array shm = SharedMemory(name=None, create=True, size=nbytes) File "C:\Users\ANL\anaconda3\envs\SI_env_updated\lib\multiprocessing\shared_memory.py", line 77, in init raise ValueError("'size' must be a positive integer") ValueError: 'size' must be a positive integer

[zm711]

What version of spikeinterface are you using? That bug was fixed in #2621, this is only on main though. We should probably also port this to bug-fixes for 100, @alejoe91 ?

[adam-hockley]

This was with version 100.5

Should I install from source?

Thanks!

[zm711]

We have an api change on source going from waveform_extractor -> sorting_analyzer so you'll have to rewrite some code, but if you're willing to install from source you can get used to the slight api change (we also have some back compatibility code to help with the switch) since that bug fix is currently only in the source code (on the main branch) and not in a release.

[adam-hockley]

Okay thanks, I'll install from source to get past this and see how I get on with the API changes!

[zm711]

Let us know if you have any issues, but all of our main docs have switched over and we have an excellent how_to on the switch.

[adam-hockley]

Thanks, thats all very easy to implement and the how-to is great!

Only issue I'm having is with exporting to Phy, as the create_sorting_analyzer was forced return_scaled=False. These data are binary files in volts.

Error trace below:

si.export_to_phy(sorting_analyzer=sa, output_folder=x[:-4]+'/Sorting/SC2/Phy',copy_binary=False) Traceback (most recent call last):

Cell In[36], line 1 si.export_to_phy(sorting_analyzer=sa, output_folder=x[:-4]+'/Sorting/SC2/Phy',copy_binary=False)

File ~\spikeinterface\src\spikeinterface\exporters\to_phy.py:219 in export_to_phy sorting_analyzer.compute("spike_amplitudes", **job_kwargs)

File ~\spikeinterface\src\spikeinterface\core\sortinganalyzer.py:853 in compute return self.compute_one_extension(extension_name=input, save=save, **kwargs)

File ~\spikeinterface\src\spikeinterface\core\sortinganalyzer.py:931 in compute_one_extension extension_instance.run(save=save, **job_kwargs)

File ~\spikeinterface\src\spikeinterface\core\sortinganalyzer.py:1532 in run self._run(**kwargs)

File ~\spikeinterface\src\spikeinterface\postprocessing\spike_amplitudes.py:112 in _run nodes = self.get_pipeline_nodes()

File ~\spikeinterface\src\spikeinterface\core\sortinganalyzer.py:1700 in get_pipeline_nodes return self._get_pipeline_nodes()

File ~\spikeinterface\src\spikeinterface\postprocessing\spike_amplitudes.py:92 in _get_pipeline_nodes extremum_channels_indices = get_template_extremum_channel(

File ~\spikeinterface\src\spikeinterface\core\template_tools.py:150 in get_template_extremum_channel peak_values = get_template_amplitudes(templates_or_sorting_analyzer, peak_sign=peak_sign, mode=mode)

File ~\spikeinterface\src\spikeinterface\core\template_tools.py:90 in get_template_amplitudes templates_array = get_dense_templates_array(templates_or_sorting_analyzer, return_scaled=return_scaled)

File ~\spikeinterface\src\spikeinterface\core\template_tools.py:30 in get_dense_templates_array raise ValueError(

ValueError: get_dense_templates_array: return_scaled=True is not possible SortingAnalyzer has the reverse

[zm711]

Yeah we ran into that issue here #2751 and it is fixed here #2752. Just waiting for that to be merged! Likely next week. So if you are in a rush you can install from that PR. :)

[adam-hockley]

Amazing, thanks so much!!

[h-mayorquin]

Thanks a lot @zm711 for being on top of this!