[Bug]: Double probe recording for KiloSortSortingInterface

[vigji]

What happened?

I tried to load two spike sorters from an experiment with two probes, then wrote a single file using ConverterPipe([KiloSortSortingInterface1, KiloSortSortingInterface2]).

When I open the file, I can't find half of the units, which apparently got overwritten. I was planning to just import the sorted units and not the corresponding raw electrical series; am I missing some arguments/metadata specification that would fix this? In that case, sorry in advance!

Steps to Reproduce

interface1 = KiloSortSortingInterface(folder_path=probe1_path, verbose=False)
interface2 = KiloSortSortingInterface(folder_path=probe2_path, verbose=False)

converter_pipe = ConverterPipe([interface1, interface2])
metadata = interface.get_metadata()
session_start_time = datetime(2020, 1, 1, 12, 30, 0, tzinfo=ZoneInfo("US/Pacific"))
metadata["NWBFile"].update(session_start_time=session_start_time)
converter_pipe.run_conversion(nwbfile_path= nwb_spike_file, metadata=metadata)

with NWBHDF5IO(nwb_spike_file, 'r') as io:
    nwbfile = io.read()
    units_table_df = nwbfile.units.to_dataframe()

Traceback

No response

Operating System

macOS

Python Executable

Conda

Python Version

3.10

Package Versions

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct
• [X] Have you ensured this bug was not already reported?

[h-mayorquin]

Hi, neuroconv decides on unit identity based on two things: "unit_name` and its corresponding electrodes (the electrodes in the electrode table associated with it). There are two ways of solving this from within neuroconv

Change the unit name of the sorting

from neuroconv.tools.testing import MockSortingInterface
from neuroconv import ConverterPipe

sorting_interface1 = MockSortingInterface(num_units=4)
sorting_interface2 = MockSortingInterface(num_units=4)

sorting_extractor1 = sorting_interface1.sorting_extractor
sorting_interface1.sorting_extractor = sorting_extractor1.rename_units(new_unit_ids=["a_sorting1", "b_sorting1", "c_sorting1", "d_sorting1"])

sorting_extractor2 = sorting_interface2.sorting_extractor
sorting_interface2.sorting_extractor = sorting_extractor2.rename_units(new_unit_ids=["a_sorting2", "b_sorting2", "c_sorting2", "d_sorting2"])

data_interfaces = {"sorting1": sorting_interface1, "sorting2": sorting_interface2}
converter = ConverterPipe(data_interfaces = data_interfaces)

# You can also run converter.run_conversion(kwargs)
nwbfile = converter.create_nwbfile()
nwbfile.units.to_dataframe()

Write them in two separate tables

from neuroconv.tools.testing import MockSortingInterface
from neuroconv import ConverterPipe

sorting_interface1 = MockSortingInterface(num_units=4, seed=0)
sorting_interface2 = MockSortingInterface(num_units=4, seed=1)

data_interfaces = {"sorting1": sorting_interface1, "sorting2": sorting_interface2}
converter = ConverterPipe(data_interfaces=data_interfaces)

conversion_options = {
    "sorting1": {"write_as": "processing", "units_name": "UnitsSorting1"},
    "sorting2": {"write_as": "processing", "units_name": "UnitsSorting2"},
}

# You can also run converter.run_conversion(kwargs)
nwbfile = converter.create_nwbfile(conversion_options=conversion_options)

There is another way of solving this but I realize we have not propagated that yet. I will work on that.

[h-mayorquin]

Also, can you tell us more on why you want to add more than one sorting interface? As in, what is the experimental setup or result you want to achieve, just more context.

@bendichter was asking recently if there were cases like this and we thought that this feature was sort of artificial so this is good for us to know. Knowing more about use cases is always useful for us.

[vigji]

This worked, thank you very much!

In my case, I used kilosort4 called from spikeinterface one probe at the time. Not sure if this was ill advised. Yet, that resulted in two folders with data sorted for different probes

[h-mayorquin]

In that case, I also advise to add an annotation to the sorting unit about the probe if you are using one table or if you are using two tables add the probe in the description.

[vigji]

Yes, I went for units ids with probe name included. Thanks again for your prompt and useful help!

[h-mayorquin]

Thanks, I should add a how-to with this kind of info. I would need https://github.com/catalystneuro/neuroconv/pull/1065 to be merge first to write though.