curation_dict from sortingview doesn't have 'mergeGroups' key

[rkim48]

Hello, I just updated Spikeinterface from 0.100.2 to 0.101.0 and am making the transition to using sorting analyzer class which has been pretty smooth for the most part but I'm getting an error related to sortingview_curation. Here's my code:

save_folder = Path(data_folder) / "batch_sort"
analyzer = si.load_sorting_analyzer(
    folder=save_folder / "stage1/stage1_analyzer.zarr")

analyzer.compute("random_spikes") 
analyzer.compute("waveforms", ms_before=1.0, ms_after=2.0)
analyzer.compute("templates")
analyzer.compute("template_similarity")
analyzer.compute("correlograms")
analyzer.compute("spike_amplitudes")
analyzer.compute("unit_locations")

si.plot_sorting_summary(analyzer, curation=True, backend="sortingview")
uri = input("\nPlease enter uri: ")
manual_curation_sorting = si.apply_sortingview_curation(
    analyzer.sorting, uri_or_json=uri)

However, apply_sortingview_curation throws me a KeyError.

si.apply_sortingview_curation(
    analyzer.sorting, uri_or_json=uri)
Traceback (most recent call last):

  Cell In[59], line 1
    si.apply_sortingview_curation(

  File ~\anaconda3\envs\si_env3\Lib\site-packages\spikeinterface\curation\sortingview_curation.py:74 in apply_sortingview_curation
    curation_dict = convert_from_sortingview_curation_format_v0(curation_dict)

  File ~\anaconda3\envs\si_env3\Lib\site-packages\spikeinterface\curation\curation_format.py:96 in convert_from_sortingview_curation_format_v0
    merge_groups = sortingview_dict["mergeGroups"]

KeyError: 'mergeGroups'

Printing curation_dict returns this:

{'labelsByUnit': {'12': ['reject'], '13': ['accept'], '14': ['accept'], '15': ['accept'], '16': ['accept'], '17': ['accept'], '22': ['accept'], '23': ['accept'], '24': ['accept'], '25': ['reject'], '26': ['accept'], '30': ['accept'], '31': ['accept'], '32': ['accept'], '37': ['accept'], '40': ['accept'], '41': ['accept'], '48': ['accept'], '49': ['accept'], '50': ['accept'], '51': ['accept'], '52': ['accept'], '54': ['reject'], '55': ['accept'], '56': ['accept'], '57': ['accept'], '58': ['accept'], '59': ['accept'], '67': ['accept'], '68': ['accept'], '69': ['reject'], '7': ['accept'], '72': ['reject'], '76': ['accept'], '77': ['accept']}}

The curation dict is returned from kcl.load_json function from sortingview_curation.py:

# download
if Path(uri_or_json).suffix == ".json" and not str(uri_or_json).startswith("gh://"):
    with open(uri_or_json, "r") as f:
        curation_dict = json.load(f)
else:
    try:
        import kachery_cloud as kcl
    except ImportError:
        raise ImportError(
            "To apply a SortingView manual curation, you need to have sortingview installed: "
            ">>> pip install sortingview"
        )

    try:
        curation_dict = kcl.load_json(uri=uri_or_json)
    except:
        raise Exception(f"Could not retrieve curation from SortingView uri: {uri_or_json}")

# convert to new format
if "format_version" not in curation_dict:
    curation_dict = convert_from_sortingview_curation_format_v0(curation_dict)

Is there a reason why kcl.load_json isn't returning a curation_dict with the correct format?

Also, thank you guys for doing a great job with SpikeInterface. It's been a pleasure working with it.

[zm711]

I think @alejoe91 will have to comment on this and maybe @magland. There has been work to provide a smooth translation layer for sortingview and the newer spikeinterface curation, but maybe something was missed. Is an update required in sortingview @magland? Or do we need to start troubleshooting the translation functions?

[rkim48]

Here are the relevant packages and their versions. spikeinterface 0.101.0 kachery_cloud 0.4.9 sortingview 0.13.5

[rkim48]

Some interesting observations while testing different conditions: 1) When I go to the generated sortingview url, only label units as accepted or rejected (no merging), save as snapshot and pass URI into the apply_sortingview_curation, I get the KeyError. 2) However, when I go back to sortingview and merge two random units, with the same following steps, I get no error. 3) Then, when I unmerge those two units, I get a different error: ValueError: Curation format: some labeled units are not in the unit list

I think that last error is something to do with a mismatch between the labeled_unit_set and unit_set types in curation_format/validate_curation_dict.

A quick fix for 1) is to check if the key exists and if not initialize as empty list into convert_from_sortingview_curation_format_v0:

assert destination_format == "1"
>>> if "mergeGroups" not in sortingview_dict.keys():
>>>       sortingview_dict["mergeGroups"] = []
merge_groups = sortingview_dict["mergeGroups"]

That fixes the KeyError issue (1) but now I get ValueError (3).

[zm711]

Thanks @rkim48, I think we have to wait. @samuelgarcia implemented the translation functionality, but I also wanted to ping the others who know sortingview way better than I do. So I think we just wait for them to respond!

[rkim48]

@zm711 Sure thing. I opened a pull request. Hopefully that makes things a little easier.

Edit: I think the PR is ready to be reviewed.