search

flatironinstitute / spikeforest_old

SpikeForest -- spike sorting analysis for website
Apache License 2.0
16 stars 7 forks source link

Use SortingComparison from spikeinterface. #33

Open samuelgarcia opened 5 years ago

samuelgarcia commented 5 years ago

Dear spikeforest team. the SortingComparison you use here is an old copy/paste from the fork of spiekinterface.

Recently we made strong refactoring for SortingComparison inside spiketoolkit. We clearly split in two cases : when there is a GT and when there is not. This change a bit teh code. We also put clear semantics about performance. We also make internal use of pandas wich help a lot for agragating results over sorter and recordings. We add some other measurement to detect redundant and false positive units. We made robust unitest for all piece of code (matching, counting, ...)

I would strongly encourage to use this code base for comparison for SF. And if some metrics would be missing in spiketoolkit for SF, you could make PR directly in spiketoolkit.

See:

Best,

Samuel

magland commented 5 years ago

Absolutely. That was the plan, and this seems like the time. We'll aim to do this by tuesday. Thanks Samuel. Jeremy

On Friday, May 24, 2019, Garcia Samuel notifications@github.com wrote:

Dear spikeforest team. the SortingComparison you use here is an old copy/paste from the fork of spiekinterface.

Recently we made strong refactoring for SortingComparison inside spiketoolkit. We clearly split in two cases : when there is a GT and when there is not. This change a bit teh code. We also put clear semantics about performance. We also make internal use of pandas wich help a lot for agragating results over sorter and recordings. We add some other measurement to detect redundant and false positive units. We made robust unitest for all piece of code (matching, counting, ...)

I would strongly encourage to use this code base for comparison for SF. And if some metrics would be missing in spiketoolkit for SF, you could make PR directly in spiketoolkit.

See:

Best,

Samuel

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/flatironinstitute/spikeforest/issues/33?email_source=notifications&email_token=AA4CIQBYB7TI5EMHGT5MBGLPW6VWJA5CNFSM4HPNTYP2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GVU2VYA, or mute the thread https://github.com/notifications/unsubscribe-auth/AA4CIQBYNHMEIN3UKDIRG5TPW6VWJANCNFSM4HPNTYPQ .

-- Sent from my phone

magland commented 5 years ago

@samuelgarcia. Is there an example notebook or test script for using your forked version of sorting comparison with ground truth? I tried to run this one in the SpikeInterface repo, but that seems to be broken.

AttributeError: module 'spiketoolkit.comparison' has no attribute 'compare_two_sorters'

Also, I'm wondering what workflow works best here. Should I fork your fork?

Thx

samuelgarcia commented 5 years ago

If you go for it please wait this PR: https://github.com/SpikeInterface/spiketoolkit/pull/121

maybe some example are broken, I will see.

it should be merge today. @alejoe91

magland commented 5 years ago

haven't done this yet because i'd like to simplify and make more efficient. haven't studied the code in detail at this point. @samuelgarcia

samuelgarcia commented 5 years ago

Hi @magland. Have you see this https://spiketoolkit.readthedocs.io/en/latest/comparison_example.html ?

Do you want to simplify in SI side or in SF side ?

In SI side is has been cleaned and refactor a lot. It now use pandas internally. This more convenient to aggregate many comparison for many recordings and many sorter.

There is now a GroundTruthComparison class specialized in Gt comparison. You can reah this with this fun:

comp = compare_sorter_to_ground_truth(gt_sorting, tested_sorting)
print(comp.count)
perf1 = comp.get_performance(method='by_spiketrain')
print(perf1)
perf2 = comp.get_performance(method='pooled_with_sum')
print(perf1)

comp.print_performance()
comp.print_summary()
comp.get_well_detected_units()
comp.get_false_positive_units()
comp.get_redundant_units()

Best

magland commented 5 years ago

Thanks @samuelgarcia. When I get to actually working on this part, I will try my best to work with the SI code. At this point I haven't looked at it in enough detail.