magland
commented
4 years ago The short answer is that spikeforest does not compute such a metric. But the more helpful answer is that such information is available with a relatively short python script. For each ground truth unit we have the accuracy computed across all the sorters. So I think what you are suggesting is to take the max over all sorters.
On Mon, Nov 4, 2019 at 2:57 PM Yaroslav Halchenko notifications@github.com wrote:
I don't remember if I saw something like that on spikeinterface's SfN poster, or not... and I think there is nothing like that reported on spikeforest.
I am interested to get some metric which would reflect amount of "recovered" units across all spike sorters, which would hint that some joint/meta spike sorting using multiple algorithms could be of benefit. E.g. it could be "joint accuracy" -- if we take 2 (3, 4, ...) algorithms, "cherry pick" correctly made decisions across them, estimate accuracy. And then report max accuracy for 2, 3, 4 ... I hope that at some point that "curve" arrives to 100%. But it would be interesting to see if it does for any specific dataset, and which critical value (# of spike sorters, and which bundle) necessary to potentially identify all units. If it doesn't reach 100% -- report the top % (with all of the spike sorters)
Such metric could be useful to e.g. hint on idiosyncracies of spike sorters vs of data itself (noise etc). E.g. if there is a dataset with 100% accuracy achievable with 2 sorters, when any individual one does 70%, we would know that data is good, and algorithms are picking up really based on different features of units. With a dataset which obtains top level accuracy not far from max individual sorter - we would know that it is likely data (noise, or some aspects not covered by any sorter).
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/flatironinstitute/spikeforest/issues/78?email_source=notifications&email_token=AA4CIQH4QE44H6J3NJJ3ZFDQSB5BZA5CNFSM4JIYMZ7KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HWWM6KQ, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4CIQEMOCFUTDC2IRW5SWDQSB5BZANCNFSM4JIYMZ7A .
yarikoptic
shouldn't 
samuelgarcia
I don't remember if I saw something like that on spikeinterface's SfN poster, or not... and I think there is nothing like that reported on spikeforest.
I am interested to get some metric which would reflect amount of "recovered" units across all spike sorters, which would hint that some joint/meta spike sorting using multiple algorithms could be of benefit. E.g. it could be "joint accuracy" -- if we take 2 (3, 4, ...) algorithms, "cherry pick" correctly made decisions across them, estimate accuracy. And then report max accuracy for 2, 3, 4 ... I hope that at some point that "curve" arrives to 100%. But it would be interesting to see if it does for any specific dataset, and which critical value (# of spike sorters, and which bundle) necessary to potentially identify all units. If it doesn't reach 100% -- report the top % (with all of the spike sorters)
Such metric could be useful to e.g. hint on idiosyncracies of spike sorters vs of data itself (noise etc). E.g. if there is a dataset with 100% accuracy achievable with 2 sorters, when any individual one does 70%, we would know that data is good, and algorithms are picking up really based on different features of units. With a dataset which obtains top level accuracy not far from max individual sorter - we would know that it is likely data (noise, or some aspects not covered by any sorter).