klust_import may not be creating empty datasets for zero-spike trials

[dmeliza]

Need to check whether trials with no spikes lead to empty datasets or to missing datasets. This is important because missing datasets may also imply excluded trials.

[dmeliza]

A bit more explanation and history.

The bug occurs when trials are excluded because of excessive noise. This helps in spike sorting because you don't have as many artifact spikes. But when the spikes are extracted from klusters format and turned into toelis or whatever, those exclusions need to be remembered so that empty trials don't appear as trials where there weren't any spikes.

This bug wasn't caught until version 2 of mspikes (specifically revision 3c51b69), on 12/2010. The fix was to add a list of skipped epochs to the klusters xml file, which was then used to correctly skip those trials when reading the sorted spikes back in. So any data prior to 2010 may incorrectly include some empty trials.

The patch for 3.0 should use the same or a similar mechanism, so that xml files generated by previous versions of mspikes are processed correctly.

[dmeliza]

The 3.0 exclusion model is as follows:

Filter classes (e.g. zscale) identify recording segments where the RMS exceeds some threshold (which is calculated relative to a sliding window). These exclusions become events in a separate channel of the datastream. The standard workflow is to store the spikes and exclusion data in a separate arf file, export the data to klusters format, and then import sorted spikes into the same arf file, in which case the exclusion information is preserved with the sorted spikes.

So the default behavior of klust_import is okay (but needs to be documented), although it should create exclusion datasets when reading klusters xml files from older versions of mspikes.

[dmeliza]

This is kind of a pain to solve, because mspikes 2 labeled the excludes based on a "recid" attribute stored on the entry. There'd have to be some kind of global check to appropriately exclude those intervals, and it's not worth it for what's a fairly rare case. Better to notify the user that there are exclusions in the data and use a separate tool.