search

EnnyvanBeest / UnitMatch

Ephys toolbox to match single units (electrophysiology) either within the same day (over splits) or across two days
Other
43 stars 9 forks source link

Error in UnitMatch after extracting waveform from openephys #51

Closed Kayv-cmb closed 6 months ago

Kayv-cmb commented 7 months ago

Hello,

I encounter an error when trying to use your pipeline. I have open ephys recording and I spikesorted my data using Kilosort4. I have 1 recording. First question is UnitMatch capable of matching unit in only recording. I extracted the waveform from my openephys data using your demo Matlab file for openEphys. However I came across this error and I don't know how to solve it.

Unable to perform assignment because the size of the left side is 61-by-1 and the size of the right side is 82-by-1.

Extracting raw waveforms. Progress: 100%
Extracting raw waveforms took 27 minutes for 501 units
Extracting waveform information. Progress:   0%Unable to perform assignment because the size of the left side is 61-by-1 and the size of the right
side is 82-by-1.
Error in ExtractParameters (line 266)
        ProjectedWaveform(:,uid,cv) =
        nansum(spikeMap(:,ChanIdx,cv).*repmat(weight,1,size(spikeMap,1))',2)./sum(weight);

Error in UnitMatch (line 104)
[AllWVBParameters,param] = ExtractParameters(Path4UnitNPY,clusinfo,param);

Error in DEMO_UNITMATCH_OpenEphys (line 57)
[UniqueIDConversion, MatchTable, WaveformInfo, UMparam] = UnitMatch(clusinfo, UMparam);
EnnyvanBeest commented 7 months ago

Hi there,

This error should have been fixed by the last update on the Main branch. KS4 uses less samples to extract waveforms (61 versus 82) and therefore for KS4 users some of the parameters had to change. As long as Kilosort is stored in a folder called 'KS4' or 'Kilosort4', the pipeline should recognize you're using KS4 and initialize parameters accordingly. Alternatively you can set UMparam.Kilosortversion=4 before running UnitMatch. You can alwasy check by looking into UMparam.spikeWidth indeed is 61.

If you still have trouble when repulling the main branch, please let me know. Alternatively, if it works now, please close the issue.

Kayv-cmb commented 7 months ago

Hi thank you for the answer, I checked it was UMparam.spikeWidth indeed was 61. I changed it to 82 to see and it is working now.

However I encountered a new error after,


Extracting raw waveforms. Progress: 100%
Extracting raw waveforms took 18 minutes for 501 units
Extracting waveform information. Progress: 100%
Extracting raw waveforms and parameters took 29.6776 seconds for 501 units
Computing location distances between pairs of units...
Computing Metric similarity between pairs of units...
Computing waveform similarity between pairs of units...
Calculating waveform similarity took 0 seconds for 501 units
Computing location distances between pairs of units, per individual time point of the waveform...
Computing location distances between pairs of units, per individual time point of the waveform, Recentered...
Computing location angle (direction) differences between pairs of units, per individual time point of the waveform...
Extracting projected location took 0.89138 seconds for 501 units
Computing total score...
Error using  < 
Arrays have incompatible sizes for this operation.

Error in ExtractSimilarityMetrics (line 786)
    [muw, sw] = normfit(tmp(~isnan(tmp) & tmp<ThrsOpt));

Error in UnitMatch (line 107)
param = ExtractSimilarityMetrics(Scores2Include,AllWVBParameters,clusinfo,param);% All
Scores2Include are pushed to the workspace

Error in DEMO_UNITMATCH_OpenEphys (line 57)
[UniqueIDConversion, MatchTable, WaveformInfo, UMparam] = UnitMatch(clusinfo, UMparam);
EnnyvanBeest commented 7 months ago

Hi,

I have two suspicions for the cause of the error, both related to Kilosort4 changes.

1) Kilosort4 is so good now that it couldn't find any oversplits, and therefore ThrsOpt was empty when running it on just 1 session. I have never had this before! I've added a safety mechanism which should hopefully help on the workinprogress branch. Could you please try the latest version of the work in progress branch? I will try to change some things there and once you confirmed it works, I can pull these changes to the main branch. 2) Did you use UnitMatch's ExtractAndSaveAverageWaveforms to extract the raw waveforms? I haven't yet checked if the waveforms come out okay for KS4 sorted data (we use waveform extraction from Bombcell). Could you please show me for a good unit what the waveform looks like? Perhaps the waveform samples that are extracted are not optimal for UnitMatch.

Thanks!

Kayv-cmb commented 7 months ago

I try the version in work in progress and got to this warning

`Extracting raw waveforms. Progress: 100%
Extracting raw waveforms took 41 minutes for 501 units
Extracting waveform information. Progress: 100%
Extracting raw waveforms and parameters took 52.2044 seconds for 501 units
Computing location distances between pairs of units...
Computing Metric similarity between pairs of units...
Computing waveform similarity between pairs of units...
Calculating waveform similarity took 0 seconds for 501 units
Computing location distances between pairs of units, per individual time point of the waveform...
Computing location distances between pairs of units, per individual time point of the waveform, Recentered...
Computing location angle (direction) differences between pairs of units, per individual time point of the waveform...
Extracting projected location took 0.84783 seconds for 501 units
Computing total score...
Warning: Cannot identify any larger than threshold... `

And If I continue there is this error 


priorMatch =

     1

Creating the Naive Bayes model...
Undefined function 'smooth' for input arguments of type 'double'.

Error in CreateNaiveBayes (line 44)
        Parameterkernels(:,scid,Ck) =
        smooth(histcounts(ScoresTmp(label==Cond(Ck)),Edges),Smoothtmp);

Error in RunNaiveBayes (line 90)
            [Parameterkernels,Performance] = CreateNaiveBayes(Tbltmp,label,Priors);

Error in UnitMatch (line 110)
[MatchProbability,label,Tbl,BestMdl] =
RunNaiveBayes(Predictors,TotalScore,Scores2Include,clusinfo,param,SortingOrder,EuclDist);

Error in DEMO_UNITMATCH_OpenEphys (line 58)
[UniqueIDConversion, MatchTable, WaveformInfo, UMparam] = UnitMatch(clusinfo, UMparam);

I highly doubt that there is no over splits in Kilosort even Kilosort4. And I only use UnitMatch_Open_ephys so I would say I used ExtractAndSaveAverageWaveforms to extract the raw waveforms

EnnyvanBeest commented 7 months ago

In that case I think it's a problem of waveform extraction. Could you please rerun (after pulling the work in progress branch) with UMparam.RedoExtraction=1? This should generate new waveforms, more compatible with KS4 (if you have UMparam.Kilosortversion=4).

Kayv-cmb commented 6 months ago

I now have an error with memory, that I didn't get with the previous version. I am running on Mac and I think it's a function exclusive to windows. I am not sure to understand what is happening

Extracting raw waveforms. Progress:   0%Error using memory
Function MEMORY is not available on this platform.
EnnyvanBeest commented 6 months ago

you're absolutely right, the function memory doesn't work on Mac. Changed it now!

Kayv-cmb commented 6 months ago

It took a very long time but it is working I was able to complete the Demo with the new extracted waveforms.

However on a new recording I get this error Same type of data, extracted with Kilosort4 but it is another recording


Unrecognized function or variable 'channelpostmpconv'.

Error in ExtractKilosortData (line 260)
    if size(channelpostmp,1) == size(channelpostmpconv,1) &&
    any(channelpostmpconv(:)~=channelpostmp(:))

Error in DEMO_UNITMATCH_OpenEphys (line 55)
UMparam = ExtractKilosortData(UMparam.KSDir, UMparam); % Extract KS data and do some noise removal,
optionally decompresses cbin to bin data and uses BOMBCELL quality metric to define good single
units
EnnyvanBeest commented 6 months ago

Hi,

Re. duration of extracting waveforms: Normally I check if there's enough memory and then decide, but for now I made it serial extraction for Mac. If you have enough RAM you can force using the memmap extraction rather than the serial extraction in the extractwaveform script in the future.

Re. ExtractKilosortData, there were some incompatibilities still because not all versions of Kilosort save out channel data the same way. I was just struggling with this myself as I now also started using KS4 which also do not store the empty channels, causing incomplete channelmaps. I hope the recent push has fixed it.

There might be some hick-ups further along the path that I will look into while coming across them, but please also keep reporting the issues you find. This will help me speed up making UnitMatch compatible with KS4.

Thanks!

Kayv-cmb commented 6 months ago

Thanks a lot for all the help!!! It seems to work with my new KS4 data (currently extracting the data). I have small question :

Is there a way to extract the unit found as merge by UnitMatch. and export to phy the curated sorting?

EnnyvanBeest commented 6 months ago

Currently this is not a feature. The only way for now is to save out/display the original IDs for merges (for example by just selecting everything with the same UniqueID from within the same session) and look for their original IDs (=clusid) in phy, and then decide to merge them.

I personally solve merges by just using UniqueID instead of original clusterID downstream of UnitMatch, for example by merging spikes in my downstream analysis by extracting all spikes from clusterIDs (=Kilosort) that have the same UniqueID (=UnitMatch), if that makes sense?

Kayv-cmb commented 6 months ago

Thank you very much, that would be a very cool feature if we could transform the merge to phy format.

I will close the issue since it has been solved