Motion correction not working after high pass spatial filter

[guidomeijer]

When I apply a high pass spatial filter I can't do motion correction afterwards. The script just hangs during motion correction without providing any feedback, error or progress bar. When I substitute the high pass spatial filter for a common reference the motion correction does work. Here is my code:

            # Load in recording         
            if len(glob(join(probe_path, '*.cbin'))) > 0:
                # Recording is already compressed by a previous run, loading in compressed data
                rec = se.read_cbin_ibl(probe_path)
            else:
                rec = se.read_spikeglx(probe_path, stream_id=f'imec{split(probe_path)[-1][-1]}.ap')

            # Pre-process 
            print('Applying high-pass filter.. ', end='')
            rec_filtered = spre.highpass_filter(rec)
            print('Done\nCorrecting for phase shift.. ', end='')
            rec_shifted = spre.phase_shift(rec_filtered)
            print('Done\nDetecting and interpolating over bad channels.. ', end='')
            bad_channel_ids, all_channels = spre.detect_bad_channels(rec_shifted)
            rec_interpolated = spre.interpolate_bad_channels(rec_shifted, bad_channel_ids)
            print('Done\nDestriping.. ', end='')
            rec_destriped = spre.highpass_spatial_filter(rec_interpolated)
            print('Done\nPerforming motion correction.. ', end='')
            rec_corrected = spre.correct_motion(rec_destriped, preset='nonrigid_accurate',
                                                n_jobs=-1, progress_bar=True)
            print('Done')

To check whether something strange is happening during the high pass spatial filter I made some plots but they look fine to me:

I'd like to use the spatial filter instead of the common reference so any help on how to get the motion correction to work with the spatial filter would be appreciated!

[alejoe91]

@guidomeijer

Can you try to pre-cache the highpass-spatial filtered recording to see if that fixes the motion correction?

Just add:

rec_destriped = rec_destriped.save(folder=destriped", n_jobs=...)

before the motion correction step. If that also hangs, then it must be something with the highpass spatial filter parallelization performance

[guidomeijer]

Yes, now it hangs at the destriping step.

[alejoe91]

Can you try to run it with n_jobs=1? Then we know is the parallelization of it

[JoeZiminski]

Hey @guidomeijer! hope things are good. I had a go and replicating now from main branch with the below:

import spikeinterface.full as si
from spikeinterface.postprocessing.amplitude_scalings import _plot_collisions, _plot_one_collision
# Generate curated sorting

if __name__ == '__main__':  # I am on windows 
    recording, _ = si.generate_ground_truth_recording(durations=[100], num_channels=384)

    recording = si.bandpass_filter(recording)
    recording = si.highpass_spatial_filter(recording)
    recording = si.correct_motion(recording, preset='nonrigid_accurate', n_jobs=-1, progress_bar=True)

I wonder if you still see this error on main branch, either with the MRE above or your original dataset?