recording and ground truth not match when template & recording frequencies are different

[ZoeChen96]

Hi there, sorry for troubling again.

I observe that there is a small problem for the recording when template freq and recording freq are different. Here is an example:

As shown in the image, you can see that the high frequency recording always delay than the low frequency recording. However the ground truth is always the same absolute time value. That will make the recording & ground truth not match with each other. (PS: the above recording are generated with 240k/40k/32k templates from left to right. In each template, I generated 4 recording @ 20k/30k/40k/240kHz)

I had a closer look at it. I found that when the recording & template frequency matches, the ground truth and recording will match. As you can see from the next two pictures (240kHz recording + 240kHz templates, and 40kHz recording +40kHz templates):

However, when the recording and template frequency are different, the ground truth and recording does not match. As you can see from the next two pictures (30kHz recording + 240kHz templates, and 240kHz recording +40kHz templates):

Can you please have a look at this problem? Thanks!

BTW： I am using the 1.8.0.dev version, which already corrects the problem of templates https://github.com/alejoe91/MEArec/pull/134

[ZoeChen96]

I attached my script for generating recording here:

import MEArec as mr
import scipy.io
import spikeinterface.extractors as se
import warnings
from pprint import pprint
import sys
warnings.simplefilter("ignore")
############ set global parameters ####################
original_stdout = sys.stdout # Save a reference to the original standard output
probe       = 'Neuropixels2-128'
n_exc       = 48
n_inh       = 12
duration    = 60 #seconds
filtering   = False #filtering True/false
noise_type  = 'far-neurons' #uncorrelated/distance-correlated/far-neurons
recording_params = mr.get_default_recordings_params()
recording_params['spiketrains']['n_exc'] = n_exc
recording_params['spiketrains']['n_inh'] = n_inh
recording_params['spiketrains']['duration'] = duration
recording_params['spiketrains']['seed'] = 0
recording_params['templates']['min_amp'] = 40
recording_params['templates']['max_amp'] = 300
recording_params['templates']['seed'] = 0
recording_params['recordings']['modulation'] = 'electrode'
recording_params['recordings']['noise_mode'] = noise_type
#recording_params['recordings']['noise_color'] = False #NEW, FIND THE NOISE CAN'T BE WHITE NOW! CAN TRY FOR LAST VERSION
recording_params['recordings']['noise_level'] = 10
recording_params['recordings']['chunk_conv_duration'] = 20
recording_params['recordings']['chunk_noise_duration'] = 20
recording_params['recordings']['chunk_filter_duration'] = 20
recording_params['recordings']['seed'] = 0
#filter settings
recording_params['recordings']['filter'] = filtering
recording_params['recordings']['filter_cutoff'] = [300, 6000] # highpass from 0
#seeds settings
recording_params['seeds']['spiketrains'] = 0
recording_params['seeds']['templates'] = 0
recording_params['seeds']['convolution'] = 0
recording_params['seeds']['noise'] = 0
########## load templates ###################
dir_full_templates = 'new_templates300_40kHz_minampnull_'+ probe +'.h5'
tempgen = mr.load_templates(dir_full_templates)
############## set looping parameters and generate datasets ############
fs          = [20000,30000,40000,240000]
#fs          = [240000]
for i in fs:
    print('generating recording for fs', str(i))
    recording_params['recordings']['fs'] = i
    set_name = 'set60dur_40kt_'+str(i)+'fs_nofilt'
    dir_full_recordings = set_name + '/'+set_name+ '_recording.h5'
    dir_full_gttimes = set_name + '/'+set_name+ '_gttimes.mat'
    filename =         set_name  + '/' + set_name +'_info.txt'
    recgen = mr.gen_recordings(templates=dir_full_templates, params=recording_params, verbose=True)
    # save recording
    mr.save_recording_generator(recgen, dir_full_recordings)
    #save gt_times
    if (n_exc+n_inh >0):
        sorting_GT = se.MEArecSortingExtractor(dir_full_recordings)
        scipy.io.savemat(dir_full_gttimes, {'gt_times': sorting_GT.to_spike_vector()})
    with open(filename, "w") as f:    
        sys.stdout = f # Change the standard output to the file we created. 
        pprint(recgen.info)
    sys.stdout = original_stdout

[ZoeChen96]

And the scripts for generating templates here:

import MEArec as mr
from pprint import pprint
import sys
#settings
probe = 'Neuropixels2-128'              #available: CLI: mearec available-probes
#folders
original_stdout = sys.stdout # Save a reference to the original standard output
templates_params = mr.get_default_templates_params()
cell_models_folder = mr.get_default_cell_models_folder()
#parameters
templates_params['probe'] = probe
templates_params['n'] = 300                 #for every cell model (in total 13 cell model), generate 300 templates
templates_params['seed']=0
templates_params['min_amp']=0

freq = [32,40,240]
for i in freq:
    dir_full_templates = 'new_templates300_'+str(i)+'kHz_minampnull_'+ probe +'.h5'
    filename = 'new_templates300_'+str(i)+'kHz_minampnull_'+ probe +'_info.txt'
    templates_params['dt']=1/i
    tempgen = mr.gen_templates(cell_models_folder=cell_models_folder, templates_tmp_folder = 'templates_32', params=templates_params, verbose=True)
    mr.save_template_generator(tempgen=tempgen, filename=dir_full_templates)
    with open(filename, "w") as f:    
        sys.stdout = f # Change the standard output to the file we created. 
        pprint(tempgen.info)
    sys.stdout = original_stdout

[alejoe91]

@ZoeChen96 no trouble at all! thanks for stress testing the resampling option. Taking a look now (I thinkI know where the problem is ;) )

[alejoe91]

@ZoeChen96 I can reproduce the error. Here are some traces resampled at different freqs (original is 32Khz):

The problem was that the initial padding was taken from the indicated fs, and not form the template spike_fs. I pushed a fix here #141, can you test it out? Here is how the same traces look with the bug fix:

[ZoeChen96]

Hi Alessio, thanks for the response! I just tested. I am not very familiar with git, I quickly checked and do: git checkout 1980fd758beafe87c77367ce27ca0b8da52a50de to switch to the commit https://github.com/alejoe91/MEArec/pull/141 , I hope I did it correctly. Then I plot this:

I think now it's perfect. Thank you very much!