How to run multiple tif files of the same session

[alsrikywn]

For better support, please use the template below to submit your issue. When your issue gets resolved please remember to close it.

Sometimes errors while running CNMF occur during parallel processing which prevents the log to provide a meaningful error message. Please reproduce your error with setting dview=None.

If you need to upgrade CaImAn follow the instructions given in the documentation.

• Tell us a bit about your setup:

  1. Operating system (Linux/macOS/Windows): windows10

  2. Python version (3.x): 3.7

  3. Working environment (Python IDE/Jupyter Notebook/other): Jupyter notebook

  4. Which of the demo scripts you're using for your analysis (if applicable): demo_multisession_registration

  5. CaImAn version*: 1.8.9

  6. CaImAn installation process (pip install ./pip install -e ./conda): conda-forge

*You can get the CaImAn version by creating a params object and then typing params.data['caiman_version']. If the field doesn't exist, type N/A and consider upgrading)

• Describe the issue that you are experiencing Hello, I'm trying to use the caiman for my two-photon imaging data analysis. I have mutiple tiff files on the same session because the total file size would be more than 6GB. Also, my behavioral paradigm requires to do imaging for multiple days. I found that I can use multisession_registration in this case however, I am completely new to use python so I don't know how to replace the code for my actual data. I ran the code as below:

file_path = 'D:\CSA-2PI\demo' #download_demo('alignment.pickle')
infile = open(file_path,'rb')
data = pickle.load(infile)
infile.close()

spatial = data[0]
templates = data[1]
dims = templates[0].shape

And I got an error message

PermissionError Traceback (most recent call last)

in 1 # Load multisession data (spatial components and mean intensity templates) (should be replaced by actual data) 2 file_path = 'D:\CSA-2PI\demo' #download_demo('alignment.pickle') ----> 3 infile = open(file_path,'rb') 4 data = pickle.load(infile) 5 infile.close() PermissionError: [Errno 13] Permission denied: 'D:\\CSA-2PI\\demo'

This permission denied error message can be usually occurred when the path or filename is wrong in windows. But I have no idea how to change the code for opening multiple files at the same time. Also, the demofile format is "pickle" but my actual data format is "tif". Can you let me know how to change the code as below? data = pickle.load(infile)

Thanks for your help in previous!

[j-friedrich]

Hi @alsrikywn,

if you want to analyze the video data of one session that's split into multiple tif files, just provide a list of filenames when generating the CNMFParams object.

If you analyzed 2 sessions using caiman and now want to combine the results, use register_multisession

Here some demo code illustrating the 2 scenarios

import caiman as cm
from caiman.source_extraction.cnmf import cnmf, params
from caiman.utils.utils import download_demo
from caiman.base.rois import register_multisession

# split demo data into 2 tif files for sake of illustration
cm.load(download_demo('demoMovie.tif'), subindices=range(1000)).save('session1.tif')
cm.load(download_demo('demoMovie.tif'), subindices=range(1000, 2000)).save('session2.tif')

# Scenario 1: one session split into multiple tif files
# provide a list of files to automatically concatenate the video data
fnames = ['session1.tif', 'session2.tif']
params_dict = {'fnames': fnames}
opts = params.CNMFParams(params_dict=params_dict)
cnm = cnmf.CNMF(n_processes=2, params=opts)
cnm = cnm.fit_file()

# Scenario 2: Multisession registration
# analyze 1st session
fname1 = ['session1.tif']
params_dict1 = {'fnames': fname1}
opts1 = params.CNMFParams(params_dict=params_dict1)
cnm1 = cnmf.CNMF(n_processes=2, params=opts1)
cnm1 = cnm1.fit_file()

# analyze 2nd session
fname2 = ['session2.tif']
params_dict2 = {'fnames': fname2}
opts2 = params.CNMFParams(params_dict=params_dict2)
cnm2 = cnmf.CNMF(n_processes=2, params=opts2)
cnm2 = cnm2.fit_file()

# Multisession registration
spatial = [cnm1.estimates.A, cnm2.estimates.A]
templates = [cm.load('session1.tif').mean(0), cm.load('session2.tif').mean(0)]
dims = templates[0].shape

spatial_union, assignments, matchings = register_multisession(A=spatial, dims=dims, templates=templates)

For saving and loading analysis results see issue #860

[alsrikywn]

Hi @j-friedrich, I followed your code for scenario 1 first to analyze one session split into multiple tif files, unfortuneately I got an error messeage in opts = params.CNMFParams(params_dict=params_dict)

Here's the message:

Exception Traceback (most recent call last)

in 24 'cnn_lowest': cnn_lowest} 25 ---> 26 opts = params.CNMFParams(params_dict=opts_dict) 27 cnm = cnmf.CNMF(n_processes=2, params=opts) 28 cnm = cnm.fit_file() ~\Anaconda3\envs\caiman\lib\site-packages\caiman\source_extraction\cnmf\params.py in __init__(self, fnames, dims, dxy, border_pix, del_duplicates, low_rank_background, memory_fact, n_processes, nb_patch, p_ssub, p_tsub, remove_very_bad_comps, rf, stride, check_nan, n_pixels_per_process, k, alpha_snmf, center_psf, gSig, gSiz, init_iter, method_init, min_corr, min_pnr, gnb, normalize_init, options_local_NMF, ring_size_factor, rolling_length, rolling_sum, ssub, ssub_B, tsub, block_size_spat, num_blocks_per_run_spat, block_size_temp, num_blocks_per_run_temp, update_background_components, method_deconvolution, p, s_min, do_merge, merge_thresh, decay_time, fr, min_SNR, rval_thr, N_samples_exceptionality, batch_update_suff_stat, expected_comps, iters_shape, max_comp_update_shape, max_num_added, min_num_trial, minibatch_shape, minibatch_suff_stat, n_refit, num_times_comp_updated, simultaneously, sniper_mode, test_both, thresh_CNN_noisy, thresh_fitness_delta, thresh_fitness_raw, thresh_overlap, update_freq, update_num_comps, use_dense, use_peak_max, only_init_patch, var_name_hdf5, max_merge_area, use_corr_img, params_dict) 879 } 880 --> 881 self.change_params(params_dict) 882 883 ~\Anaconda3\envs\caiman\lib\site-packages\caiman\source_extraction\cnmf\params.py in change_params(self, params_dict, verbose) 1061 if flag: 1062 logging.warning('No parameter {0} found!'.format(k)) -> 1063 self.check_consistency() 1064 return self ~\Anaconda3\envs\caiman\lib\site-packages\caiman\source_extraction\cnmf\params.py in check_consistency(self) 888 self.data['last_commit'] = '-'.join(caiman.utils.utils.get_caiman_version()) 889 if self.data['dims'] is None and self.data['fnames'] is not None: --> 890 self.data['dims'] = get_file_size(self.data['fnames'], var_name_hdf5=self.data['var_name_hdf5'])[0] 891 if self.data['fnames'] is not None: 892 if isinstance(self.data['fnames'], str): ~\Anaconda3\envs\caiman\lib\site-packages\caiman\source_extraction\cnmf\utilities.py in get_file_size(file_name, var_name_hdf5) 1059 else: 1060 dims, T = zip(*[get_file_size(fn, var_name_hdf5=var_name_hdf5) -> 1061 for fn in file_name]) 1062 if len(set(dims)) > 1: 1063 raise Exception('Files have FOVs with different sizes') ~\Anaconda3\envs\caiman\lib\site-packages\caiman\source_extraction\cnmf\utilities.py in (.0) 1059 else: 1060 dims, T = zip(*[get_file_size(fn, var_name_hdf5=var_name_hdf5) -> 1061 for fn in file_name]) 1062 if len(set(dims)) > 1: 1063 raise Exception('Files have FOVs with different sizes') ~\Anaconda3\envs\caiman\lib\site-packages\caiman\source_extraction\cnmf\utilities.py in get_file_size(file_name, var_name_hdf5) 1048 dims = tuple(dims) 1049 else: -> 1050 raise Exception('File not found!') 1051 elif isinstance(file_name, tuple): 1052 from ...base.movies import load Exception: File not found!

I would like to ask if you help me again with this issue. Thanks.

[j-friedrich]

You need to provide the correct paths of your tif files in the list fnames

[jesusdpa1]

Hi @j-friedrich, could you explain further the multisession registration?

Currently I have processed our datasets per day, each day is composed by multiple tif files that get motion corrected aligned and then the components get extracted. This gives me the following output:

When looking the code for the scenario 2, I see that the tiff files are indicated again, and then the mean(0) gets calculated. My first question is if this file represents the maximum intensity projection or the complete movie? If not what files from the two already processed days should be used to run the multi-session registration

# Scenario 2: Multisession registration
# analyze 1st session
fname1 = ['session1.tif']
params_dict1 = {'fnames': fname1}
opts1 = params.CNMFParams(params_dict=params_dict1)
cnm1 = cnmf.CNMF(n_processes=2, params=opts1)
cnm1 = cnm1.fit_file()

# analyze 2nd session
fname2 = ['session2.tif']
params_dict2 = {'fnames': fname2}
opts2 = params.CNMFParams(params_dict=params_dict2)
cnm2 = cnmf.CNMF(n_processes=2, params=opts2)
cnm2 = cnm2.fit_file()

# Multisession registration
spatial = [cnm1.estimates.A, cnm2.estimates.A]
templates = [cm.load('session1.tif').mean(0), cm.load('session2.tif').mean(0)]
dims = templates[0].shape

spatial_union, assignments, matchings = register_multisession(A=spatial, dims=dims, templates=templates)

[jesusdpa1]

I also notice that some of our recordings are move in space between recording days, is there a way to compensate for this?

green color: day01
blue color: day02

[j-friedrich]

I used the complete movie file and calcuated the mean image as template, hence the mean(0). If you have the max projection image you can use that as template as well.

You could compensate for the misalignment by cropping the FOV beforehand. E.g crop at the right side for day 1 and the left side for day 2.

Remaining misalignments are corrected by register_multisession as long as align_flag=True. Here the example with the 2 sessions misaligned by 5 pixels:

import caiman as cm
from caiman.source_extraction.cnmf import cnmf, params
from caiman.utils.utils import download_demo
from caiman.utils.visualization import plot_contours
from caiman.base.rois import register_multisession
import matplotlib.pyplot as plt
from matplotlib.lines import Line2D
import numpy as np

# split demo data into 2 tif files for sake of illustration
cm.load(download_demo('demoMovie.tif'), subindices=range(1000))[...,5:].save('session1.tif')
cm.load(download_demo('demoMovie.tif'), subindices=range(1000, 2000))[...,:-5].save('session2.tif')

# analyze 1st session
fname1 = ['session1.tif']
params_dict1 = {'fnames': fname1, 'K': 10}
opts1 = params.CNMFParams(params_dict=params_dict1)
cnm1 = cnmf.CNMF(n_processes=2, params=opts1)
cnm1 = cnm1.fit_file()

# analyze 2nd session
fname2 = ['session2.tif']
params_dict2 = {'fnames': fname2, 'K': 10}
opts2 = params.CNMFParams(params_dict=params_dict2)
cnm2 = cnmf.CNMF(n_processes=2, params=opts2)
cnm2 = cnm2.fit_file()

# Multisession registration
spatial = [cnm1.estimates.A, cnm2.estimates.A]
templates = [cm.load('session1.tif').mean(0), cm.load('session2.tif').mean(0)]
dims = templates[0].shape

spatial_union, assignments, matchings = register_multisession(
    A=spatial, dims=dims, templates=templates)

# Plots
plt.figure(figsize=(10,4))
plt.subplot(121)
plot_contours(spatial[0], templates[0])
plt.title('Session 1')
plt.subplot(122)
plot_contours(spatial[1], templates[1])
plt.title('Session 2')

session1 = np.isnan(assignments[:,1])
session2 = np.isnan(assignments[:,0])
both = np.sum(~np.isnan(assignments), axis=1) >= 2

plt.figure()
plot_contours(spatial_union[:, both], templates[-1])
plot_contours(spatial_union[:, session1], templates[-1], colors='r')
plot_contours(spatial_union[:, session2], templates[-1], colors='b')
plt.legend(handles=[Line2D([], [], color='w', label='Both sessions'),
    Line2D([], [], color='r', label='Only 1st session'),
    Line2D([], [], color='b', label='Only 2nd session')])