Different shape of operands in CNMF

[rintrah]

For better support, please use the template below to submit your issue. When your issue gets resolved please remember to close it.

Sometimes errors while running CNMF occur during parallel processing which prevents the log to provide a meaningful error message. Please reproduce your error with setting dview=None.

If you need to upgrade CaImAn follow the instructions given in the wiki.

• Tell us about your operating system (Linux/macOS/Windows), Python version (3.x), and working environment (Spyder/Jupyter Notebook/other)

Ubuntu 18.04.2 LTS, Python Python 3.6.8, Jupyter Notebook

• Describe the issue that you are experiencing

Dear all,

I am using the demo_pipeline_cnmfE.ipynb to perform motion correction and source extraction using the CNMF-E algorithm on a 1-photon lightsheet data. While I have made small changes (load the 1-p data and change the size of the high pass filter), I keep the values of other parameters, e.g. tsub = 2, ssub = 1, ssub_B=2, etc. The data is a single tif file of 454x352x5000 (Yr.shape=(159808, 5000), dims=(352, 454), and T=5000) and the motion correction went fine. However, when I execute the following lines of code cnm = cnmf.CNMF(n_processes=n_processes, dview=dview, Ain=Ain, params=opts) cnm.fit(images) I obtain the following error

• Copy error log below `

  ValueError Traceback (most recent call last)

  in 11 'rval_thr': r_values_min, 12 'use_cnn': False}) ---> 13 cnm.estimates.evaluate_components(images, cnm.params, dview=dview) 14 15 print(' ***** ')

~/anaconda3/lib/python3.6/site-packages/caiman/source_extraction/cnmf/estimates.py in evaluate_components(self, imgs, params, dview) 831 thresh_cnn_lowest=opts['cnn_lowest'], 832 r_values_lowest=opts['rval_lowest'], --> 833 min_SNR_reject=opts['SNR_lowest']) 834 self.idx_components = idx_components 835 self.idx_components_bad = idx_components_bad

~/anaconda3/lib/python3.6/site-packages/caiman/components_evaluation.py in estimate_components_quality_auto(Y, A, C, b, f, YrA, frate, decay_time, gSig, dims, dview, min_SNR, r_values_min, r_values_lowest, Npeaks, use_cnn, thresh_cnn_min, thresh_cnn_lowest, thresh_fitness_delta, min_SNR_reject, gSig_range) 523 -min_SNR_reject) * Nsamples 524 --> 525 traces = C + YrA 526 527 , _, fitnessraw, , r_values = estimate_components_quality(

ValueError: operands could not be broadcast together with shapes (534,5000) (535,5000) `

I understand that the problem lies with the shapes of C and YrA, but I cannot find where I have to make the change so the program runs properly. I have just started to use CaImAn and my knowledge of the documentation is still incomplete. However, I have go through the issues in the gitHub without finding a solution to my problem.

Many thanks for considering my request.

Best wishes

• If you're not reporting an error, type your message below

[epnev]

@rintrah That's a weird one. What do you get if you type:

print(cnm.estimates.C.shape)
print(cnm.estimates.YrA.shape)

I'm trying to understand where the shape discrepancy occurs.

[rintrah]

Dear Eftychios,

Thank you for your reply.

When I typed print(print(cnm.estimates.C.shape); print(cnm.estimates.YrA.shape), I obtained (536, 5000), (539, 5000).

Best wishes,

[epnev]

What about the value of cnm.estimates.merged_ROIs?

[rintrah]

Dear Eftychios,

The value of cnm.estimates.merged_ROIs is [] (empty).

Best wishes,

[epnev]

I'm not sure why this is happening. Do you see this error with all your files or just that?

[rintrah]

This is the first of my own files that I have used with CaImAn. The demo included in the program worked without any problem.

[epnev]

@rintrah If you can truncate the file to a smaller size where you still face this problem and send it over I'll be happy to take a look when I get a chance.

[rintrah]

OK. I see if I can get the same problem with a smaller file.

[rintrah]

Dear Eftychios,

A small update.

I created a small file (639MB) and the extraction worked fine. I will try with a bigger tiff file.

The main differences (beside the size) were that I followed your suggestion in the wiki (on file types and sizes). My previous file was a ImageJ type tif file. The other thing is that since I followed the demo demo_pipeline_cnmfE.py merge_thresh should be changed to merge_thr (as was indicated in the issue merge_thr not being properly assigned in CNMF-E notebook demo). While I am going to try a large file now, I think that this information might be useful.

Best wishes

[epnev]

@rintrah thanks for catching that typo! Let me know if the larger file worked without that error.

[rintrah]

Dear Eftychios,

An update.

I have tried a file of 3000 frames. Now I have an error in the inspect_correlation_pnr function (see attached code). Now I am going through the issues to see if this problem has been previously addressed.

Best wishes,

`--------------------------------------------------------------------------- ValueError Traceback (most recent call last) ~/Code/Python/ImageProcessing/extractNeuronsCNMF.py in 161 162 if name=="main": --> 163 main() 164

~/Code/Python/ImageProcessing/extractNeuronsCNMF.py in main() 118 cn_filter, pnr = cm.summary_images.correlation_pnr(images, gSig=gSig[0], swap_dim=False) 119 # inspect the summary images and set the parameters --> 120 inspect_correlation_pnr(cn_filter, pnr) 121 # print parameters set above, modify them if necessary based on summary images 122 print(min_corr) # min correlation of peak (from correlation image)

~/anaconda3/lib/python3.6/site-packages/caiman/utils/visualization.py in inspect_correlation_pnr(correlation_image_pnr, pnr_image) 1012 1013 s_cn_max = Slider(pl.axes([0.05, 0.01, 0.35, 0.03]), 'vmax', -> 1014 correlation_image_pnr.min(), correlation_image_pnr.max(), valinit=correlation_image_pnr.max()) 1015 s_cn_min = Slider(pl.axes([0.05, 0.07, 0.35, 0.03]), 'vmin', 1016 correlation_image_pnr.min(), correlation_image_pnr.max(), valinit=correlation_image_pnr.min())

~/anaconda3/lib/python3.6/site-packages/matplotlib/widgets.py in init(self, ax, label, valmin, valmax, valinit, valfmt, closedmin, closedmax, slidermin, slidermax, dragging, valstep, **kwargs) 344 self.valfmt = valfmt 345 ax.set_yticks([]) --> 346 ax.set_xlim((valmin, valmax)) 347 ax.set_xticks([]) 348 ax.set_navigate(False)

~/anaconda3/lib/python3.6/site-packages/matplotlib/axes/_base.py in set_xlim(self, left, right, emit, auto, xmin, xmax) 3225 3226 self._process_unit_info(xdata=(left, right)) -> 3227 left = self._validate_converted_limits(left, self.convert_xunits) 3228 right = self._validate_converted_limits(right, self.convert_xunits) 3229

~/anaconda3/lib/python3.6/site-packages/matplotlib/axes/_base.py in _validate_converted_limits(self, limit, convert) 3137 if (isinstance(converted_limit, Real) 3138 and not np.isfinite(converted_limit)): -> 3139 raise ValueError("Axis limits cannot be NaN or Inf") 3140 return converted_limit 3141

ValueError: Axis limits cannot be NaN or Inf `

[rintrah]

Sorry I closed the issue by mistake.

[epnev]

@rintrah What is the value of np.sum(np.isnan(cn_filter))? If it's non-zero it means that your correlation image contains NaN values, possibly because there are pixels that have constant value across your experiment. You can use something like cn_filter(np.isnan(cn_filter)) = 0 before plotting them to remove them. Does that help?

[rintrah]

@epnev

There was NaN values in the correlation image, but it seems that there are also a problem with pnr (<class 'numpy.ndarray'>) . The are (in this case) 28 Inf values in pnr (np.sum(np.isinf(pnr))). When the program tries to plot it it fails because axis limits cannot be Inf. I can truncate the values of pnr, but I am not sure if this is the right approach. If truncate is a good idea, what should be the maximum value?

Thank you.

[epnev]

@rintrah I'm pretty sure you can set them to 0 (or to the max value of the non infinite values).

[rintrah]

@epnev I did what you suggested and it works. I have thought about the constant values and wonder whether it would be helpful if I use a polygonal section of my data (one that exclude constant components). Do you think it is a good idea?

Thank you.

[epnev]

@rintrah That won't be necessary and by the way the FOV has to be in an orthogonal grid. As long your patch size is large enough such that each patch contains pixels that don't have constant values there shouldn't be a problem in your analysis. You can control for NaN/Inf values in the correlation and PNR images using what we discussed above. I can also modify the code to remove that automatically. If the original problem is resolved then you can close the issue.

[rintrah]

@epnev

Thank you very much. I close the issue.