EXTRACT on striatum data

[bahanonu]

@inanhkn re: test movie 2013_06_23_p51_m348_oft1

I'm getting an error when running EXTRACT (I'll go with Mark's naming for now) on a test movie.

• Running the function with only the default options enabled, e.g. extractor('2013_06_23_p51_m348_oft1_fullSession_DFOF_crop.h5'); for the movie in the Dropbox link below.
• Error is Index exceeds matrix dimensions. on line 244 of compute_spatial_weights().
• Seems to stem from line 242: regionprops(mask_candid,'centroid') giving an empty output.

I've included the error workspace at that point along with the test movie I was using in the Dropbox link below. Let me know when you grab the movie so I can remove it from my Dropbox. The HDF5 dataset name is `/1'.

https://www.dropbox.com/sh/i78iarncmrl40xf/AAC5O6oTDr7zQz3AIpiC7_dva?dl=0

Let me know if you need any more information from me, I'll run on a different movie to see if I get the same error.

[inanhkn]

I don't have access to my work computer now, I'll get to this first thing tomorrow. I haven't seen this error before, but it should be straightforward to fix.

[inanhkn]

I downloaded the movie and the output, you can remove them. I'll make a pull request with the fix soon.

[inanhkn]

I committed the fix.

I looked at your movie, and I have some immediate suggestions on some of the parameters. First of all your movie is downsampled by a very big factor in space; in this case I would use opts.spat_linfilt_halfwidth = 2. I would also suggest using a small number of SVD components (somewhere between 300 and 500 maybe). Just as a safeguard to very small cells that could be caused by unnecessary SVD components (which I think what caused the cell centroid error), you should set opts.ss_cell_size_threshold to at least something like 5. One critical thing I noticed is that the movie has a large amount of background fluctuations. Maybe you could do a bit more aggressive normalization. This is important chiefly because this causes some cells to be masked by the background activity and also causes crosstalk between cells that are extremely hard for an algorithm to eliminate.

[inanhkn]

Also, I don't think you should use median filtering, there doesn't seem to be camera artifacts. Last but not the least, I would use very little pixel trimming in your case since your background fluctuations make up a large bright area in the max-projection.

[bahanonu]

Sounds good, I'll try again/report back on the same movie and on more recent data that has better normalization (that was a very old test movie that I had used as an edge case for cellmax).

Normally do 4x spatial downsampling, so I'll change opts.spat_linfilt_halfwidth and opts.ss_cell_size_threshold to account for that.

[bahanonu]

It's running fine on my end now. See below for a preview, I'll upload a manually sorted cell map tonight from the two example movies with comparison's to PCAICA.

Is the inhomogeneous distribution of filters across the FOV normal (e.g. are you really aggressive at not including neuropil or 'bad' cell filters)?

[bahanonu]

@inanhkn Below is some initial analysis on manually sorted cells from 2014_10_16_p215_m790_reversalAcq04 (same as the video I sent earlier today in the Dropbox) using EXTRACT, PCA-ICA, and CELLMax. Let me know if you want information/feedback on specific things.

• Overall, PCA-ICA and EXTRACT appear to identify more cells than CELLMax.
• The cells only identified in PCA-ICA or EXTRACT appear to have a lower SNR and calcium transient rate than those identified in all three algorithms. Not sure if this is due to CELLMax's subsampling.
• For EXTRACT, I noticed a fair number of bad sources that basically were off-centered from actual good sources (see last figures below for examples). Is this something that can be reduced by changing any of the parameters?
• EXTRACT appears to produce a very inhomogeneous distribution of raw cell filters compared to CELLMax but is quite similar to PCA-ICA, which I assume is due to you using SVD as a starting point?
• Are the traces cleaned up at all after least squares like is done in CELLMax? They seemed to contain some of the same fluctuations at baseline as the movie and don't appear as flat at baseline as CELLMax's traces.

@kimtonyhyun Do the above observations match what you've seen when comparing the three algorithms?

Overlap of EXTRACT, PCA-ICA, and CELLMax

Sorted outputs

Raw outputs | Brighter = good cells, dim = bad cells.

Comparison of sources from EXTRACT, PCA-ICA, and CELLMax based on basic signal statistics

Bad EXTRACT sources next to good sources

[inanhkn]

Judging from the discrepancy between number of good cells and initially found cells for EXTRACT, I assume there are too many SVD components used for extraction. one way to check how many components to use is to look at the singular values of the SVD output. In many cases you can tell the what the maximum number of SVD cmponents to use. One thing I am aiming for with you guys trying it out is to collect enough statistics to automate that :) My suspicion for this case is that singular values are very negligible beyond 300 or so.

What do you mean by bad sources very near good sources? Are sources by definition cells extracted by an algorithm? One potential answer as to why there are many bad sources next to good sources in this case is that using much more SVD components than necessary leads to spurious sources, and they happen to show up in an active region. I would recommend ( in fact ask :) ) to use the existing SVD output and run extraction with less and less SVD components until the point you start losing good cells (assuming you can do this easily by checking the overlap between different runs).

[inanhkn]

@bahanonu I have a general request. After you manually do cell checking, can you share with me the classification results and the output .rec file?

[bahanonu]

@inanhkn regarding sharing results/.rec file: sounds good, will do.

Sorry for the confusion, when I say sources I mean any filter/trace pair, regardless of whether it's a cell (e.g. could be neuropil or some other non-cell fluctuation). It seemed like several sources were just the edges of cells (e.g. like what PCA-ICA does sometime for really strong/large cells).

I'll try the test with using less SVD components and report back:

• I can just rerun on the same movie with opts.ss_num_comp set to less and less components?
• Does the algorithm automatically look for a SVD file in the movie's folder or do I need to supply opts.existing_SVD_file?

[inanhkn]

Thanks a lot, user data is what allows developers to make things better :)

I see, then as I mentioned above less SVD components might make that situation better.

Yes, you can simply set opts.ss_num_comp to smaller and smaller values. You need to provide an SVD file, but it's actually good idea to simply look for the most recent SVD file, I'll make it happen soon.

[inanhkn]

@bahanonu could you share the output and classification result with the movie 2014_10_16_p215_m790_reversalAcq04_turboreg_crop_dfof_downsample.h5 ?

[bahanonu]

Yep, added to shared dropbox at \schnitzer_lab_code_output_debug\2014_10_16_p215_m790_reversalAcq04

Also added PCAICA and CELLMax outputs.

[inanhkn]

Thanks :)