Very long runtime and stuck at inter_section

[ForoughHa]

Hi all,

I have recently started using MIN1PIPE with our standard Miniscope data and with only 1 session of msCam.avi file, the analysis seems to be taking much longer than expected even using a parallel pool.

It basically gets stuck at this stage:

Begin inter-section ... Done data prep Done loop #1/3 Done loop #2/3 Done loop #3/3

and then no output or prompt for more than 15 minutes with a very high disk memory (almost 2TB available).

When I manually break the program, it is at the following line:

_Operation terminated by user during intersection (line 163)

[JinghaoLu]

Can you be specific on the key numbers? What is the frame number (I guess 1000 since it is one standard msCam.avi file)? What is the spatial downsampling rate? What is your GPU size? Real RAM size (2TB you mean is the size of the disk rather than RAM I guess)? It might also be that there is some swap between memory and disk during the execution of movement correction, which can cause such freezing behavior.

[ForoughHa]

Thank you very much for your reply. I was working on my code in the past few days trying to solve the problem; however, it still seems to be taking more than other analysis toolboxes. Regarding your questions, yes it is a1000 frame file. I have tried the analysis with both no spatial downsampling (spatialr =2) and spatialr=0.5. I am using a 2.8 GHz Quad-Core Intel Core i7 processor with 16 GB 2133 MHz memory. If the problem is because of this swap between memory and disk, what would you suggest could be done to fix it?

Moreover, after waiting enough until the movement correction is done, I randomly get this error sometimes even when running the demo_min1pipe.m script with the demo data and absolutely no change to the code whatsoever:

Unable to perform assignment because the left and right sides have a different number of elements.

Error in inter_section (line 163) idbatch(i) = idtmp;

Error in frame_reg (line 107) m = inter_section(m, sttn, se, pixs, scl, sigma_x, sigma_f, sigma_d, maskc);

Error in min1pipe (line 108) [m, corr_score, raw_score, scl] = frame_reg(m, imaxy, se, Fsi_new, pixs, scl, sigma_x, sigma_f, sigma_d);

Error in MIN1PIPE_testRun (line 15) [fname, frawname, fregname] = min1pipe(Fsi, Fsi_new, spatialr, se, ismc, flag);

More interestingly, the error goes away by itself in the second run again with absolutely no manual change to the code or the data file. So 2 consecutive and exactly similar runs will give me first an error like the above and then just run smoothly (although still slow) and give out all the results.

I would appreciate your thoughts and help on this matter especially this error which causes a big waste of time after waiting so long for the code to run and without no apparent reason to me.

Best Regards, Forough

On Wed, Dec 4, 2019 at 11:47 PM JinghaoLu notifications@github.com wrote:

Can you be specific on the key numbers? What is the frame number (I guess 1000 since it is one standard msCam.avi file)? What is the spatial downsampling rate? What is your GPU size? Real RAM size (2TB you mean is the size of the disk rather than RAM I guess)? It might also be that there is some swap between memory and disk during the execution of movement correction, which can cause such freezing behavior.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/JinghaoLu/MIN1PIPE/issues/26?email_source=notifications&email_token=AIJAVY4S7SOU6UFPZ74BXW3QW6RFBA5CNFSM4JVCO7BKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEF44KCI#issuecomment-561628425, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIJAVY42N2FGJAKKBF4U7TDQW6RFBANCNFSM4JVCO7BA .

[ForoughHa]

[JinghaoLu]

It is weird. Would you please set a breakpoint on line 161 of inter_section.m and run the program. When the program stops at the breakpoint, run the while block (line 161-166 of inter_section.m). If there is an error, check idtmp and idbatch. Let's see what the problem really is.

[ForoughHa]

Hi Jinghao,

I have now worked around the run time issue somehow using the university's high-performance computing services. I have also noticed that the above error occurs when I am trying to change the detection thresholds such as "pix_select_corrthres" or "pix_select_sigthres". The reason for doing so is that I have noticed MIN1PIPE is detecting much fewer neurons than other packages and than expected. So my questions are:

1. Is there a reason for this small number of neurons I get from MIN1PIPE? (I get 72 where I usually got around 300 with other packages and the same field of view) Can I fix that somehow? because 72 in Hippocampus and with a Miniscope does not sound rational. Maybe the neurons overlap in "x" and "y" but have different "z" components and MIN1PIPE cannot extract those?

2. Can you please explain more what the matrix "spkfn" shows? It says in the ReadMe that these are spike trains but they are not just binary vectors. So do you really mean spike trains by "spkfn" or are these just the deconvolved signals? what I get is vectors of the same length for all neurons with either "0" or non-binary values in its elements. Can you explain what these values are representing and how I should interpret the "spkfn" matrix?

[JinghaoLu]

@ForoughHa From the previous posts there is still missing information for me to really understand what is going on. Did you change variable "se" or did you use the default number? As for the error you mentioned, could you please try the debug as I suggested in a previous post, or may I have a look at the video with the error? You can send it through min1pipe2018@gmail.com. Basically I do not recommend changing the values you mentioned above; the problem could be from and solved in other dimensions.

"spkfn" is something inherited from the CNMF package, you may find a comprehensive explanation in it.

[ForoughHa]

Thank you for your reply @JinghaoLu. No, I did not change the "se" value. The changes were made on _Params.mc_sigma_x, Params.pix_select_sigthres, Params.pix_select_corrthres, and Params.merge_roicorrthres

I would be more than glad to send you the video via the email address and hear your thoughts on it. Please let me know if you can figure out how I can solve the issue for the number of neurons detected.