search

bahanonu / ciatah

CIAtah (pronounced cheetah): a software package for calcium imaging analysis of one- and two-photon imaging datasets. Documentation: https://git.io/ciatah_docs. Formerly known as calciumImagingAnalysis (ciapkg).
https://git.io/ciatah_docs
MIT License
83 stars 21 forks source link

Memory problem #87

Open harshk95 opened 3 years ago

harshk95 commented 3 years ago

When I try to preprocess videos bigger than my JAVA heap memory (34Gb, max allowed) I get the error below and I was wondering how can I analyse videos with size bigger than that. Thank you very much in advance for your help

pre-allocating single ones matrix: 800 1280 38801 @@@@@@@ Error using ones Requested 800x1280x38801 (148.0GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take a long time and cause MATLAB to become unresponsive. See array size limit or preference panel for more information. Error in loadMovieList (line 472) outputMovie = ones(preallocSize,imgClass); Error in ciatah/modelPreprocessMovieFunction (line 729) thisMovie = loadMovieList(movieList,'convertToDouble',0,'frameList',thisFrameList,'inputDatasetName',options.datasetName,'treatMoviesAsContinuous',options.turboreg.treatMoviesAsContinuousSwitch,'loadSpecificImgClass','single'); Error in ciatah/modelPreprocessMovie (line 37) obj.modelPreprocessMovieFunction('folderListPath',folderListInfo,'fileFilterRegexp',options.fileFilterRegexp,'datasetName',options.datasetName,'frameList',[]); Error in ciatah/runPipeline (line 202) obj.(thisFxn{1}); Error in ciatah/display (line 6) obj.runPipeline; @@@@@@@ pre-allocating movies to display... +++++++ no movie!

MIJ 1.3.9: Matlab to ImageJ Interface

More Info: http://bigwww.epfl.ch/sage/soft/mij/ Help: MIJ.help JVM> 1.3.9 JVM> Version: 1.8.0_181 JVM> Total amount of memory: 387584 Kb JVM> Amount of free memory: 244514 Kb ImageJ> Version:1.53k ImageJ> Memory:141MB of 29964MB (<1%) ImageJ> Directory plugins: Not specified ImageJ> Directory macros: Not specified ImageJ> Directory luts: C:\Users\realtime\Documents\MATLAB\luts\ ImageJ> Directory image: Not specified ImageJ> Directory imagej: C:\Users\realtime\Documents\MATLAB\ ImageJ> Directory startup: C:\Users\realtime\Documents\MATLAB\ ImageJ> Directory home: C:\Users\realtime\

Status> ImageJ is running.

Warning: The DrawMode property will be removed in a future release. Use the SortMethod property instead.

In ciapkg.overloaded.msgbox (line 280) In ciatah/modelPreprocessMovieFunction>playOutputMovies (line 2432) In ciatah/modelPreprocessMovieFunction (line 1090) In ciatah/modelPreprocessMovie (line 37) In ciatah/runPipeline (line 202) In ciatah/display (line 6) +++++++ no movie! ImageJ instance ended cleanly

bahanonu commented 3 years ago

Been working on improvements to get around RAM-limitation based memory issues when using modelPreprocessMovie (our newer cell extraction methods do not have this issue). Other work arounds can be done by directly using the ciatah command line api (e.g. https://bahanonu.github.io/ciatah/api_example_pipeline/), but I can get back once modelPreprocessMovie is updated.

A couple quick questions:

There is the possibility that you can downsample the raw movies (https://bahanonu.github.io/ciatah/pipeline_detailed_downsample_raw/) to reduce file/matrix size, as often the higher pixels/micron resolution is not needed for cell detection and activity trace extraction.

harshk95 commented 3 years ago

Hi! Thanks for the prompt reply and looking forward to the update.

Maximum possible array: 139597 MB (1.464e+11 bytes) Memory available for all arrays: 139597 MB (1.464e+11 bytes) Memory used by MATLAB: 3567 MB (3.740e+09 bytes) Physical Memory (RAM): 130749 MB (1.371e+11 bytes)

Limited by System Memory (physical + swap file) available.

We tried downsampling in the inscopix software and that decreased the size of the videos and made it work. Nonetheless, once we manage to see the snippet of the MC movie, the software opens a dialog box to continue to the next movie (but we are currently working on one only) and if we close that dialog box the MC movie ends up not being saved in the processing folder. How can we sort this out?

Thanks again for your help!!

bahanonu commented 3 years ago

Great to hear the downsampling works! For Inscopix miniscope movies often 2x or 4x downsampling in each dimension usually works fine.

Did a movie play in ImageJ or MATLAB? Can you send me the _preprocess.log file in the processing_info sub-folder that is in the same folder as the raw movie being processed? Should start with a date-time string for when that processing run started. That'll help me debug why you aren't seeing the output movie.

harshk95 commented 3 years ago

20210910_113823_preprocess.log

Here the file! A snippet played in ImageJ, yes. Thanks a lot for your help again

bahanonu commented 3 years ago

Looks like a turboreg error occurred during 2nd round of motion correction. Do you see 0000_00_00_p000_m1_NULL000_medFlt_detrend_1.h5 movie in the M1_4 folder?

Also, can you run with the below RegisType affine settings and let me know if it finishes fine? In general, unless you suspect significant warping in your Inscopix recordings, I would recommend against running with RegisType set to any projective options, as that can lead to sub-optimal results.

image

Else, if you can run with a single turboreg iteration and send me a copy of the movie (only a couple frames, e.g. see below), that can help me diagnose what happened with projective. e.g. the _snippet.h5 movie as below:

thisPath='full_path_to_output_movie';
[thisDir,filename1,ext1]=fileparts(thisPath);
saveMatrixToFile(loadMovieList(thisPath,'frameList',1:500),[thisDir filesep filename1 '_snippet' ext1]);
harshk95 commented 3 years ago

We managed in the end to make it work: 5 iterations with a 4x downsampled video (size 5Gb approx.) managed to be preprocessed on a 512GB RAM computer and the results of motion correction are really great.

After preprocessing we need to use extract as an extraction method. For our application, people in our lab have worked in conjuction with the extract team to finetune the config and get results. Nonetheless, this finetuning was done on non-downsampled videos. So we went back and tried to run ciatah preprocessing on non downsampled files (approx. 80Gb) on a 512Gb RAM computer but this led to the out memory error again.

Is there any way we can improve RAM efficiency?

Very sorry for the trouble and thanks a lot for your help.

bahanonu commented 3 years ago

Thanks for the update on this and glad to hear the downsampled movies worked well for you-all.

re: non-downsampled movies. It is possible to improve the RAM efficiency by running ciatah pre-processing via the command-line (e.g. https://bahanonu.github.io/ciatah/api_example_pipeline/) so that certain steps (e.g. motion correction) are only done on chunks of the movie at a time (this is what we do for some of our cell extraction algorithms or other steps).

Could you let me know what steps you are running for ciatah pre-processing? I can get back with a script that can reduce RAM usage as still working to update CIAtah GUI for other users to handle this issue.

harshk95 commented 3 years ago

Hey, thanks a lot, that would be very helpful!

The preprocess steps we currently wish to use are:

Thanks again!