Extremely slow saving of movies

[HoniSanders]

I am trying to convert a .memmap file (after motion correction) into an .avi movie so that I can easily view it. The .mmap file is quite large (181GB; 125710 frames). I load the .mmap file with

Yr, dims, T = cm.load_memmap(fname_new)
images = Yr.T.reshape((T,) + dims, order='F')

from the demos.

I convert into a movie object with m=cm.base.movies.movie(images,start_time=0,fr=30).

I attempt to save using m.save() and m.play(save_movie=True). Both methods take huge amounts of memory, which is expected; I have 450GB of available memory. The problem is that they are extremely slow. Even after several days, memory usage is still climbing very slowly and CPU usage is very low (0.5%) and nothing has been written. Is there any workaround for saving .mmap files into playable videos?

• Tell us a bit about your setup:

  1. Operating system (Linux/macOS/Windows): Linux

  2. Python version (3.x): 3.6.7

  3. Working environment (Python IDE/Jupyter Notebook/other): python script

  4. Which of the demo scripts you're using for your analysis (if applicable):

  5. CaImAn version*: 1.8.1

  6. CaImAn installation process (pip install ./pip install -e ./conda): conda

[epnev]

@HoniSanders There is a few things that come to mind although with a file that large it might just not be possible to play the movie.

• The memory mapped file is saved in serialized order C that favors pixels, meaning that you read first the value of the first pixels for all the frames, then the second pixel etc. This practically means that in order to read just one frame you practically need to access the entire dataset which will take a long time. However, during motion correction, each motion corrected file is also saved in F order in the same folder which prioritize frames. If you load those separately into different objects you will probably be able to view them efficiently. The path to these files can be found in mc.mmap_file (I believe).

• There is a chance that the movie object is still memory mapped. @agiovann might remember whether this is true. In this case convert it into a numpy array before turning it into a movie to make sure that is loaded in memory. You can possibly verify that by using the play method to load only a subset of frames (e.g. m[:100].play()) and see how fast that is.

• The play method by default normalizes with certain percentiles that are determined by q_max and q_min. Make sure they are set to 100 and 0 respectively to avoid their computation, although they are computed in a subset of the data so that wouldn't take too long.

• When you call m.save specify an HDF5 file as these are usually the fastest to interact with.

Let me know if any of that helps.

[HoniSanders]

A few interim comments/questions (all of this testing is being done in a jupyter notebook on windows, using a single 1000 frame file):

1. Documentation of play() is misleading. https://caiman.readthedocs.io/en/master/core_functions.html#caiman.base.movies.movie.play says that the fr parameter is the the inter-frame interval in seconds, but actually it is the frame rate in Hz.

2. Does m.save() work to save a movie as an .avi file? I keep getting an error when I try to run that:

   
   m=cm.base.movies.movie(images,start_time=0,fr=30)
   m.save(os.path.join(temp_dir, 'tmp.avi'))
   ​

---

ValueError Traceback (most recent call last)

in 1 m=cm.base.movies.movie(images,start_time=0,fr=30) ----> 2 m.save(os.path.join(temp_dir, 'tmp.avi')) C:\ProgramData\Anaconda3\envs\caiman\lib\site-packages\caiman\base\timeseries.py in save(self, file_name, to32, order, imagej, bigtiff, excitation_lambda, compress, var_name_hdf5, sess_desc, identifier, imaging_plane_description, emission_lambda, indicator, location, starting_time, experimenter, lab_name, institution, experiment_description, session_id) 190 codec = cv2.VideoWriter_fourcc(*'IYUV') 191 np.clip(self, np.percentile(self, 1), --> 192 np.percentile(self, 99), self) 193 minn, maxx = np.min(self), np.max(self) 194 data = 255 * (self - minn) / (maxx - minn) <__array_function__ internals> in clip(*args, **kwargs) C:\ProgramData\Anaconda3\envs\caiman\lib\site-packages\numpy\core\fromnumeric.py in clip(a, a_min, a_max, out, **kwargs) 2035 2036 """ -> 2037 return _wrapfunc(a, 'clip', a_min, a_max, out=out, **kwargs) 2038 2039 C:\ProgramData\Anaconda3\envs\caiman\lib\site-packages\numpy\core\fromnumeric.py in _wrapfunc(obj, method, *args, **kwds) 59 60 try: ---> 61 return bound(*args, **kwds) 62 except TypeError: 63 # A TypeError occurs if the object does have such a method in its C:\ProgramData\Anaconda3\envs\caiman\lib\site-packages\numpy\core\_methods.py in _clip(a, min, max, out, casting, **kwargs) 130 else: 131 return _clip_dep_invoke_with_casting( --> 132 um.clip, a, min, max, out=out, casting=casting, **kwargs) 133 134 def _mean(a, axis=None, dtype=None, out=None, keepdims=False): C:\ProgramData\Anaconda3\envs\caiman\lib\site-packages\numpy\core\_methods.py in _clip_dep_invoke_with_casting(ufunc, out, casting, *args, **kwargs) 83 # try to deal with broken casting rules 84 try: ---> 85 return ufunc(*args, out=out, **kwargs) 86 except _exceptions._UFuncOutputCastingError as e: 87 # Numpy 1.17.0, 2019-02-24 ValueError: output array is read-only ``` 3. When I run `m.play(fr=30, save_movie=True, movie_name=os.path.join(temp_dir, 'tmp.avi'))`, the movie shows up in a python window and plays through the whole thing. `tmp.avi` does get created, but ends up only being 6kb and won't open. 4. Are the order_F files that are output during motion correction pre- or post- motion correction? Thanks for your help so far. I will continue testing things out and report back.

[j-friedrich]

m.save() works for me, but can you please try replacing the following 3 lines (191-194) in C:\ProgramData\Anaconda3\envs\caiman\lib\site-packages\caiman\base\timeseries.py
np.clip(self, np.percentile(self, 1), np.percentile(self, 99), self) minn, maxx = np.min(self), np.max(self) data = 255 * (self - minn) / (maxx - minn) with minn, maxx = np.percentile(self, 1), np.percentile(self, 99) data = 255 * (self - minn) / (maxx - minn) np.clip(data, 0, 255, data) and report back. I'd expect the data array to be not read-only.

[HoniSanders]

@j-friedrich that worked!

it seems that the order_F files output during motion correction are pre-motion correction. the files that look like they are post-motion correction are order_C. I will try loading and saving those order_C files.

[epnev]

Both order ‘F’ and order ‘C’ files are motion corrected. The motion correction produces the order ‘F’ files which are then combined into a ‘C’ file.

On Thu, Mar 12, 2020 at 2:09 PM Honi Sanders notifications@github.com wrote:

@j-friedrich https://github.com/j-friedrich that worked!

it seems that the order_F files output during motion correction are pre-motion correction. the files that look like they are post-motion correction are order_C. I will try loading and saving those order_C files.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/flatironinstitute/CaImAn/issues/724#issuecomment-598341882, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABV3HNQ4OHD2U7MIOOGPIEDRHEQMFANCNFSM4LEPA7DQ .

[HoniSanders]

thanks.

if there is still a lot of motion in those files, is there anything I can change in the motion correction params to improve that?

[HoniSanders]

why is the clipping being done to the avi output at all? it puts the outputted video on a different brightness scale than the original video... is there a flag to opt out of that?

it's especially problematic because the clipping will scale the different movies differently, so if I end up concatenating them, there will be weird brightness shift at the video boundaries.

[epnev]

@HoniSanders It's residue from old code. We can now adjust the contrast when playing the movie so it's not needed.

@j-friedrich Can you modify the function? I believe there is no need to clip a movie when saving it unless I'm missing something. If there is a need for clipping then percentiles should be parameters q_min, q_max like in movie.play() and be computed only on a subset of frames to avoid slowdowns on large files like this one.

[j-friedrich]

There's no need for clipping, so I can just remove the line that does the clipping. However, there will still be some scaling to the range [0,255] cause cv2 expects uint8.

[agiovann]

Clipping is useful because sometimes your extreme values will affect the resolution of the signals (by increasing the dynamic range). It should be an option.

On Fri, Mar 13, 2020 at 12:58 PM Johannes Friedrich < notifications@github.com> wrote:

There's no need for clipping, so I can just remove the line that does the clipping. However, there will still be some scaling to the range [0,255] cause cv2 expects uint8.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/flatironinstitute/CaImAn/issues/724#issuecomment-598820581, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4DYTXLXZAGD6VYGOETIULRHJQ47ANCNFSM4LEPA7DQ .

-- Andrea Giovannucci, PhD Assistant professor Biomedical Engineering Department

UNC Chapel-Hill Chapel Hill, NC 27514 609.865.8251 main

nel.bme.unc.edu/ http://nel.bme.unc.edu/

[epnev]

That’s a good point Andrea. I think if you need to save a movie for further processing you should probably avoid avi files altogether. The 8-bit format is too restrictive and you’ll end up losing resolution one way or another.

On Fri, Mar 13, 2020 at 1:56 PM Andrea Giovannucci notifications@github.com wrote:

Clipping is useful because sometimes your extreme values will affect the resolution of the signals (by increasing the dynamic range). It should be an option.

On Fri, Mar 13, 2020 at 12:58 PM Johannes Friedrich < notifications@github.com> wrote:

There's no need for clipping, so I can just remove the line that does the clipping. However, there will still be some scaling to the range [0,255] cause cv2 expects uint8.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/flatironinstitute/CaImAn/issues/724#issuecomment-598820581 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AB4DYTXLXZAGD6VYGOETIULRHJQ47ANCNFSM4LEPA7DQ

.

-- Andrea Giovannucci, PhD Assistant professor Biomedical Engineering Department

UNC Chapel-Hill Chapel Hill, NC 27514 609.865.8251 main

nel.bme.unc.edu/ http://nel.bme.unc.edu/

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/flatironinstitute/CaImAn/issues/724#issuecomment-598844397, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABV3HNSAMIXMQDGZSCZR4ZLRHJXTJANCNFSM4LEPA7DQ .

[HoniSanders]

I'm not using the avi files for processing, I'm using them for visualization. One reason is to be able to compare the raw video and the motion corrected video side by side. The original videos were in avi, so saving the motion corrected videos to avi shouldn't lose resolution that was in the original videos (if I'm not mistaken). Somehow it should be possible to output the avi file at the same brightness scale as the original videos, right?

[agiovann]

Yes, this should be possible via a flag in this case, since the original AVI will also be 8 bits.

On Fri, Mar 13, 2020 at 4:32 PM Honi Sanders notifications@github.com wrote:

I'm not using the avi files for processing, I'm using them for visualization. One reason is to be able to compare the raw video and the motion corrected video side by side. The original videos were in avi, so saving the motion corrected videos to avi shouldn't lose resolution that was in the original videos (if I'm not mistaken). Somehow it should be possible to output the avi file at the same brightness scale as the original videos, right?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/flatironinstitute/CaImAn/issues/724#issuecomment-598899747, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4DYTTMSYXQFWLRSFV2JJLRHKJ47ANCNFSM4LEPA7DQ .

-- Andrea Giovannucci, PhD Assistant professor Biomedical Engineering Department

UNC Chapel-Hill Chapel Hill, NC 27514 609.865.8251 main

nel.bme.unc.edu/ http://nel.bme.unc.edu/

[j-friedrich]

@HoniSanders playing the original and the motion corrected movie side by side is demonstrated in demo_motion_correction.ipynb.

In the dev branch, I added q_min and q_max values to the movie's save method, so you can set these to 0 and 100 if you don't want to crop.

I only expect very few other users to use it, however, if you don't want to automatically scale to [0,255], but save your movie as is (albeit auto-converted to uint8) set (one of) them to None. However, be aware of the risks of simply converting to uint8 and ensure your movie actually lives in the [0,255] (or [0,256) for floats) range.

[HoniSanders]

thanks all

[HoniSanders]

Can you merge that commit into the main branch? Without it, m.save() still errors out on the clip line.

[epnev]

@HoniSanders No, sorry but this is not a bug and does not warrant a release on its own. You can modify your source files to incorporate this as shown in https://github.com/flatironinstitute/CaImAn/commit/acaab455eeb880011580ef741af5b5ce33562db4

[HoniSanders]

That's fine as long as it gets included in future releases

[epnev]

Yes, everything that is in the dev branch will be included in the next release (and we actually might end up making a new one soon after all)

On Wed, Mar 25, 2020 at 10:40 AM Honi Sanders notifications@github.com wrote:

That's fine as long as it gets included in future releases

— You are receiving this because you modified the open/close state.

Reply to this email directly, view it on GitHub https://github.com/flatironinstitute/CaImAn/issues/724#issuecomment-603877385, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABV3HNV7IH5D4QWMXDSELFLRJIJVHANCNFSM4LEPA7DQ .

[HoniSanders]

thanks!