Running out of memory

[willcitizen]

Hi there

any thoughts on to check / improve how your script is handling memory during the encoding.

I have a 400 image sequence - and 32Gb of ram, crashes the script after about minutes of processing.

not serious as I have found through trial and error that can run up to 100 or so without running out of mem,

I will look at making this improvement myself, with my soso python skills - thanks for your work btw, really useful little tool, this is my first inifinite zoom run and I got right to the part where I was going, hmm - how much work is it going to be to stagger these by hand :)

[mwydmuch]

Hi @willcitizen, wow, 400 images give quite a long sequence, I would like to see the result later! Let's then solve your issue.

I run some tests and calculations, and a video from 512 images of size 1024x1024 should require more or less 12 GB of RAM (on Mac). The memory is mostly allocated at the beginning when blended images are created and depend on their number and resolution. Then memory usage should stay stable for the rest of the running time, with small addition when encoding the video, so I'm surprised that you run our of memory in this last stage, I would suspect that this problem occurs at the beginning.

Could you confirm that this is a case, or I misunderstood your description, and the error appears at the beginning? Could you also provide some details on the images you are trying to use? Like resolution? And OS you are using? Might be OS related thing, with ffmpeg having larger memory consumption (or even leak) on one OS than another.

[mwydmuch]

Ok, in 0.3.2, I've just introduced the most obvious optimization that should reduce the max memory requirement by half. Meaning that 512 images of size 1024x1024 can be turned into video using ~6 GB. Maybe this solves your issue.

Further optimization is very much possible but requires more significant code changes.

[willcitizen]

Thanks so much, just pulled the new release - I'll let you know how it goes.

The project is my first stable diffusion infinite zoom run, its a generative, mutating prompt, surreal cosmic road trip :D Images generated were upscaled to 2048x2048 - i plan to crop down to 1920 and have some room on the edges to spin the thing :)

Will definitely share a link here as soon as I have it posted somewhere.

[willcitizen]

I ran into some issues with 0.32 - it works fine for shorter batches - I don't think you broke anything, but the very long run sort of stop talking to gradio at some point. It did still generate all 8999 frames and convert them into the video, but the video was messed up and did not seem to have arranged the frames or interpolated (that is the word I think) them correctly. I tried a few other things, like converting my images to jpg to reduce size in memory, but 0.32 didnt seem to like that and then didnt seem to be happy with my 0.31 batch sizes anymore so I went back to 0.31 to finish up the renders for my project. For now.

incidentally I was thinking the batches would be easy to run from the cmdline, but I tried to run both 0.31 and 0.32 from the cmdline with my files and get an error similar to this - it works with your example files.

File "E:\AITools\ZoomVideoComposer\helpers.py", line 288, in blend_images image.paste(inner_image, margin, margin) File "E:\AITools\ZoomVideoComposer\helpers.py", line 73, in paste self.image[y : y + image.height, x : x + image.width] = image.image ValueError: could not broadcast input array from shape (2458,2458,3) into shape (1741,1741,3)

[mwydmuch]

Hi @willcitizen, I'm not an expert in gradio, so at the moment I can't really address the problem of it stopping responding, also it could be caused by a lot of stuff. When it comes to other issues:

(...) the video was messed up and did not seem to have arranged the frames or interpolated (that is the word I think) them correctly.

I would need to know what were your filenames and setting you used to be able to address it, are you sure you used the correct parameters and named the files in lexicographical order?

I tried a few other things, like converting my images to jpg to reduce size in memory

The image input format doesn't matter, the images are decompressed when stored in memory for manipulation.

incidentally I was thinking the batches would be easy to run from the cmdline, but I tried to run both 0.31 and 0.32 from the cmdline with my files and get an error similar to this

Again hard to address it without knowing more about the specific command and images. Are all your images of the same size?

Generally, it would be great if you could prepare a small sample and command that produce this error/problem and share it with me.

[willcitizen]

Thanks for your help so far. I completely appreciate this will need a proper debugging session - I just didnt want to leave you with no feedback after being kind enough to release an updater so quickly.

I'm an old school tecchie, we don't comlpain - we troubleshoot :D

I dont code often and python is a new space for me, but I'd love to help figure this out and see how I can improve it for my use case. I'll spend some time running a few more tests with 0.32 in the next few days and try and create some debug logs as well. I'll post here again when I have something substantial.

thanks again

update I'm also old school enough to know that most times - it is the users fault :-D I somehow did not pick up that my source image (which is actually the last image had been upscaled more than once by my workflow and was 3072x3072 not 2048x2048.

Only picked that up when I got to rendering the 4th batch on 0.31 - I couldnt understand why it failed when the other 3 worked perfectly. I think that also explains my errors at the cmdline, and why the full run of 400 images that actually succeeded was all skew.

On that basis I'm going to rerun the full batch of 400 on 0.32 and let you know when it finishes.

update 2 After sorting out my user error I was able to run 0.32 from the cmdline and process all 400 images in reverse order for a video length of 400 seconds - a total of 12000 frames at 2048x2048. and your algo is on point for handling the frame blends - the final video looks great.

I'm just busy with post now, adding audio and some flash, will definitely post a link here and credit you when I share it :)

[mwydmuch]

Thanks for confirming that it works. I'm closeing this issue then. Good luck with your project!