search

kijai / ComfyUI-ADMotionDirector

ComfyUI nodes for training AnimateDiff motion loras
Apache License 2.0
157 stars 12 forks source link

[Feature request] Cpu offloading of training data #13

Open gitmylo opened 3 months ago

gitmylo commented 3 months ago

Hey, I noticed that when training with multiple batches I got crashes due to running out of vram. Here is an example patch which will offload batches to cpu and copy them to gpu when required Cpu_offloading_for_batches.patch I did not make it offload if you only have one batch, because that would just waste your time. (Since you're going to have at least one batch on gpu at a time anyways.)

Having this as an option (in InitializeTraining for example) would be nice. Because I run out of memory on larger amount of batches. Offloading the batches to cpu fixes the out of memory crashes.

kijai commented 3 months ago

How many batches are we talking about exactly?

gitmylo commented 3 months ago

In my case, with 12gb vram, At 22 batches of 16 frames of 512x512 resolution I was running out of video memory, So I had to offload to cpu. It might also run out of memory at slightly less batches, I don't know the exact number it would run out at, but 22 is definitely over it for me.

I loaded the batches using vhs load video (path), automated with flownodes to load every file on the path, then process them, and flatten them (which ends with a bunch of image batches, same as manually loading and merging, just automated so I could easily load multiple and pre-process them.)