search

Doubiiu / DynamiCrafter

[ECCV 2024, Oral] DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors
https://doubiiu.github.io/projects/DynamiCrafter/
Apache License 2.0
2.64k stars 211 forks source link

How to avoid large batch been killed #132

Open lxrswdd opened 2 months ago

lxrswdd commented 2 months ago

I have 5176 images with 5176 prompts

(dynamicrafter) xiangrui@ward-lambda01:~/video_generators/DynamiCrafter$ sh scripts/run_1024.sh 1024
@DynamiCrafter cond-Inference: 2024-09-18-15-48-40
Global seed set to 666
AE working on z of shape (1, 4, 32, 32) = 4096 dimensions.
>>> model checkpoint loaded.
Inference with 16 frames
Killed
(dynamicrafter) xiangrui@ward-lambda01:~/video_generators/DynamiCrafter$

The model worked well with 8 samples but once I replaced the default prompts with my 5000+ prompts. The task is killed.

Doubiiu commented 2 months ago

Hi. It may be due to the memory leakage issue, as we load all input data into memory at the same time. I think a solution is to use a data loader to fetch the input data by batch.