search

ali-vilab / VGen

Official repo for VGen: a holistic video generation ecosystem for video generation building on diffusion models
https://i2vgen-xl.github.io
2.8k stars 246 forks source link

Best config for generating more frames ? #60

Open Sanath91009 opened 6 months ago

Sanath91009 commented 6 months ago

when I set increase max_frames from 16 to 32 then I see loss of temporal dynamics in the video after 16 frames. Config I was using is 

TASK_TYPE: inference_i2vgen_entrance
use_fp16: True
guide_scale: 9.0
use_fp16: True
chunk_size: 2
decoder_bs: 2
max_frames: 32
target_fps: 16      # FPS Conditions, not the encoding fps
scale: 8
seed: 8888
round: 4
batch_size: 1
use_zero_infer: True 
# For important input
vldm_cfg: configs/i2vgen_xl_train.yaml
test_list_path: data/test_list_for_i2vgen.txt
test_model: models/i2vgen_xl_00854500.pth

I tried increasing guide_scale to 12.0, still no change. What is the best config to generate more frames maintaing temporal dynamics ?

Steven-SWZhang commented 6 months ago

We trained this model to output both 16 frames and 32 frames (simply by modifying the max_frames parameter). However, we are more inclined towards the generation of 16-frame videos. We have not validated the results for 32-frame videos. Thank you.

Sanath91009 commented 6 months ago

Is there feature of auto regressive video frame generation ? I mean passing 16-frame video as condition to generate next 16-frames

Steven-SWZhang commented 6 months ago

The current model does not support the recursive generation, where you might use the last frame of the generated video as a condition for video expansion. Thank you for your suggestion, and we will consider this setup.