search

kabachuha / sd-webui-text2video

Auto1111 extension implementing text2video diffusion models (like ModelScope or VideoCrafter) using only Auto1111 webui dependencies
Other
1.28k stars 106 forks source link

[Feature Request]: Add ControlNet support for 2D frames #176

Open kabachuha opened 1 year ago

kabachuha commented 1 year ago

Is there an existing issue for this?

What would your feature do ?

ControlNet controlling the 2D frames slice, possibly controlling the 3D stuff later as well (But it's called InfiNet for that 😉).

So the things such as below will be possible in ModelScope as well

let-dance-64

Proposed workflow

  1. Add a ControlNet tab and do some stuff with the 2D frames using it

Additional information

So, it's basically implemented in VideoCrafter under the name of T2I adapter

kabachuha commented 1 year ago

Control a video

See

https://arxiv.org/abs/2305.13840

https://github.com/camenduru/control-a-video-colab

kabachuha commented 1 year ago

https://github.com/Weifeng-Chen/control-a-video - main repo