[Test] new controlnet (animatediff_controlnet))

[s9roll7]

https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/118420657/aa3622ae-693b-4d83-96b2-5dd27b840057

original / animatediff_controlnet + controlnet_openpose(0.5) / animatediff_controlnet + controlnet_openpose(0.5) 1024x1280

animatediff stylize create-config PATH_TO_MOVIE_FILE -gh animatediff stylize create-region PATH_TO_STYLIZE_DIR animatediff stylize generate PATH_TO_STYLIZE_DIR

[s9roll7]

https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/118420657/b8602a2c-ce36-40f0-80f8-3b808e4b8321

original / animatediff_controlnet + controlnet_openpose(0.5) + controlnet_normalbae(0.5) + ip_adapter/ animatediff_controlnet + controlnet_openpose(0.5) + controlnet_normalbae(0.5) + ip_adapter 768x1152

animatediff stylize create-config PATH_TO_MOVIE_FILE -gh animatediff stylize generate PATH_TO_STYLIZE_DIR

[KaustubhPatange]

Hey @s9roll7, tried your exact commands but it seems like the background is not generated properly and looks green a bit. Here is my prompt.json. What could be wrong?

Also it seems like animatediff_controlnet did not ran and it requires some conditioning images. What preprocess images have you supplied for openpose?

Original	Generated

[s9roll7]

Hey @s9roll7, tried your exact commands but it seems like the background is not generated properly and looks green a bit. Here is my prompt.json. What could be wrong?

Also it seems like animatediff_controlnet did not ran and it requires some conditioning images. What preprocess images have you supplied for openpose?

Original Generated input.mp4 output.mp4

The json file looks fine. I'm guessing 00_controlnet_image\animatediff_controlnet is empty? If there's nothing in the directory, nothing will happen. You need to copy all the images in controlnet_openpose to animatediff_controlnet.

[s9roll7]

https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/118420657/e65e0dcc-1ce3-4389-9361-51690df1176d

original / animatediff_controlnet + controlnet_openpose 896x768 generation time 196sec (4090) vram 10.3 GB

animatediff stylize create-config PATH_TO_MOVIE_FILE -gh animatediff stylize generate PATH_TO_STYLIZE_DIR

[dancemanUK]

use the animatediff_controlnet , 4090,speed too slow, share DDR4 RAM 6gb.not in 4090 24GB VRAM

[s9roll7]

use the animatediff_controlnet , 4090,speed too slow, share DDR4 RAM 6gb.not in 4090 24GB VRAM

    "controlnet_map": {
        "input_image_dir": "..\\stylize\\2023-12-06T13-02-02-sample-mistoonanime_v20\\00_controlnet_image",
        "max_samples_on_vram": 200,           # <-----------
        "max_models_on_vram": 3,                # <---------
        "save_detectmap": true,

[dancemanUK]

use the animatediff_controlnet , 4090,speed too slow, share DDR4 RAM 6gb.not in 4090 24GB VRAM

    "controlnet_map": {
        "input_image_dir": "..\\stylize\\2023-12-06T13-02-02-sample-mistoonanime_v20\\00_controlnet_image",
        "max_samples_on_vram": 200,           # <-----------
        "max_models_on_vram": 3,                # <---------
        "save_detectmap": true,

thank you！

[AivoGenX]

Hello, I used the img2img function in Stable Diffusion 1.5, which only added prompt words. The effect is quite different from that of Animatediff cli prompt travel. How can I maintain better consistency with SD? Thank you

[aycaecemgul]

hi, can you share your json for this style? @s9roll7