[Test] Multi ControlNet tests!

[toyxyz]

I prepared an image sequence of ControlNet (openpose/line art) of full frame (16). And I used these to generate the video. works very well!

https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/8006000/63202b4d-192f-47f6-a6f9-bff6db2d6727

https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/8006000/975efaf7-bdd8-438e-b600-a349e4561a8e

[toyxyz]

Photorealistic model test!

https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/8006000/f2cdd4e4-a20f-4db1-9e8a-e37585d9782c

https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/8006000/5f657e0d-6248-4edf-aad9-ed16ee0715e1

[p0mad]

Is it possible to use in Comfy?

[toyxyz]

Is it possible to use in Comfy?

It's different from this, but there is a repo that ported the webui extension to ComfyUI.

https://github.com/ArtVentureX/comfyui-animatediff

[Jaxkr]

Hey @toyxyz, is there any reason you're using this fork for your animation tests? It doesn't seem like you're "prompt traveling". Is it just because of LoRA support?

[toyxyz]

Hey @toyxyz, is there any reason you're using this fork for your animation tests? It doesn't seem like you're "prompt traveling". Is it just because of LoRA support?

Of course, prompt travel is also a very useful feature! In this repo, it is possible to use multiple control-nets (tiles, open poses, line art, etc.) at the same time. And you can adjust the frame to which each controlnet image is applied and the weight of the effect it will have on other frames. As far as I know, this is the only animate-diff repo that provides this functionality. In fact, even repos that don't support Lora aren't such a big deal, as you can just merge Lora into your checkpoint and use it.

[s9roll7]

I have tried the full frame method with the source video as the input for the tile. The background of the source video was blurred, so the result is subtle, but the animation of the people generally works well.

        "controlnet_tile": {
            "enable": true,
            "controlnet_conditioning_scale": 0.5,
            "control_guidance_start": 0.0,
            "control_guidance_end": 1.0,
            "control_scale_list": [
            ]
        },

[toyxyz]

I was able to place the object where I wanted using ControlNet Depth/Seg!

https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/8006000/dc7a40e1-837f-4c2d-930b-0ee1e5df97ce

https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/8006000/8cde2f94-6396-4598-94ee-b02d2fa92ead

https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/8006000/bfe49685-fee8-4f37-baf9-9ca91207c68a

https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/8006000/a187fb1a-26c6-4b8b-aa96-cedf1cb0577f

[diStyApps]

Very cool

[s9roll7]

It looks like I should use mm_sd_v14.ckpt if I want to control all frames in the control net.

full frame openpose + mm_sd_v14.ckpt

[liam1125]

I prepared an image sequence of ControlNet (openpose/line art) of full frame (16). And I used these to generate the video. works very well!

8.mp4 6.mp4

That is very cool! can you share the config json file? Thanks so much!!

[madriss]

What would be your approach to generate something like this :

https://twitter.com/CitizenPlain/status/1710343775392829636?s=20

[s9roll7]

What would be your approach to generate something like this :

https://twitter.com/CitizenPlain/status/1710343775392829636?s=20

qr_prompt.json

[madriss]

What would be your approach to generate something like this : https://twitter.com/CitizenPlain/status/1710343775392829636?s=20

2horizontal.mp4 qr_prompt.json

That's so cool, could you please show what mask/r0, ip_adapter_image/test and controlnet_image/test_mask represent ? I am guessing the mask on the left is mask/r0. But I fail to understand what ip_adapter_image/test and controlnet_image/test_mask are ? Thanks in advance

[s9roll7]

"mask/r0" "controlnet_image/test_mask/qr_code_monster_v2" The files in these two directories are exactly the same. Note that preprocessing of qr_code_monster_v2 is disabled. Mask images are generated by this procedure https://github.com/s9roll7/animatediff-cli-prompt-travel#video-stylization-with-region

"ip_adapter_image/test" Since this directory does not exist, ip_adapter is not used this time. (Of course, you can use it if you wish.)

[zhanglaplace]

I have tried the full frame method with the source video as the input for the tile. The background of the source video was blurred, so the result is subtle, but the animation of the people generally works well.

fullframe_tile_test.webm

        "controlnet_tile": {
            "enable": true,
            "controlnet_conditioning_scale": 0.5,
            "control_guidance_start": 0.0,
            "control_guidance_end": 1.0,
            "control_scale_list": [
            ]
        },

does the controllnet images here need to be prepared with a pose or depth in advance? Can the program automatically extract the corresponding pose and depth by passing in the original image internally

[zhanglaplace]

It looks like I should use mm_sd_v14.ckpt if I want to control all frames in the control net.

full frame openpose + mm_sd_v14.ckpt

fullframe_openpose_module14.webm fullframe_openpose_module15.webm

very cool, can you share the config file of this sample?