ValueError: could not broadcast input array from shape (512,768,4) into shape (512,768,3)

[Optimusik]

When i try to do txt2vid i got this error after creating first frame

File "D:\sd\stable-diffusion-webui\extensions\SD-CN-Animation\scripts\base_ui.py", line 116, in process yield from txt2vid.start_process(*args) File "D:\sd\stable-diffusion-webui/extensions/sd-cn-animation/scripts\core\txt2vid.py", line 97, in start_process clip_frames[-1] = cv2.resize(prev_frame, size) ValueError: could not broadcast input array from shape (512,768,4) into shape (512,768,3)

when i try to make vid2vid i get this

File "D:\sd\stable-diffusion-webui\extensions\SD-CN-Animation\scripts\base_ui.py", line 114, in process yield from vid2vid.start_process(args) File "D:\sd\stable-diffusion-webui/extensions/sd-cn-animation/scripts\core\vid2vid.py", line 195, in start_process warped_styled_frame = curr_frame.astype(float) alpha_mask flow_mask + warped_styled_frame.astype(float) (1 - alpha_mask * flow_mask) ValueError: operands could not be broadcast together with shapes (960,512,4) (960,512,3)

controlnet installed, all updated, maybe python version should be specific? v0.7 didnt work aswell so i just wait for update in hope all be fine, but error in v0.7 was different

[volotat]

Is there an alpha channel in the animation you are trying to process? If so it is not supported for now.

[halb-ei]

I am having the same problem. I tried it with multiple mp4 files but I didn't find anything useful online on how to see if the files have an alpha channel, but yet again the same error code appears with text2video and there is no option with any alpha channels.

[volotat]

@halb-ei Yeah, make sense. No idea where are the 4th channel coming from. I tried to make a rough fix for this issue, please update the extension and tell me if it works or the error if it still occurs.

[halb-ei]

@volotat that fixed it. the videos are now being generated. But the output videos are a lot darker than the input videos and I don't think that was supposed to happen.

[volotat]

@halb-ei Could you show an example? May there be any settings in web-ui active on your side that might change the color space of the images? Because beside the alpha channel and different color space (like CMYK) I could hardly imagine what also may cause 4 channels at the processing step.

[halb-ei]

@volotat here is an example. I kept the clip short (less than a second) because my gpu isn't very fast. The output video is significally darker. Maybe I used the wrong proessing and fix frame strength but I only changed the processing strength very slighty to what it is set up to when you launch it. I also tried changin the correct color for img2img setting an the do ecactly the same steps as specified setting but it didn't make anything better.

https://github.com/volotat/SD-CN-Animation/assets/120658457/6d615654-6b52-46bd-a327-40e4b5b144d2

https://github.com/volotat/SD-CN-Animation/assets/120658457/a1d00eef-2aa8-41a0-af39-dfca1ffa5a9f

[volotat]

@halb-ei 'processing strength' and 'fix frame strength' should not have any negative effect on the colors. I also suspect that it might be related to the model and related VAE nuances. Could you please try to use default SD-v1-5 with default VAE and see if it helps? Make sure that the option 'Apply color correction to img2img results to match original colors' in 'Stable Diffusion' settings disabled.

[halb-ei]

@volotat color correction and VAE dissabled seems to do the trick. both outputs have VAE dissabled, the first one has color correction turned on and the second one has color correction turned off. And I noticed that sometimes the there won't be an output video if the width and height parameters are "wrong". It seems to me as random which aspect ratio and pixel height and width are causing this but when i got one it always did not give me an output , same with the ones that gave me an output.

https://github.com/volotat/SD-CN-Animation/assets/120658457/ab0c96a0-7b04-40d4-8670-8635fc25aeb1

https://github.com/volotat/SD-CN-Animation/assets/120658457/ab2399ff-18bf-474f-8280-9b939531375c

https://github.com/volotat/SD-CN-Animation/assets/120658457/c0d457b2-c951-47e9-a081-3860d0e08252

[volotat]

Great. So active color correction was the reason for darkening. I guess this issue is done. By the way, check out updated CN parameters in readme.md. Turns out the tile model works much much better than any other controls for vid2vid animations.

[emersound]

I figured out that if you set video dimensions that are not he dimensions that auto snap on the height and width sliders it will not like it, and it will throw that message.