Kosinkadink / ComfyUI-VideoHelperSuite

Nodes related to video workflows
GNU General Public License v3.0
513 stars 90 forks source link

[Feature Request] Alpha channel support #135

Open SeanBannister opened 7 months ago

SeanBannister commented 7 months ago

I was wondering if it would be possible to support alpha channels, I just ran some tests with the video combine node and was unable to get an alpha channel.

ProRes 444 supports alpha channel but the node only exports 422 HQ. But I also tried webp which should supports it but it exported a white background instead of transparency.

Alpha channels are useful when you want to mask a object that's generated and then generate a background separately.

AustinMroz commented 7 months ago

After #126, the Video Combine node supports alpha channels, but there's a number of inconveniences that need to be ironed out to make it accessible.

While some video containers/codecs support reading transparency data, the existing video formats (with the exception of 16bit-png) don't currently specify that the output pix format should include a transparency channel. I'm reluctant to always output alpha channel information and am still considering how best to conditionally set output pix_fmts with transparency. In the meantime, here's a couple of example formats I can confirm work to produce transparent output.

For ProRes, I'm seeing some unusual interactions I'll need to investigate with the profile setting overriding an explicit pix_fmt, but the following seems sufficient. (Special thanks to the prior work of @Chentmin)

{
    "main_pass":
    [
        "-n", "-c:v", "prores_ks",
        "-profile:v","4",
    ],
    "audio_pass": ["-c:a", "pcm_s16le"],
    "extension": "mov"
}

For webm, changing the pix_fmt is sufficient, but comes with an exceedingly bizarre issue (which I've not yet verified occurs on systems other than my own): If that output video is put back into ffmpeg, ffmpeg fails to recognize that the video includes transparency information. The video will display in browser with transparency if "VHS Advanced Previews" is disabled, but viewing the video in external programs that share ffmpeg libraries (mpv) or the re-encoding of the advanced previews option destroys the transparency information.

{
    "main_pass":
    [
        "-n",
        "-pix_fmt", "yuva420p",
        "-crf", ["crf","INT", {"default": 20, "min": 0, "max": 100, "step": 1}],
        "-b:v", "0"
    ],
    "audio_pass": ["-c:a", "libvorbis"],
    "save_metadata": ["save_metadata", "BOOLEAN", {"default": true}],
    "extension": "webm"
}

ComfyUI has a "Join Image with Alpha" node I only recently learned of. This is sufficient to get transparency information into an input format that Video Combine is able to detect and utilize. While it adds an extra step compared to taking an optional "mask" input on Video Combine, I think it the more correct implementation.

The Load Video node does not currently output mask information. As the memory usage has been a repeat complaint, I would like to take particular care in implementing this to ensure that it does not come with an increase in cost of memory or performance when input videos are provided that do not include any transparency information.