hamoid
commented
6 years ago I don't have the time now to research this, but links and info would make it easier for me to implement it :)
Open ubidefeo opened 6 years ago
hamoid
commented
6 years ago I don't have the time now to research this, but links and info would make it easier for me to implement it :)
hamoid
commented
6 years ago By running ffmpeg -pix_fmts I see all these pixel formats:
Pixel formats:
I.... = Supported Input format for conversion
.O... = Supported Output format for conversion
..H.. = Hardware accelerated format
...P. = Paletted format
....B = Bitstream format
FLAGS NAME NB_COMPONENTS BITS_PER_PIXEL
-----
IO... yuv420p 3 12
IO... yuyv422 3 16
IO... rgb24 3 24
IO... bgr24 3 24
IO... yuv422p 3 16
IO... yuv444p 3 24
IO... yuv410p 3 9
IO... yuv411p 3 12
IO... gray 1 8
IO..B monow 1 1
IO..B monob 1 1
I..P. pal8 1 8
IO... yuvj420p 3 12
IO... yuvj422p 3 16
IO... yuvj444p 3 24
..H.. xvmcmc 0 0
..H.. xvmcidct 0 0
IO... uyvy422 3 16
..... uyyvyy411 3 12
IO... bgr8 3 8
.O..B bgr4 3 4
IO... bgr4_byte 3 4
IO... rgb8 3 8
.O..B rgb4 3 4
IO... rgb4_byte 3 4
IO... nv12 3 12
IO... nv21 3 12
IO... argb 4 32
IO... rgba 4 32
IO... abgr 4 32
IO... bgra 4 32
IO... gray16be 1 16
IO... gray16le 1 16
IO... yuv440p 3 16
IO... yuvj440p 3 16
IO... yuva420p 4 20
..H.. vdpau_h264 0 0
..H.. vdpau_mpeg1 0 0
..H.. vdpau_mpeg2 0 0
..H.. vdpau_wmv3 0 0
..H.. vdpau_vc1 0 0
IO... rgb48be 3 48
IO... rgb48le 3 48
IO... rgb565be 3 16
IO... rgb565le 3 16
IO... rgb555be 3 15
IO... rgb555le 3 15
IO... bgr565be 3 16
IO... bgr565le 3 16
IO... bgr555be 3 15
IO... bgr555le 3 15
..H.. vaapi_moco 0 0
..H.. vaapi_idct 0 0
..H.. vaapi_vld 0 0
IO... yuv420p16le 3 24
IO... yuv420p16be 3 24
IO... yuv422p16le 3 32
IO... yuv422p16be 3 32
IO... yuv444p16le 3 48
IO... yuv444p16be 3 48
..H.. vdpau_mpeg4 0 0
..H.. dxva2_vld 0 0
IO... rgb444le 3 12
IO... rgb444be 3 12
IO... bgr444le 3 12
IO... bgr444be 3 12
IO... ya8 2 16
IO... bgr48be 3 48
IO... bgr48le 3 48
IO... yuv420p9be 3 13
IO... yuv420p9le 3 13
IO... yuv420p10be 3 15
IO... yuv420p10le 3 15
IO... yuv422p10be 3 20
IO... yuv422p10le 3 20
IO... yuv444p9be 3 27
IO... yuv444p9le 3 27
IO... yuv444p10be 3 30
IO... yuv444p10le 3 30
IO... yuv422p9be 3 18
IO... yuv422p9le 3 18
..H.. vda_vld 0 0
IO... gbrp 3 24
IO... gbrp9be 3 27
IO... gbrp9le 3 27
IO... gbrp10be 3 30
IO... gbrp10le 3 30
IO... gbrp16be 3 48
IO... gbrp16le 3 48
IO... yuva422p 4 24
IO... yuva444p 4 32
IO... yuva420p9be 4 22
IO... yuva420p9le 4 22
IO... yuva422p9be 4 27
IO... yuva422p9le 4 27
IO... yuva444p9be 4 36
IO... yuva444p9le 4 36
IO... yuva420p10be 4 25
IO... yuva420p10le 4 25
IO... yuva422p10be 4 30
IO... yuva422p10le 4 30
IO... yuva444p10be 4 40
IO... yuva444p10le 4 40
IO... yuva420p16be 4 40
IO... yuva420p16le 4 40
IO... yuva422p16be 4 48
IO... yuva422p16le 4 48
IO... yuva444p16be 4 64
IO... yuva444p16le 4 64
..H.. vdpau 0 0
IO... xyz12le 3 36
IO... xyz12be 3 36
..... nv16 3 16
..... nv20le 3 20
..... nv20be 3 20
IO... rgba64be 4 64
IO... rgba64le 4 64
IO... bgra64be 4 64
IO... bgra64le 4 64
IO... yvyu422 3 16
..H.. vda 0 0
I.... ya16be 2 32
I.... ya16le 2 32
IO... gbrap 4 32
IO... gbrap16be 4 64
IO... gbrap16le 4 64
..H.. qsv 0 0
..H.. mmal 0 0
..H.. d3d11va_vld 0 0
..H.. cuda 0 0
IO... 0rgb 3 24
IO... rgb0 3 24
IO... 0bgr 3 24
IO... bgr0 3 24
IO... yuv420p12be 3 18
IO... yuv420p12le 3 18
IO... yuv420p14be 3 21
IO... yuv420p14le 3 21
IO... yuv422p12be 3 24
IO... yuv422p12le 3 24
IO... yuv422p14be 3 28
IO... yuv422p14le 3 28
IO... yuv444p12be 3 36
IO... yuv444p12le 3 36
IO... yuv444p14be 3 42
IO... yuv444p14le 3 42
IO... gbrp12be 3 36
IO... gbrp12le 3 36
IO... gbrp14be 3 42
IO... gbrp14le 3 42
IO... yuvj411p 3 12
I.... bayer_bggr8 3 8
I.... bayer_rggb8 3 8
I.... bayer_gbrg8 3 8
I.... bayer_grbg8 3 8
I.... bayer_bggr16le 3 16
I.... bayer_bggr16be 3 16
I.... bayer_rggb16le 3 16
I.... bayer_rggb16be 3 16
I.... bayer_gbrg16le 3 16
I.... bayer_gbrg16be 3 16
I.... bayer_grbg16le 3 16
I.... bayer_grbg16be 3 16
IO... yuv440p10le 3 20
IO... yuv440p10be 3 20
IO... yuv440p12le 3 24
IO... yuv440p12be 3 24
IO... ayuv64le 4 64
..... ayuv64be 4 64
..H.. videotoolbox_vld 0 0
IO... p010le 3 15
IO... p010be 3 15
IO... gbrap12be 4 48
IO... gbrap12le 4 48
IO... gbrap10be 4 40
IO... gbrap10le 4 40
..H.. mediacodec 0 0
IO... gray12be 1 12
IO... gray12le 1 12
IO... gray10be 1 10
IO... gray10le 1 10
I.... p016le 3 24
I.... p016be 3 24
..H.. d3d11 0 0
IO... gray9be 1 9
IO... gray9le 1 9
..... gbrpf32be 3 96
..... gbrpf32le 3 96
..... gbrapf32be 4 128
..... gbrapf32le 4 128
..H.. drm_prime 0 0Both rgb24 (the one I'm currently using) and rgba seem to be supported for input and output. I wonder which codec supports transparency. The documentation doesn't mention alpha or transparency a single time...
hamoid
commented
6 years ago Here's a list of codecs that seem to support alpha: https://stackoverflow.com/questions/644684/turn-image-sequence-into-video-with-transparency#4372204
ubidefeo
commented
6 years ago thank you I've been traveling over the past weeks but will give this a shot soon :)
narkai
commented
4 years ago Hi there,
The library is great and we love de idea of feeding back the capture time (with capture.getCurrentTime()) in the code to even export intensive realtime stuff.
We try to achieve a tool that export hand recorded animations from a PGraphics to be used in a film, but we miss the ability to keep the PGraphics alpha channel.
Is it really possible with this part of the VideoExport code :
int byteNum = 0;
for (final int px : img.pixels) {
pixelsByte[byteNum++] = (byte) (px >> 16);
pixelsByte[byteNum++] = (byte) (px >> 8);
pixelsByte[byteNum++] = (byte) (px);
}For the alpha to be present in ffmpeg, shouldn't there be a fourth value like :
pixelsByte[byteNum++] = (byte) (px >> 24);
It would be great to know, we spent hours trying to tweak the settings.json file with no result...
Thanks !
hamoid
commented
4 years ago Hi! That might work. Have you tried? Do you know how to compile the library?
I think if you add the extra byte then "-pix_fmt", "rgb24" also needs to be adjusted.
I could add try that feature but I'm not sure how can I test if it works in Linux. Suggestions welcome :)
narkai
commented
4 years ago Hi thanks for quick reply,
No I haven't try yet and I don't know how to compile the library but I was going to try with javac and "-pix_fmt", "rgb24".
Is there anything special that I must know about recompiling the library ?
For Linux I have no insights, is it any different ?
hamoid
commented
4 years ago Easiest might be to open the project in IntelliJ Idea and then run the ant build script. rgb24 is 3 bytes (R G B) and you would need 4 bytes. I assume you need to use rgba instead.
For Linux... I think in general, in all OSes, playing videos with alpha can be tricky. I think they are not widely supported, but maybe I'm wrong. I don't know which video codecs support alpha, which containers support alpha. I normally use mp4 with h264. I don't know if that's an option in this case.
And about testing... I guess I need to open the video file in a video editor, place something under it and see if it's transparent. Or does VLC show the transparent parts in some special way?
narkai
commented
4 years ago Yes you are right rgb24 needs to be adjusted for something like rgb32 or rgba...
hamoid
commented
4 years ago I was reading https://blender.stackexchange.com/questions/2050/any-video-formats-that-render-with-alpha-channel and wondered if the most flexible approach would be to actually export two videos: the current one in RGB and a gray scale one for the mask. What do you think about that?
narkai
commented
4 years ago I think it's a nice solution. It could be an optional feature to export an additional file with alpha extract mask, so whenever we export we could have a mask video of exactly the same duration.
Some interesting related info (kind of) : The production team I'm working with told me that when they use video files in their editing software (Adobe Premiere), files can be slower or faster to decode depending on the codec / container. Therefore when I give them .mp4 they encode it in Apple Prores before to import them.
Apple Prores has alpha ability but it's quite slow to encode (it almost worked in my tests with VideoExport but it was black with a little alpha in it, not full alpha), that's why I'm trying with the png codec but without success for now. Looking at the codecs page, an alpha extract video might be the best solution (not plenty of the alpha keyword).
I use Photoshop to check for alpha, didn't find a way with VLC. MacOs quickview also support transparency but it only work with some codec / container.
narkai
commented
4 years ago Hi everyone, I just want to let you know I succeeded at exporting video with alpha / transparency !
🥳 🎉
I tried to compile the library with Intellij Idea, foreseeing a pull request but unfortunately it didn't work because of missing dependencies.
It took me a while to try different settings and I didn't have the time to dig deeper the compilation part.
It's only a first step in making it available in this repo and we have to think about the right way to include this capability, but I'll list here what I found to make it work :
• Important and strange (took a while to figure this one out) : using a PGraphics, alpha is not kept if buffer is refreshed with pg.clear();, we must use pg.background(0,0);
• Change ffmpeg input pixel format here as :
"-pixel_format", "argb",• Add a byte for alpha here as :
pixelsByte = new byte[img.pixelWidth * img.pixelHeight * 4];• And here as :
for (final int px : img.pixels) {
pixelsByte[byteNum++] = (byte) (px >> 24);
pixelsByte[byteNum++] = (byte) (px >> 16);
pixelsByte[byteNum++] = (byte) (px >> 8);
pixelsByte[byteNum++] = (byte) (px);
}• Getting rid of crf ffmpeg flag here as it's not used by some codecs.
• Codecs, here as (complete ffmpeg command) :
CMD_ENCODE_VIDEO_DEFAULT = new JSONArray(new StringList(new String[]{
// --------------- init
"[ffmpeg]",
"-y",
// --------------- input
"-f", "rawvideo",
"-vcodec", "rawvideo",
"-pixel_format", "argb",
"-s", "[width]x[height]",
"-r", "[fps]",
"-i", "-",
// --------------- output
"-an",
// --------------- codecs
// > codec : output name / container
// *************** tested successfully :
// > Png sequence : "output.mov"
"-vcodec", "png",
// > Quicktime animation : "output.mov"
// "-vcodec", "qtrle",
// > Apple prores : "output.mov"
// "-vcodec", "prores_ks",
// "-pix_fmt", "yuva444p10le",
// "-profile", "4444",
// "-alpha_bits", "8",
// "-quant_mat", "hq",
// "-q:v", "0",
// > Png images : "output-%04d.png"
// "-f", "image2",
// *************** to test (does not open in Photoshop to check for alpha) :
// > HAP : "output.mov"
// "-vcodec", "hap",
// "-format", "hap_alpha",
// > VP9 : "output.webm"
// "-vcodec", "libvpx",
// "-pix_fmt", "yuva420p",
// "-metadata:s:v:0", "alpha_mode=\"1\"",
// "-auto-alt-ref", "0",
// > DNxHR : "output.mov"
// "-vcodec", "dnxhd",
// "-profile", "dnxhr_444",
// "-pix_fmt", "yuv444p10le",
// > DNxHR : "output.mov"
// "-vcodec", "dnxhd",
// "-profile", "dnxhr_hqx",
// "-pix_fmt", "yuv422p10le",
// *************** VideoExport default, no alpha
// > h264 : "output.mp4"
// "-vcodec", "h264",
// "-pix_fmt", "yuv420p",
// "-crf", "70",
// --------------- container
"-metadata", "comment=[comment]",
"[output]"
// ---------------
}));Hope it helps someone. Let me know if you find / test successful codec / container match working with alpha that I'm not aware of !
hamoid
commented
4 years ago Hey congratulations for making it work! And thanks for sharing! I may try to merge this.
I started (but didn't finish) with the other approach: producing two video files, one RGB and one alpha.
Got a bit lost cleaning up and splitting the only java file into smaller ones :) Will share it when done.
narkai
commented
4 years ago @hamoid that's excellent !
Here a small export in gif from the Quicktime animation codec with alpha.
ubidefeo
commented
4 years ago Nice to see someone taking a stab at this :) I have not attempted this since I posted, but I look forward to trying again
hamoid
commented
2 years ago I created a fork that can export RGB and Alpha videos separately: https://github.com/hamoid/video_export_processing/tree/kotlinGradle
Only tested in Linux with Processing 3 and 4.
narkai
commented
2 years ago @hamoid well done 👌
narkai
commented
2 years ago I will give it a try on our project asap.
I wonder how could that be possible. I'd prefer to have an alpha channel rather than having to key the exported result.
I'm sure there's some ffmpeg set of options for this :)