Allow remuxing of AV1 MP4 video files from input into AVIF image sequences.

[dalecurtis]

Low priority feature request: Would let authors who are familiar with video work flows to quickly produce an appropriate AVIF without having to re-encode. I think it will help adoption of sequences.

[joedrago]

I'll think about how to achieve this one; it'll require some special magic at the avifEncoder level.

[dalecurtis]

Thanks! Another option is ffmpeg growing avif or libavif support, so if this is a mess feel free to WontFix.

As far as solutions go, a null/passthrough decoder is what I've used in the past for similar efforts.

[joedrago]

FWIW, avifenc has --stdin support, which can take a Y4M input pipe from ffmpeg-based tools, if you don't mind the reencode.ill look into skipping that though

[1480c1]

I would be nice if ffmpeg had avif and avifs writting support since right now it currently lacks it

btw, I found this commit while searching for avif in ffmpeg and laughed a bit internally https://github.com/FFmpeg/FFmpeg/commit/2f08dad0286f962316b2ceaf31a32423e05648ed

[dalecurtis]

There's an attempt at adding HEIF support to ffmpeg here: https://github.com/Swaraj1998/FFmpeg/commit/9a885cddb3550ab863a60d02c5fb78e4ae206cf1 -- with some minor modifications it works with AVIF files, but breaks a lot of other things. Last I heard the author was still planning to get to it, but I haven't heard anything on that since Oct 2019.

[joedrago]

@dalecurtis I know this is borderline off-topic as you want a repackaging, not a reencoding, but until I have something fancier for you, this most certainly works:

ffmpeg -i dog.mp4 -f yuv4mpegpipe -pix_fmt yuv444p - | avifenc --stdin dog.avif --cicp 1/13/6 --speed 10 --fps 30

Obviously, you'd pick whatever pixel format you wanted on the FFMPEG side (and avifenc will honor it, if supported), and you'll want to make sure the CICP and FPS matches what ffmpeg is feeding, but the results are pretty good and fast-ish!

dog.zip

[joedrago]

>avifdec --info dog.avif
Image decoded: dog.avif
 * Resolution     : 576x720
 * Bit Depth      : 8
 * Format         : YUV444
 * Alpha          : Absent
 * Range          : Limited
 * Color Primaries: 1
 * Transfer Char. : 13
 * Matrix Coeffs. : 6
 * ICC Profile    : Absent (0 bytes)
 * XMP Metadata   : Absent (0 bytes)
 * EXIF Metadata  : Absent (0 bytes)
 * Transformations: None
 * 30 timescales per second, 13.17 seconds (395 timescales), 395 frames
 * Frames:
   * Decoded frame [0] [pts 0.00 (0 timescales)] [duration 0.03 (1 timescales)]
   ...

[dalecurtis]

Thanks! It's a good work around for now. FWIW, this was one of the first questions from someone on the AMP team: https://twitter.com/cramforce/status/1298634474213842945

[dalecurtis]

This is coming up a lot during public AVIF discussions actually, so maybe it's higher priority than I thought: https://bugs.chromium.org/p/chromium/issues/detail?id=791658#c47 https://twitter.com/Lady_Ada_King/status/1303393219896659974

[dalecurtis]

https://twitter.com/cramforce/status/1298634474213842945

[joedrago]

Cyril offered a means to do this in another bug back in March using MP4Box:

https://github.com/AOMediaCodec/libavif/issues/112#issuecomment-598393096

I honestly think if someone is interesting in rearranging a BMFF so that instead of it just being a valid MP4 it is also a valid AVIF, this is a better play than adding a one-off feature to libavif that is counter to its architecture. Any tool I'd make to do this would effectively just be a slightly more turnkey subset of functionality MP4Box already offers.

Perhaps we could make a simple script for people that invokes MP4Box properly for them?

@cconcolato @dalecurtis

(Edit: To clarify, the mechanism he provides here does not reencode the video at all, and is exactly what is being requested, I believe.)

[joedrago]

(I didn't mean to hit Close)

[joedrago]

Honestly, all of the links you've provided are less specifically about AVIF and more of a shift in modern web attitudes and habits. I'm all for AVIF adoption, of course, but perhaps these questions are way more of a hint that browsers should stop making an arbitrary distinction between img and video internally?

[dalecurtis]

Thanks for that link! That's very helpful. It looks like it duplicates the first frame, but otherwise does what's on the label. Nice!

I agree, ultimately we should have support in tools like MP4Box/FFmpeg, but until we do, most folks will discover avif encoding via these tools. Having them cover common use cases will help adoption.

The img/video distinction would be a very large distraction to discuss here.

[joedrago]

As a further note, I'm not abandoning the idea of offering such a thing in libavif, I just wanted to note that people can do this with off-the-shelf tools today if they're chomping at the bit for it, as it is mostly just BMFF box reorganization.

I'll look now into how possible it'd be to accept AV1-filled MP4s as the "source image" for avifenc and try to keep the samples intact, but it might be a big effort.

[cconcolato]

Just a small note. The script using MP4Box does not re-encode the video but it is not optimal in the sense that the first image is present twice: as the primary item and as the first frame in the video. Ideally, you would want to reuse the same image data.

[joedrago]

Yes, I think Dale picked up on that. The right answer might honestly be to do what I actually downplayed earlier:

Any tool I'd make to do this would effectively just be a slightly more turnkey subset of functionality MP4Box already offers.

@cconcolato Perhaps we could leverage MP4Box.js and make some simple drop-in web interface to facilitate this? Drop in AV1 MP4s and hit Save or something?

[dalecurtis]

FWIW, it looks like the only command that's needed for Chrome to treat it as an avif is:

MP4Box -ab avis -ab msf1 -ab miaf -ab MA1B -rb mif1 -brand avis <mp4_to_be_rewritten_as_avif>

So we can probably close this as WontFix if that's all it takes.

[joedrago]

I'll label it as backlog and keep it around in case we have a "passthrough" codec implementation someday and it is easy to drop in.

[dalecurtis]

@joedrago WDYT about adding @cconcolato's method (or my 1-liner above) to your documentation on sequences so it's all in one place for developers? I'm thinking just a sub-heading "If you already have an AV1 video: " - I'm not opposed to a reply that says this should be in the MP4Box documentation though :)

[joedrago]

I'm on board with the Sequences wiki having a section named that if @cconcolato can give us any MP4Box commandline magic that will survive a ComplianceWarden check.

[joedrago]

I say that simply because I think the last MP4Box line cited here (by you, Dale) I don't believe will generate a meta box with an item. I think MP4Box needs some kind of -add-image thing for that. We'll let Cyril tell us the magic.

[cconcolato]

As Joe pointed, Dale's one liner is not correct. It wrongly indicates the file is AVIF compatible but it does not contain an AVIF Image Item, just the track. So a reader that just understands items will wrongly think it can process the file.

If the purpose is to extract a frame from a video to make it a compliant AVIF file, the command line is:

MP4Box -add-image chimera.mp4:primary:time=0 -ab avif -ab miaf -new chimera-time0.avif

If the purpose is to add an AVIF image to an AV1 video, in order to be able to use the same resource in img@src video@poster or video@src, without making a copy of the image bytes, then the command line is:

MP4Box -add-image ref:primary:tk=1:samp=1 -ab avif -ab miaf chimera.mp4

ref means that the source of the image is the same input file (chimera.mp4) and that the item should be created using the sample number given by samp (1-based number) from the track with id tk. ref also indicates that the sample bytes should not be duplicated. Note that samp has to point to a Key Frame.

Note that there are some differences in the syntaxes above (ref requires tk and samp, while the first command line uses time and no tk). We are working on harmonizing them.

[dalecurtis]

A small addendum to @cconcolato's suggestion above: For AVIF sequences, you should set the 'avis' brand to ensure proper processing by libavif (and maybe -out depending on your version of MP4Box due to the issue above) :

MP4Box -add-image ref:primary:tk=1:samp=1 -brand avis -ab avis -ab avif -ab miaf chimera.mp4 -out chimera.avif

[yaronshmueli]

I would appreciate your help muxing color + alpha AV1 mp4 files into a 'transparent animated AVIF' with the MP4Box in-place suggestion for adding images references and passing conformance tests at: https://gpac.github.io/ComplianceWarden-wasm/avif.html

Adding reference for transparent animated AVIF: https://github.com/AOMediaCodec/libavif/issues/112#issuecomment-598987424

[vigneshvg]

for the record, the latest version of ffmpeg can repackage AV1 videos into animated AVIFs without re-encoding:

ffmpeg -i input_av1.mp4 -c:v copy output.avif