search

AOMediaCodec / av1-avif

AV1 Image File Format Specification - ISO-BMFF/HEIF derivative
https://aomediacodec.github.io/av1-avif/
BSD 2-Clause "Simplified" License
451 stars 40 forks source link

Please add a 12bit "professional" profile to avif spec #128

Closed gitoss closed 3 years ago

gitoss commented 3 years ago

The current avif spec doesn't disallow 12bit encoding, but there is no profle for it - just "baseline" and "advanced".

Please add a "professional" profile to avif to make 12bit encoding official - esp. for image compression this is a competetive disadvanctage against other and emerging codecs

From the current https://aomediacodec.github.io/av1-spec/av1-spec.pdf "The Professional profile extends support over the High profile to also bitstreams with bit depth equal to 12, and also adds support for the 4:2:2 video format."

Original ticket was here: https://github.com/AOMediaCodec/libavif/issues/524

cconcolato commented 3 years ago

Just to repeat what I indicated in the other thread and expand a bit.

The current AVIF specification supports storage of 12b images, because AV1 supports encoding them. AVIF does not specify a profile for it. All this means is that you won't have a 4CC to signal that in the ftyp box. But, the configuration record in the av1C property will still indicate that it is 12 bits.

gitoss commented 3 years ago

All this means is that you won't have a 4CC to signal that in the ftyp box.

The question is: Without a dedicated profile in the spec, or at least something mentioning 12bit support, will software implementations in browsers, operating systems, ... support 12bit decoding?

If 12bit is just there for diy encoding and viewing, but not predictable distribution I can simply use jpeg-xl or whatever works best for personal storage.

cconcolato commented 3 years ago

FWIW Firefox and Chrome play 12b files fine. See https://aomediacodec.github.io/av1-avif/testFiles/Link-U/fox.profile2.12bpc.yuv420.avif

leo-barnes commented 3 years ago

Given that we have AVIF profiles/brands that map to the AV1 Main and Advanced profiles, it does seem like an omission to not have a corresponding AVIF profile for the AV1 Professional profile.

I don't really think it makes much difference in practice, but it would be good for completeness.

negge commented 3 years ago

Given that we have AVIF profiles/brands that map to the AV1 Main and Advanced profiles, it does seem like an omission to not have a corresponding AVIF profile for the AV1 Professional profile.

I don't really think it makes much difference in practice, but it would be good for completeness.

One place it will make a difference is in stopping the spread of misinformation about the AVIF format. See the article and infographic recently posted by the JPEG-XL author that claims (among other things) that AVIF does not support 12-bit images:

https://cloudinary.com/blog/time_for_next_gen_codecs_to_dethrone_jpeg

gitoss commented 3 years ago

Well, apart from every decoder (I know of) supporting 12bit, and knowledge that it does like on Wikipedia or on the Internet ... it needed a competing codec to figure out it actually doesn't :-p

However, it would if the spec people would give a heads up it'll be added as a professional profile sooner or later.

indolering commented 3 years ago

I don't really think it makes much difference in practice, but it would be good for completeness.

That sentiment hasn't been well received:

"It is allowed, you just cannot signal it yet" does not seem to be a good approach to standardization and interoperability. It implies that if I want to make an AVIF decoder, I don't just have to implement the spec, but I also need to look at what other decoders happen to do with out-of-spec cases, and make sure I do the same thing.

cconcolato commented 3 years ago

Note that it is not correct to say "you just cannot signal it yet". It cannot be signaled with a 'brand' but but it can definitely be signaled in av1C box. That said, the consensus in 2018-2019 was that interoperability would likely be achieved on the Baseline or Advanced profiles. In 2021, things seem to have changed and the spec should reflect that.

leo-barnes commented 3 years ago

This topic was discussed during the last meeting. Here's a brief summary of the discussion.

Cyril has proposed a pull request (#130) to better clarify what is and is not supported in AVIF. In short, AVIF supports everything that AV1 supports, which includes 12-bit encoding. The current software decoder implementations also support more or less all of AV1, which can be seen if you try to decode a 12-bit AVIF file in a web browser.

The point of creating the AVIF profiles (and really the MIAF profiles as well) was mainly to create interoperability points for hardware implementations to converge on. From the discussion in the meeting, it seems unlikely that AV1 hardware decoders will support 12-bit data. From this perspective it doesn't seem to make sense to create a specific profile for it, since it would give file creators a false impression of being able to expect hardware acceleration.

We're definitely open to hearing about other use-cases for a constrained level 12-bit profile, but unless we can come up with one, we're left with two choices:

  1. We think Cyril's proposed changes are enough to clearly indicate that 12-bit data is something that is supported in AVIF, but likely won't be getting HW acceleration. No need for a separate 12-bit profile.
  2. We create a new AVIF "full" profile that maps to the AV1 Professional profile with maximum level. It is then possible for a file creator to clearly mark a file as using features not present in the existing AVIF profiles. This is more explicit than simply not specifying an AVIF profile.
gitoss commented 3 years ago

Imho Cyril's clarifications in the documentation are nice, but don't counter the main point - either a real concern, or something to discredit avif: Is there a reliability or at least valid expectation that out-of-profile avif files can be decoded in all upcoming implentations ... if not in hardware, then in software? A mere "it's a valid avif file" is like saying "You are allowed to ski outside the slopes, but if something happens, then it's on you'.

A "full" profile with maximum level is an excellent solution if "professonal" profile decoding won't happen in hardware anyway. Beyond the explicit inclusion of 12bit, it would clarify that avif picture dimensions aren't as constrained as it appears just looking at the existing profiles. Is there any reason not to add such a "full" profile?

leo-barnes commented 3 years ago

Is there any reason not to add such a "full" profile?

The main reason would be that profiles in other specs are usually meant to specify a restricted subset of tools/features and that this new profile would not actually be a subset.

I do see your point in that it might be clearer for implementations to be able say "we support all AVIF profiles" rather than "we support all AVIF profiles and also all features of AV1".

gitoss commented 3 years ago

To the layman, this whole profile and brand business with image formats is a bit confusing esp. because there's no hardware decoding of images yet. Further clarification would be welcome and boost avif adoption, even if it doesn't submit to a substandard-"profile" logic.

The ancient jpeg format seem to have profiles, probably somewhere in a spec for sale https://jpeg.org/jpeg/ ... but no one seems to have noticed because its a de-facto standard. I do see that this approach falls appart with something like tiff and its gazillion of possible implementations and additions.

Webp as the first video-derived format doesn't seem to have profiles - there is an unoffical feature-set like animation that might not be supported. Same for avif - current Firefox implementation struggle is not about profiles, but about missing undefined "features" https://bugzilla.mozilla.org/show_bug.cgi?id=1696045

The heic format didn't get any traction outside the Apple world, so I guess no one noticed about "brand" and profiles: https://nokiatech.github.io/heif/technical.html ... it required the Firefox avif implementation to crash to realize that people didn't purchase the spec document: https://bugzilla.mozilla.org/show_bug.cgi?id=1689806

With a "full" profile, at least some confusion would be cleared up concerning bit depth and image dimensions - leaving just enough confusion about features be be handled during implemtation :-)

leo-barnes commented 3 years ago

The heic format didn't get any traction outside the Apple world

It probably has more traction than you realize. :) It's supported in pretty much all recent OS versions (Windows, Linux, Android, everything from Apple). It's just not supported in browsers.

gitoss commented 3 years ago

Well, I stand corrected - it's just that I'm on Windows Enterprise Microsoft didn't grace LTSC with HEIC support. I do agree browser support isn't there yet :-> https://www.caniuse.com/heif

I'm sticking with my point that because of legacy file formats, esp. non-video derived codecs, profiles for image codecs isn't a well-known entity and will lead to the fallacy that no profile = no support, no matter how many annotations the specs get :-\

indolering commented 3 years ago

I'm just a fan-person with zero implementation experience, so please let me know if this is anti-helpful and I will refrain from commenting further. However, I've been part of similar discussions on the AV1 subreddit and wanted to share two ideas that came up.

IIRC the small number of profiles was a deliberate feature of AV1: the engineering effort required to support all of the h.26x profiles is akin to supporting multiple codecs. The fact that everyone already supports 12 bit decoding suggests this isn't a large burden for implementors. But professional variant (.avifpro?) might be a useful place for other features; like 32 bit color, a large number of channels, massive block sizes, etc.

On-the-other-hand ... WebP and friends are apparently not decoded in hardware in most browsers. However, Apple's use of HEIC is constrained in ways to make it more friendly to hardware decoding, such as fixing tile boundaries to 256 blocks so that hardware doesn't need to be re-initialized. This would seem to make the case for a similarly constrained profile (main-still?) that makes it ergonomic/efficient to decode images using GPGPU or fixed-function hardware decoders.

leo-barnes commented 3 years ago

Hi @indolering,

The main point of the existing two AVIF profiles was to give reasonable sets of features for HW implementers to converge on, exactly like you describe in your last paragraph. But from the discussion in the meetings it seems unlikely that 12-bit support will be implemented in HW.

So the question is:

indolering commented 3 years ago

So my post was a bit off-topic. I think adding the professional profile to AVIF that matches the AV1 professional profile is a no-brainer, as it's already supported everywhere and no one is using hardware to decode images anyway.

A constrained main-still or maybe main-still-mobile profile would need to be as hardware friendly as possible to make it as ergonomic as possible to decode in hardware. The primary target would be mobile: where bandwidth and battery life are at a premium and images are small enough that 10-bit color banding, 4:2:0 pixel offsets, and small tile sizes won't be noticeable. But I have ZERO implementation experience and I am basing this off of other people's observations about Apple's use of HEIC.

There might also be demand for another profile: professional-still that supports things like massive tile sizes, 14/16/32 bit color, and other niceties that are only useful in professional image manipulation settings. This might actually be better off as a distinct file format (as it smells of MPEG-esque profile proliferation) or punted to AV2. Others wanting the One True Image Codec think a distinct file format would actually hurt maketshare relative to jpeg-xl and the like. But I think I've given enough armchair expert testimony for one day 😅.

leo-barnes commented 3 years ago

@indolering Not sure I understand your ask here:

A constrained main-still or maybe main-still-mobile profile would need to be as hardware friendly as possible to make it as ergonomic as possible to decode in hardware. The primary target would be mobile: where bandwidth and battery life are at a premium and images are small enough that 10-bit color banding, 4:2:0 pixel offsets, and small tile sizes won't be noticeable. But I have ZERO implementation experience and I am basing this off of other people's observations about Apple's use of HEIC.

AVIF already has two existing profiles, where the whole point is to limit the set of included features to things expected to be implemented in HW. What would main-still/main-still-mobile include that is not already solved by the existing profiles?

There might also be demand for another profile: professional-still that supports things like massive tile sizes, 14/16/32 bit color, and other niceties that are only useful in professional image manipulation settings. This might actually be better off as a distinct file format (as it smells of MPEG-esque profile proliferation) or punted to AV2. Others wanting the One True Image Codec think a distinct file format would actually hurt maketshare relative to jpeg-xl and the like. But I think I've given enough armchair expert testimony for one day 😅.

To the best of my knowledge, AV1 doesn't support more than 12 bits, and since AVIF is based on AV1, we can't really add a profile for things that the codec doesn't support. As @cconcolato noted in #130, AV1 supports a maximum size of 64k by 64k. Tiles larger than that are usually not very practical to work with. In fact, the whole point of using tiles in the first place is to give you things like random access decoding and only decoding smaller subsets of an image in full resolution, which is needed when dealing with gigapixel images. Having tiles that are too large actually makes it harder to work with very large images.

But this still doesn't answer the main question: Why do we need a profile for this? It's already supported by the file format itself. What does a user gain by marking an image as using a non-constrained profile?

The main argument so far has been that unless you read the spec very carefully, it's hard to realise that 12-bit images are actually supported. Are there any other arguments for a constrained or unconstrained 12-bit profile that we have missed?

gitoss commented 3 years ago

@leo-barnes main question:

My issue was indeed some jpeg-xl comparison claiming that avif doesn't officially support 12bit because not all decoders can be expected to decode it, even if you can encode it just fine. It was claimed you can encode a lot of jpeg files, too, that cannot be read by most decoders around.

So the idea was that an offical profile including 12bit would signal this and make it more likely that upcoming decoder designs include 12bit - if not in hardware (if that should ever happen or have an real world advantage) then at least in software.

This profile could be a "professional" profile - so software decoders wouldn't have to deal with very large images / image tiles - and/or al "full" profile reminding decoder devs that they should include software decoding support for everything avif can encode.

I do see that the spec is now ammended, so if avif profiles is only about hardware support this is probably sufficient ... if you don't get taunted by the "no profile means no reliable support" claim. https://github.com/AOMediaCodec/av1-avif/commit/7536f4bccd90010e0152965f271fdf89c8563c62

leo-barnes commented 3 years ago

I do see that the spec is now ammended, so if avif profiles is only about hardware support this is probably sufficient ... if you don't get taunted by the "no profile means no reliable support" claim.

We discussed it some more in the last meeting. Given that most implementations in use today are using SW decoders that do support 12-bits, the current support is good. It remains to be seen what future HW will support. It may indeed turn out that we can't expect widespread HW support for 12-bits, but that wouldn't change even if we created a dedicated profile for it. If we do see that HW tends to support 12-bits, we may want to add a restricted 12-bit profile in the future that tries to set a reasonable level that is supported by most HW.

Closing for now since there isn't really anything actionable we can do at this time.