Potential issues with two MIME types

[annevk]

1. There's no precedent for image formats with distinct MIME types for their still and animated variants, e.g., GIF, PNG, and WebP(?). Why was a different decision made here?
2. As far as I could tell from the document there's no distinct processing model defined for consumers. Producers are told to use specific MIME types, but there are no requirements for consumers getting MIME types with certain bits being wrong. Absence such requirements, consumers will likely use the same decoding path for both, and producers might eventually stop caring about the difference as well as there's no discernible difference.

[cconcolato]

There's no precedent for image formats with distinct MIME types for their still and animated variants, e.g., GIF, PNG, and WebP(?). Why was a different decision made here?

IANA already has entries for image/heic and image/heic-sequence, or image/heif and image/heif-sequence. We follow the same approach here.

As far as I could tell from the document there's no distinct processing model defined for consumers.

The requirements to support the static image format and the animated image (aka image sequence) format are quite different. For example, for the image sequence format, a reader needs to support the concept of ISOBMFF track. For the static image format, a reader needs to support the concept of ISOBMFF item. It is possible that a reader will have support for one and not the other, thus the mime type difference.

Producers are told to use specific MIME types, but there are no requirements for consumers getting MIME types with certain bits being wrong.

If a file containing tracks (i.e. image sequences) is advertised using the static MIME type and a reader that only supports the static MIME type receives it, there is nothing it can do. And vice-versa. That should be an incentive for producers to set the MIME type correctly.

[annevk]

It seems somewhat unfortunate to follow the one precedent that's not really a web format, but okay. What if a reader supports both formats? What's the processing model in that case?

[Kagami]

We've just had discussion with @negge on IRC regarding Alliance for Open Media Patent License and two different mimes of AVIF files.

The question basically is: since you have two different mimes for AVIF files (image/avif being the still images and image/avif-sequence for sequences) are you allowed to ship different decoder implementation for that mimes according to the license?

Notably this terms sounds like you're not allowed (emphasis mine):

2.3. Decoder. “Decoder” means any decoder that conforms fully with all non-optional portions of the Specification. 2.6. Implementation. “Implementation” means any implementation, including the Reference Implementation, that is an Encoder and/or a Decoder. An Implementation also includes components of an Implementation only to the extent they are used as part of an Implementation.

But for a practical implementation of still-only AVIF (image/avif) for e.g. a web browser it makes perfect sense to ship reduced version of AV1 decoder (i.e. intra-only) to minimize binary size and things like that. It doesn't mean you don't have fully-compliant decoder, you're just loading optimized version of decoder for the specific AV1 stream. Is this allowed?

[negge]

But for a practical implementation of still-only AVIF (image/avif) for e.g. a web browser it makes perfect sense to ship reduced version of AV1 decoder (i.e. intra-only) to minimize binary size and things like that.

To be clear, the discussion was about a WASM compiled version of dav1d that supported just the image/avif mime type. Mozilla does and will always ship a fully-compliant AV1 decoder in the browser.

A full log of the conversation in the public #dav1d channel can be found here:

https://xiph.org/~unlord/dav1d_wasm.txt

As @annevk asked above, some clarity around the processing model for the two normatively described mime formats would be helpful. It is unclear how these should be processed differently, if at all.

[cconcolato]

The Still Picture WG discussed this issue today. No decision has been reached yet. The group would like to gather more feedback from browser vendors, regarding what they will ship as part of their image code path. Will they support the image format only, or both? Also, the IANA registration is not yet final, so there might be an opportunity to change it.

[annevk]

Thanks! As I understand it, Mozilla would like to support still and animated, ideally under a single MIME type as we'd like to have a single code path for both, imitating GIF and PNG.

[jaikrishnan]

Based on the ongoing implementation work for AVIF in Chrome, there's alignment with what the Mozilla folks said above. The current mime type designations are unlikely to be respected wrt processing paths within the tag. AVIF handling will likely be similar to the GIF / PNG approach outlined above. Even combined still + animated formats like WebP are handled without mime type distinction.

In the AOM working group, there's a proposal to merge both AVIF mime types, which would remove a mechanism for sequence creators to express intent for how the images should be viewed. Does anyone with HEIF experience think this is a bad idea? Do any parsers treat heif and heif-sequence differently? AFAICT, Google parsers do not, but I'm curious about other implementations.

[aklemets]

I don’t see what problem that would be solved by removing one of the MIME types.

If the implementation in Chrome can handle both still images and image sequences, then that’s great for them. But it does not mean that content creators should not be allowed to express their intent for how the images should be viewed.

Anders

---

From: Jai Krishnan notifications@github.com Sent: Wednesday, July 31, 2019 9:49:43 AM To: AOMediaCodec/av1-avif av1-avif@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: Re: [AOMediaCodec/av1-avif] Potential issues with two MIME types (#59)

Based on the ongoing implementation work for AVIF in Chromehttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.chromium.org%2Fp%2Fchromium%2Fissues%2Fdetail%3Fid%3D960620&data=02%7C01%7C%7Ccf31e1a217394f8eb08e08d715d71679%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637001885841631751&sdata=X71hZFRb5GMgM5yC%2B9lpLm804rWbzv5HRXm8nQ56PCI%3D&reserved=0, there's alignment with what the Mozilla folks said above. The current mime type designations are unlikely to be respected wrt processing paths within the [X] tag. AVIF handling will likely be similar to the GIF / PNG approach outlined abovehttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAOMediaCodec%2Fav1-avif%2Fissues%2F59%23issuecomment-486438148&data=02%7C01%7C%7Ccf31e1a217394f8eb08e08d715d71679%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637001885841641762&sdata=Bcg%2BHHapi3UFBlyk%2Fqp7gKGKheF%2BNsXnK42iaQYh3mM%3D&reserved=0. Even combined still + animated formats like WebP are handled without mime type distinction.

In the AOM working group, there's a proposal to merge both AVIF mime types, which would remove a mechanism for sequence creators to express intent for how the images should be viewed. Does anyone with HEIF experience think this is a bad idea? Do any parsers treat heifhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.iana.org%2Fassignments%2Fmedia-types%2Fimage%2Fheif&data=02%7C01%7C%7Ccf31e1a217394f8eb08e08d715d71679%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637001885841641762&sdata=dJMKlIRVivXdeaGN2CTdvWk9D42NCdfBfUwlC4Ym0oI%3D&reserved=0 and heif-sequencehttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.iana.org%2Fassignments%2Fmedia-types%2Fimage%2Fheic-sequence&data=02%7C01%7C%7Ccf31e1a217394f8eb08e08d715d71679%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637001885841651767&sdata=NNkYpt2iMQKug5D4Kt38kTHJdUAThS7MGHUs1Bst5Jk%3D&reserved=0 differently? AFAICT, Google parsers do not, but I'm curious about other implementations.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAOMediaCodec%2Fav1-avif%2Fissues%2F59%3Femail_source%3Dnotifications%26email_token%3DAGTLPAWOCY5BE3JYIFOJEALQCG7CPA5CNFSM4G7QMHEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3H3XLI%23issuecomment-516930477&data=02%7C01%7C%7Ccf31e1a217394f8eb08e08d715d71679%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637001885841661772&sdata=EhsipoT2ctemoATEp5j27NOomFDamfVivOyt8JPAIIo%3D&reserved=0, or mute the threadhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAGTLPAWMU53J7YC5VBOMGA3QCG7CPANCNFSM4G7QMHEA&data=02%7C01%7C%7Ccf31e1a217394f8eb08e08d715d71679%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637001885841681788&sdata=PVDPWa2ikxAZuEqPFcGVDizAV5FvNYnefjr6mqYu8i0%3D&reserved=0.

[annevk]

Anders, could you explain the purpose of expressing intent if most consumers ignore the intent? (And as a consequence, mislabeled intent becoming likely to occur, leading to consumers that did use it for some purpose having to look at the bytes instead.)

[aklemets]

Hi Anne,

The reason for having two MIME types is mostly historical but if does allow for a separate implementations on the client.

Let’s say, someone has an implementation for previewing media files (basically, showing thumbnails.) This implementation may use an image decoder stack to preview images, and a separate video decoder stack preview video files. For video files, they would show a brief animation of the video frames.

Having two separate MIME types allows the above implementation to invoke their video decoder to display an animated preview of .avifs files. If the content creator used the wrong MIME type, then maybe only a still image thumbnail is shown. The experience is not broken if someone accidentally uses the wrong MIME type, it is just not ideal.

There are other cases of multiple overlapping MIME types and file extensions. For example, there is .m4v for MP4 video files and .m4a for MP4 audio files. But there is also .mp4 which could be either an audio file or a video file, or both. Using the specialized MIME type (.m4v or .m4a) usually results in the best user experience, but it still .mp4 still works. For example, a file with .mp4 extension might not be correctly categorized as an audio or video file initially, but once the user clicks on the file it is still played correctly.

Having said this, there is a historical aspect for having two distinct MIME types for still images and image sequences which does not apply to AVIF: In the HEIF base spec, there is no requirement that a HEIF file that contains image sequences contain any still images. Thus, a client that only supports displaying still images may not be able to render a HEIF image sequence file at all. Having two separate MIME types makes a lot more sense in that case.

But the MIAF specification subsequently required that all MIAF-compatible HEIF files contain at least one still image (the Primary image.) And AVIF is defined as an “extension” to MIAF. Since AVIF has inherited this rule from MIAF, there is much less of a need to have two separate MIME types – a client that does not support sequences will always have a still image as a fallback option. Nevertheless, there could still be some minor advantages to having two separate MIME types even for AVIF (the file preview example that I gave, above.)

Anders

---

From: Anne van Kesteren notifications@github.com Sent: Monday, August 5, 2019 8:14:02 AM To: AOMediaCodec/av1-avif av1-avif@noreply.github.com Cc: Anders Klemets aklemets@hotmail.com; Comment comment@noreply.github.com Subject: Re: [AOMediaCodec/av1-avif] Potential issues with two MIME types (#59)

Anders, could you explain the purpose of expressing intent if most consumers ignore the intent? (And as a consequence, mislabeled intent becoming likely to occur leading to consumers that did use it for some purpose having to look at the bytes instead.)

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAOMediaCodec%2Fav1-avif%2Fissues%2F59%3Femail_source%3Dnotifications%26email_token%3DAGTLPAVGY5FRMLHAVSQ33ALQDA7TVA5CNFSM4G7QMHEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3SEHEI%23issuecomment-518275985&data=02%7C01%7C%7Ca5d85ec3d7a84d31c8ad08d719b78c7b%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637006148426434898&sdata=F1kN7%2BZVtqGNuF4e7RH7pKhpKmQyX9E%2BnCVCbKtXiUM%3D&reserved=0, or mute the threadhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAGTLPAXC7O44HRDJ45VNWRTQDA7TVANCNFSM4G7QMHEA&data=02%7C01%7C%7Ca5d85ec3d7a84d31c8ad08d719b78c7b%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637006148426444909&sdata=2bqVAecec%2BY9aXewdVZILFBmlxKtTe%2F6jtqsVU%2BIDOw%3D&reserved=0.

[annevk]

It only really makes sense to me if you expect to end up with different decoding libraries and even then if the amount of mislabeled content gets too large the incentives will be such to switch decoding stacks so you don't have the problems you run into with switching on MIME types. (This would be different if you forced all consumers to abide by the MIME type, but it doesn't seem like that's in the cards or necessarily desirable from an implementation perspective.)

[annevk]

I made a mistake in OP. PNG has both image/png and image/apng. They do use the same code path (and in fact we rely on sniffing exclusively there), but there are distinct MIME types for labeling if folks want to do so and Chrome in fact has image/apng in its Accept header.

[jaikrishnan]

My understanding is the Android ecosystem does allow app developers to implement heif and heif-sequence differently, and would be interested in offering a similar distinction for AVIF. I can dig in further if there's still push back, but given the amount of time we have left the spec like this, I'd vote we continue allowing for both AVIF mime types and finalize it with IANA.

[annevk]

Can we figure out a story where we don't have to add both to the Accept header for the foreseeable future? (Ideally we'd only include the shorter one.)

[wantehchang]

I added "image/avif" to Chromium's HTTP Accept request headers. (See https://crbug.com/1073107 for the source code changes.)

When AVIF is disabled (the default) in Chromium, the HTTP Accept header for navigation requests is:

  "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,"
  "image/apng,*/*;q=0.8";

and the HTTP Accept header for image subresource requests is:

"image/webp,image/apng,image/*,*/*;q=0.8"

Note that these HTTP Accept headers include the wildcards "*/*" and "image/*", and do not include all the image formats that Chromium supports. Therefore, the presence of any "image/<foo>" in these HTTP Accept headers is intended to signal an additional image format on top of an implicit baseline of image formats that all versions of Chromium support, which includes GIF, JPEG, PNG, etc.

Given this understanding, and since we intend to support both AVIF static images and AVIF image sequences in Chromium from the beginning, I concluded that I needed to add just one image mime type to Chromium's HTTP Accept headers to indicate support of AVIF, and naturally I added the shorter mime type "image/avif".

[wantehchang]

Chromium can work correctly with either one or two mime types for AVIF. Having added AVIF decoding support to Chromium, I personally would prefer to have just one mime type ("image/avif") and just one filename extension ("avif"). If we really must have two mime types, I would prefer to shorten the image sequence mime type to "image/avifs".

Note: The current Chromium implementation treats "image/avif" and "image/avif-sequence" interchangeably for the most part, so mislabeled AVIF images are unlikely to be detected by Chromium or cause Chromium to malfunction.

[jaikrishnan]

Note: The current Chromium implementation treats "image/avif" and "image/avif-sequence" interchangeably for the most part, so mislabeled AVIF images are unlikely to be detected by Chromium or cause Chromium to malfunction.

@annevk does this sufficiently guide the web ecosystem on how to handle multiple mime types? (ie. ignore them?)

[annevk]

I'm not sure how an implementation can serve as the guide. It would have to be standardized, right?

[jaikrishnan]

Decision in the 6-10 meeting to move forward with a single mime type. PR to be submitted and reviewed

[cconcolato]

I implemented the removal of the MIME type. I don't recall what was the conclusion the regarding the file extension "avifs", so I kept it. Let me know if it needs to go too.

[annevk]

I think the same arguments apply there. It matters less as file extensions don't carry authority on the web, but on other systems you might run into the same kind of issues.

[wantehchang]

Cyril:

The June 10th, 2020 meeting notes say "Ideally reduce to one file name extension and mime type".

The file extension is associated with the MIME type in some contexts. If we remove the "image/avif-sequence" MIME type, it is better to also remove the corresponding "avifs" file extension.

[cconcolato]

Thanks @wantehchang I will update the PR.

[imsys]

I just want to add that Firefox currently doesn't support animated avif.

<picture>
    <source srcset="ship.avifs" type="image/avif">
    <source srcset="ship.webp" type="image/webp">
    <source srcset="ship.apng" type="image/apng">
    <img src="ship.gif" alt="ship">
</picture>

So, if I use the above, both chrome and Firefox will pick avif, but Firefox will show a still image. Chrome does not support image/avif-sequence or image/avifs MIME tags, so I think I can't use an animated avif in this case without adding some javascript to check the user browser.

[baumanj]

So, if I use the above, both chrome and Firefox will pick avif, but Firefox will show a still image.

Can you share the ship.avifs file you're using? In this case I believe the right way to signal that the intended use of the image is an animation is by using avis in the major_brand field of the ftyp box. That allows for browsers that don't support the avis brand, but do support the avif brand to fall back appropriately.

If you're wondering why a browser wouldn't just assume this when encountering an AVIF with an image sequence, it's because it's possible to have an AVIF that contains both a still image primary item and an image sequence. In the avif case the sequence may be something akin to Apple's "Live Photo" feature and in the avis case it may be an animation with a poster frame.

I realize this isn't implemented yet, but if it's an important use case to you, I encourage you to click the "vote" button on https://bugzilla.mozilla.org/show_bug.cgi?id=1739210#vote-btn

[imsys]

I put it in IPFS: ship.avifs test.html

This is how it looks in a hex editor: