Graceful fallback for future image types

[anforowicz]

To gracefully handle future image types (e.g. image/foobar) would it be desirable to change step 5 from

If mimeType's essence is "image/svg+xml", then return true.

to

If mimeType's essence starts with "image/", then return true.

?

I guess when a new image mime type is introduced then https://mimesniff.spec.whatwg.org/#image-type-pattern-matching-algorithm will be updated. OTOH, image/svg+xml is already not covered there and needs to be special-cased.

[annevk]

I was hoping we would never update the sniffing algorithm and require MIME types for new media types that have a different signature (AVIF reuses an existing signature as it happens).

[annevk]

Closing this as I think this is agreed upon. https://github.com/w3ctag/design-principles/pull/263 also documents this.

[anforowicz]

@annevk, could you help me understand how JPEG XL could be adopted in an ORB-compatible way? The new image format is not yet standardized, but AFAIU:

• The new MIME type will be image/jxl (based on searching the reference implementation)
• The binary encoding will begin either with (based on ReadSignature in jxl/decode.cc):
  • 0xff, 0x0a - JXL_SIG_CODESTREAM
  • 0x00, 0x00, 0x00, 0x0c, J, X, L ... - JXL_SIG_CONTAINER

AFAIU, current ORB algorithm would block such resources:

• The image/jxl MIME type is not an "An opaque-safelisted MIME type", so step 3.i. won't return true
• The image type pattern matching algorithm doesn't list any matching patterns, so step 6 won't return true
• The MIME type starts with image/, so step 11 will return false (i.e. blocking the response)

FWIW, it seems that JPEG XL does follow the w3ctag/design-principles#263:

• A new MIME type is defined
• It does not extend pattern matching algorithm
• (presumably) strict MIME type (e.g. no sniffing) will be enforced for the newer format

[annevk]

I think we have two options:

1. Safelist the MIME type.
2. Require folks use CORS (and also enforce the MIME type).

I somewhat prefer the latter for anything new, but I can understand if that's considered too prohibitive.

[anforowicz]

/cc @acolwell

[jonsneyers]

I think we have two options:

1. Safelist the MIME type.
2. Require folks use CORS (and also enforce the MIME type).

I somewhat prefer the latter for anything new, but I can understand if that's considered too prohibitive.

Safelisting the image/jxl media type (MIME type is the old name for this, IANA calls it "media type" now) seems like a reasonable thing to do.

Does option 2 imply that you cannot have cross-origin images by default? That would break a large fraction of the web.

I was hoping we would never update the sniffing algorithm and require MIME types for new media types that have a different signature (AVIF reuses an existing signature as it happens).

I don't see how AVIF reuses an existing signature. At least, the "image type pattern matching algorithm" in section 6.1 of https://mimesniff.spec.whatwg.org/#image-type-pattern-matching-algorithm would return "undefined" for an AVIF file.

In general, I don't really see how sniffing can be completely avoided in practice. I would expect that in some scenarios, there is no media type available, e.g. when doing local file system browsing — afaik if you want file://path/to/foo.jpg to work, the only way to do that is to inspect the file (but I guess you could argue that in that case the browser is playing both server and client and should conceptually do the sniffing in the 'server part').

[annevk]

It would mean that image/jxl would not work cross-origin by default. This would be similar to module scripts.

As for AVIF, it's my understanding it fits within https://mimesniff.spec.whatwg.org/#matching-an-audio-or-video-type-pattern. It's the union of these algorithms that matters.

Local file system is somewhat out-of-scope of the web platform, but if .jpg maps to image/jpeg and that ends up in the image decoder I would expect it to work if it was actually image/jxl. It depends. 😊

[jonsneyers]

It would mean that image/jxl would not work cross-origin by default. This would be similar to module scripts.

So if you have an img tag with a cross-origin uri, it would work if the returned content-type is, say, image/png, but not (by default) if the returned content-type is image/jxl? The request would have to be made anyway because the content-type is only known when something is actually returned, and the returned jxl would then have to be discarded, showing a broken image instead of decoding the image.

As for AVIF, it's my understanding it fits within https://mimesniff.spec.whatwg.org/#matching-an-audio-or-video-type-pattern. It's the union of these algorithms that matters.

I don't see which pattern would fit for AVIF. The one for MP4 is somewhat close (since MP4 is also ISOBMFF-based), but AVIF does not have an ftyp equal to mp4, so it does not actually match.

Local file system is somewhat out-of-scope of the web platform, but if .jpg maps to image/jpeg and that ends up in the image decoder I would expect it to work if it was actually image/jxl. It depends. 😊

If you expect that to work, then would you also expect images that are returned with content-type: image/jpeg but actually contain an avif/jxl bitstream to end up in the image decoder and work if the browser happens to support avif/jxl?

I think that if specifically image/avif and image/jxl would not work cross-origin but other image types just keep working (to not break the existing web), it would potentially have the undesirable side-effect of actually encouraging servers to lie about the media type of what they return just to circumvent the cross-origin blocking that would otherwise happen. "Oh, so you Accept: image/jxl? OK, here's a Content-type: image/png for you, nudge nudge wink wink"

[annevk]

That's a good point, that would have to not work.

[jonsneyers]

That's a good point, that would have to not work.

Yes but it would be nice if a browser would be able to know before it makes a request and gets to see the Content-type of the response whether it was supposed to make that request in the first place or not.

I think it would make sense to either allow any image/* content-type in cross-origin images, or not allow cross-origin images at all, but the latter would require some transition strategy to avoid breaking the web.

[yoavweiss]

It would mean that image/jxl would not work cross-origin by default. This would be similar to module scripts.

The usage patterns of module scripts and images are extremely different. Also, for module scripts we can "magically" turn those fetches into CORS-enabled ones without developers having to add a crossorigin attribute for them.

It's clear to me what would be the user benefit of enforcing a strict MIME type check on JXL (and anything we can really): it reduces the attack surface that MIME sniffing exposes us to.

It's not clear to me what would be the user benefit of a CORS requirement on JXL and JXL alone. At the same time, (as stated elsewhere), the deployment hurdles would result in the format being practically unusable, as automatic transcoding of e.g. JPG to JXL would not be possible, and developers would have to be burdened with moving their <img> elements to include a crossorigin attribute. Even the pattern of using <picture> for type selection won't work here, unless we add a crossorigin attribute on <source> and get developers to know they need to add it only for this case.

[annevk]

The concrete benefit is limiting the data that can leak across origins. I.e., it would mean that correctly labeled image/jxl images are safe from a certain class of attacks. I think you can make a reasonable case that an exception ought to be granted here and that they should be subject to such attacks, but that should be a conscious decision. As part of that decision ideally you would also evaluate what it means for future formats as everyone will cite this as precedent that their feature deserves to be exempted from same-origin policy considerations too.

[yoavweiss]

I understand the benefits of CORS in general, but protecting one format while not protecting others seem to provide marginal benefits at best.

I think you can make a reasonable case that an exception ought to be granted here

I think that we'd be better off finding ways to e.g. opt-in documents to request all their subresources using CORS, than to draw that line on a format-by-format basis. CO{E/R}P or credentiallessness seem like plausible ways in which we can limit on-by-default cross-origin leaks.

[annevk]

The purpose of ORB is that the formats that are not protected are on a small list. What you seem to be asking for is extending that list. Stated another way, if we enforce a MIME type for JPEG XL and do not allow sniffing, it would be protected by default and ORB would return a network error for it if it's fetched across origins without CORS.

[jonsneyers]

It would be 'protected' in the sense that it wouldn't work, which is one way to be more secure but not my favorite way to do it.

The leaked data we are talking about are the pixels that could be accessed. It does not help much to force those pixels to be delivered via JPEG or PNG instead of a more modern format. It does not help that JPEG XL gets 'protected' but the pixel leak still happens because the server will have to respond with a JPEG. The only security benefit is that it will take slightly longer for the leaked pixels to arrive (because likely the old format takes longer to transfer than the new format), but I would say that is an extremely marginal benefit.

It does however hurt users if legitimate cross-origin image serving use cases are de facto limited to old image formats. Note that even if cross-origin image servers set CORS: *, there is still the significant performance penalty of requiring two requests for one image (one extra to do the CORS check) which would remove a lot of the performance benefits of using a new format. (the penalty could be avoided by doing <img crossorigin=anonymous> in the markup, but that has other problems: it is not always feasible to change markup and any behavior that relies on cookies would break)

[annevk]

There's no CORS preflight for credentialed GETs. And again, I'm not saying an exception cannot be made, but that it has to be conscious and with a path forward.

[jonsneyers]

Maybe I am missing something, but why can not just image/* be whitelisted instead of whitelisting image/svg+xml plus the specific image codecs mentioned in https://mimesniff.spec.whatwg.org/#image-type-pattern-matching-algorithm ?

Note that JPEG 2000 in Safari and AVIF in Chrome are already examples of image formats for which exceptions seem to have been made in practice.

[jonsneyers]

As far as I understand, credentialed GETs are not typical for cross-origin img requests. Edit: my understanding is improving and the previous sentence is probably wrong :)

I agree that a path forward is worth thinking about, but I don't think it's a good path forward to wait for all the current 'exceptions' — JPEG, PNG, GIF, SVG, WebP, BMP, ICO, and probably (though not currently included) JPEG 2000 and AVIF too — to die a natural death, become deprecated and eventually unsupported in all browsers, and have only protected-by-default JPEG XL and future image formats left. That's just not going to happen, or at least not in the next 30 years or so.

[veluca93]

+1 on this: I don't believe "let's make one (or two) image formats behave differently from everything else" is the best path forward for enabling CORS on image resources.

Also, I believe it would significantly violate the Principle of Least Surprise if such a behaviour became common-place.

[jonsneyers]

Perhaps it could be useful to try to collect data on uses of cross-origin img loads that rely on cookies.

Maybe a way forward could be to make crossorigin=anonymous the default on img tags at some point.

As for media type sniffing: I am not exactly sure what the security issue is there. If an old server serves a jxl or avif without the correct media type (likely saying it is unknown), should that mean a broken image gets shown even if the media type was set correctly in the srcset? Because that could be a significant hurdle to adoption, and it could also lead to people renaming their avif or jxl files to end with .jpg or .png just so old servers give it a media type that makes it get treated as an image.

[anforowicz]

Can not just image/* be whitelisted instead of whitelisting image/svg+xml

Allowlisting image/ (and audio/, video/*, etc) is my preferred plan of action and what I have implemented in a (very WIP) Chromium CL at https://crrev.com/c/3203018. /cc @csreis

The purpose of ORB is that the formats that are not protected are on a small list.

That is a desirable goal, but I think I am resigned to allowing cross-origin image/video/audio fetches (and hoping that sensitive multimedia resources are protected using CORP).

[annevk]

@anforowicz I don't think that approach really helps with this problem as the image fetching pipeline ignores MIME types (except for image/svg+xml). This is much more about what sniffing allows for.

Also, given the huge number of image/ and video/ MIME types listed at https://www.iana.org/assignments/media-types/media-types.xhtml it seems somewhat irresponsible to just grant cross-origin access to all of those? That would allow for stealing much more local network data than warranted.

[anforowicz]

@anforowicz I don't think that approach really helps with this problem as the image fetching pipeline ignores MIME types (except for image/svg+xml). This is much more about what sniffing allows for.

I don't understand the part above, so I am not sure if the 2 items below apply or make sense:

• In Chrome, ORB (and CORB, CORP, CORS) is implemented in the NetworkService - in a separate process from Blink where image processing pipeline drives fetching and rendering of images.
• FWIW, ORB seems to have access to MIME type information as multiple steps are based on the MIME type.

Also, given the huge number of image/ and video/ MIME types listed at https://www.iana.org/assignments/media-types/media-types.xhtml it seems somewhat irresponsible to just grant cross-origin access to all of those? That would allow for stealing much more local network data than warranted.

Fair point. I can see how allowing fetches of something like image/vnd.adobe.photoshop is unnecessary (because most of image/ formats are not* supported by web browsers) and undesirable from security perspective.

If we wanted to be more selective and only allow specific/individual future image formats then we could either:

• require adding new/future sniffing patterns to https://mimesniff.spec.whatwg.org/#image-type-pattern-matching-algorithm and updating ORB whenever new multimedia formats are introduced
• allowlist new/future MIME types one-by-one (this will mean that future image types cannot be served as application/octet-stream; maybe this is reasonable (?))

Let me also summarize options that AFAIU have been put on a backburner / temporarily rejected:

• Requiring CORS for future image types as proposed in https://github.com/annevk/orb/issues/3#issuecomment-779630945 above (the discussion in this issue understandably shows opposition from authors of new image formats)
• Allowlisting all audio, image, and video MIME types (as pointed out in https://github.com/annevk/orb/issues/3#issuecomment-971380950 this is suboptimal from security perspective)

[annevk]

In order of preference:

• CORS, while this has opposition it is the correct solution. And for other formats this is what we require, e.g., module scripts. Some new image formats flew under the radar, but hopefully that does not have to continue indefinitely.
• Do something with Cross-Origin-Resource-Policy: cross-origin as suggested in https://github.com/annevk/orb/issues/30#issuecomment-971373842. I quite like this.
• Safelist a new MIME type. This is a somewhat sucky solution as it makes more bytes readable than needed, but so be it I suppose. (Requiring a new MIME type seems good though, but ideally it's coupled with CORS or CORP.)
• Extend the sniffing algorithm. This is somewhat on par with a new MIME type, but worse as sniffing has been a source of problems. (As CORB/ORB demonstrates.)

[yoavweiss]

• CORS, while this has opposition it is the correct solution. And for other formats this is what we require, e.g., module scripts

Module scripts are significantly different than traditional scripts, and have a different call site, which makes it easier to enforce CORS on them.

Just to be sure we're talking about the same thing: are you suggesting that <img src=new_format> would fail to decode if added without a crossorigin attribute, and hence loaded in no-cors mode?

[annevk]

Yeah it would network error (due to ORB).

cc @domenic

[domenic]

+1 to requiring CORS for new image types; it's a baseline requirement for security these days and we should never introduce new formats without it.

[saschanaz]

Out of curiosity, is AVIF included in the "new image types" discussed here?

[annevk]

@saschanaz it's probably too late at this point to include AVIF. https://github.com/AOMediaCodec/av1-avif/issues/149 tracks that.

[yoavweiss]

Requiring CORS for new image types would have the following consequences:

• Developer confusion - Developers would not understand why they need to add crossorigin attributes to <img> tags when they're loading a new image format. (or when adding a new image format to their <picture> <source>s).
• Developer friction - adding support for a new image format to the existing image compression pipelines would require HTML changes all over the app, which would mean that deployment of new formats would become extremely expensive. Developers would need some sort of a reporting mechanism to have confidence that they can even deploy new image formats without breaking large parts of the experience.
• Automatic deployment prohibition - image CDNs would simply not be able to adopt new formats. (as they have zero control/influence on the HTML used to load the images)

Such a move would be extremely developer-hostile, would halt adoption of any new image formats, and (IMO at least) will give us very little in terms of user security. What are we hoping to achieve here on that latter front? Is there any reason to believe that users will be safer if we do this?

[domenic]

We would be protecting those images from Spectre attacks.

[yoavweiss]

The only scenario in which we would be doing that is the one where developers actually adopt new formats (despite the hurdles pointed out in my comment above), and they do so using same-origin CORS mode, sending those image requests without credentials, unlike today.

At the same time, it seems like cross-origin credentials won't stick around forever, and it's likely they will go away before a new image format would be supported by browsers (and certainly before such a format is adopted en-masse to enable Spectre protections).

As an aside, if we want developers to switch over to load their images with CORS, we should probably provide them the tools to do that:

• Enable any kind of mechanism to load CSS images with CORS.
• Enable a Document Policy that forces CORS loading for images and other resources (with reporting and maybe even reporting-only mode that enables developers to actually test that their sites work with CORS enabled), avoiding to need to change markup for such a switch.
• Provide them carrots that actually align with the user protections they add (e.g. COI-protected features)

[csreis]

I tend to agree with @yoavweiss that it seems difficult to disallow cross-origin requests for new image formats without CORS, mainly making it hard to deploy these new formats in practice. If the goal is to move image and other media requests to require CORS or credentialless behavior to protect against Spectre, that seems worth pursuing independent of formats, and not just for new ones. CC'ing @mikewest in case there are already considerations around this.

For ORB and new image or media formats, we seem to have a spectrum of options. Requiring CORS simplifies the spec and provides the best security, but is hardest for developers. On the other side, adding a sniffer for every new format is easiest for developers (who wouldn't have to care about MIME types), but is a big ongoing headache for the spec and a constant source of risk if sniffing is tricky (e.g., range requests for videos).

Safelisting each new MIME type and requiring the MIME type to be accurate might be a pragmatic middle ground? That (1) avoids allowing all image/* cases (like the Photoshop example), (2) requires ongoing but simpler updates to the spec to list allowed web media types, and (3) moves us towards a desired future of stricter MIME types without as much developer friction as requiring CORS.

[baumanj]

@saschanaz it's probably too late at this point to include AVIF. AOMediaCodec/av1-avif#149 tracks that.

Submitting a PR to add AVIF to the mimesniff algorithm is near the top of my to-do list

[annevk]

It's not clear to me why the transition for media formats is harder than for JS (note that JS modules require CORS as well as a MIME type). Also, having new formats be safe-by-default when it comes to Spectre seems like a plus as well, especially longer term. It really seems rather undesirable to me to continue to extend the same-origin policy with a set of byte patterns or MIME types essentially indefinitely.

We have to allow AVIF at this point, but I don't think we need to allow any new formats.

[yoavweiss]

It's not clear to me why the transition for media formats is harder than for JS (note that JS modules require CORS as well as a MIME type)

JS modules require a markup change to be loaded. That markup change also enables to modify their CORS mode. Therefore, when web developers are adopting JS modules, they do that explicitly. There are no automatic services that convert classic JS to JS modules.

That is not the situation with images. There are dozens of automatic image conversion services, that take a pristine image as input and output the ideal image format (as well as other factors, e.g. dimensions) without any markup change, and sometimes without the developer even being aware of it.

Any restriction on new image formats that also required markup changes for them would do little more than inhibiting their adoption.

[annevk]

Right, it seems beneficial for the future security of the web to absorb that transition cost.

[eeeps]

@annevk said:

In order of preference:

• CORS [...snip]
• Do something with Cross-Origin-Resource-Policy: cross-origin as suggested in https://github.com/annevk/orb/issues/30#issuecomment-971373842. I quite like this.

Let me explain Cloudinary's situation, and see if I'm correctly understanding how this offers a potential solution.

• We're a media host that provides optimized delivery, and often upgrade the resources behind URLs without requiring our customers to touch their HTML or CSS. For instance as WebP support landed in Safari and AVIF support shipped in Chrome over the last couple of years, we automatically started delivering many existing URLs in those formats.
• We usually have no control over our customers' documents. We only deliver the subresources.
• All of our resources are "public", and do not vary based on authentication or identity. As such we deliver them with Access-Control-Allow-Origin: * and are gearing up to add Cross-Origin-Resource-Policy: cross-origin to them as well.
• I'm trying to pull some data, but I'd guess if we looked at the number of requests coming in with sec-fetch-mode: no-cors vs sec-fetch-mode: cors, cors would be insignificant.
• Asking existing customers to change their deployed markup is a huge amount of friction. Some will; many won't. IIUC, there's no way to request images within CSS via CORS, at the moment.
• We of course don't want to be limited to serving <=AVIF, forevermore, to customers who can't or won't update their markup (or are embedding via CSS).
• We do, of course, control the HTTP headers on responses.

If I understand Cross-Origin-Resource-Policy: cross-origin correctly, it's a way for servers to say: "it doesn't matter if this is embedded (and therefore potentially read via Spectre) across origins". This also seems like a reasonable way to signal that the resource shouldn't be subject to ORB restrictions.

[annevk]

@anforowicz @otherdaniel thoughts on using this header to bypass the algorithm altogether?

[anforowicz]

Using Cross-Origin-Resource-Policy: cross-origin as a signal to opt-out of CORB/ORB SGTM. I wonder if Access-Control-Allow-Origin: * should also be treated as such signal (even though CORB/ORB only applies to no-cors requests where such a response header would feel a bit out of place).

[annevk]

I think Access-Control-Allow-Origin: * is best excluded because:

1. It indeed has no defined semantics for "no-cors".
2. And more importantly, even when mode is "cors", it only works when credentials is "omit", which is not the case here.

[anforowicz]

Some further thoughts on using Cross-Origin-Resource-Policy: cross-origin as a signal for CORB/ORB:

• We can either require 1) using Cross-Origin-Resource-Policy: cross-origin as an *opt out for ORB (when using MIME type unrecognized by ORB), or 2) Cross-Origin-Resource-Policy: same-origin as an *opt in for ORB (when wanting to protect images that are recognized by ORB - say image/png).

  • Extra work is required either from 1) web sites adopting future image formats (opt-out approach) or 2) web sites that want Spectre protection (opt-in approach).
  • Security protection is available either to 1) all future image formats (opt-out approach) or 2) all image formats (but only if the websites takes the extra step to ask for protection - opt-in approach).

@otherdaniel pointed out to me the backcompatibility impact of going with the opt-out approach - this is something that I didn't have fully in focus when replying earlier. For backcompatibility (and future direction of web images ecosystem) I defer to @yoavweiss (e.g. see https://github.com/annevk/orb/issues/3#issuecomment-973818278) and @csreis (e.g. see https://github.com/annevk/orb/issues/3#issuecomment-974334651). AFAIU the main question here is whether the opt-out approach can be gradually adopted for existing image formats.

[yoavweiss]

I'm happy to hear that the image-CDN use case is one that we would be able to solve with CORP. I'm still concerned about developer confusion when it comes to this weird format-based barrier.

The problematic cases I see are:

• Developers have no access to the image server config
• The images in question are actually not public, and servers should not apply CORP to them

For the former case, if we want to incentivize developers to do the right thing, they need to be able to actually do that. So before going down such a route, I think we need to give developers a "load all images with CORS" document policy, and provide them with markup access to set such a policy.

That would enable them to load those images as same-origin CORS mode for public images where they can't change the server's config, and to credentialed cross-origin CORS mode when the images are not public, but safe to include in a specific origin.

Then we may be able to contemplate keeping new formats to only be supported in sites that enable that policy, or have CORP enabled for their images. (Although I still don't see what the user cost is to e.g. adding "image/jxl" to the MIME safe list)

[annevk]

I think we need to give developers a "load all images with CORS" document policy, and provide them with markup access to set such a policy.

How does that help with not having access to the image server config?

(Although I still don't see what the user cost is to e.g. adding "image/jxl" to the MIME safe list)

The cost is that more private information can potentially leak.

[yoavweiss]

How does that help with not having access to the image server config?

If developers can't change their images to emit CORP headers, they'd be able to opt their images into CORS (assuming the servers do support that).

The cost is that more private information can potentially leak.

I'm not still not convinced that jxl images are more likely to contain private info than any of the existing image formats, where we don't require CORS.

[annevk]

It's also a problem for existing image formats, yes. We just don't want to make the problem worse. See the principle of leaving things better than you found them.

[yoavweiss]

leaving things better than you found them

I'd argue that if we really want to do that, we need to provide mechanisms that'd help achieve that at scale for all image formats, rather than draw arbitrary lines, which are unrelated to the threat model nor to the reasons developers haven't been making things better up until now.