Support mixed-content opaque stream fetches

[wolenetz]

Mixed Content[1] already indicates that the following are optionally-blockable: video loaded via <video> or <source> audio loaded via <audio> or <source>

MSE extends the loading functionality of both <video> and <audio> to let applications provide the media via buffers and streams. The typical mechanism for appending buffers to <video> or <audio> using MSE involves XHRs, which are "blockable" per the mixed content spec [1]. However, if opaque streams were used to provide the loaded video or audio to <video> or <audio> using MSE's appendStream() API, then MSE would not be outright "blockable" in a mixed content application scenario, extending the viability of using MSE to applications who wish to have their origin secure but are not ready yet for the content to be secure.

[1] https://w3c.github.io/webappsec/specs/mixedcontent/

In particular, Fetch needs to support opaque responses by yielding an opaque stream. This bug tracks discussion and specification of this.

(Updated May 24, 2016 to show <audio> and <video> correctly with markdown.)

[yutakahirano]

Does that mean an opaque response's body property should be an opaque stream?

I think "opaque stream" itself will be defined in the streams spec - Is that correct, @domenic, @tyoshino? In the fetch spec, we should define isUsed / isDisturbed on opaque responses. Is there anything I'm missing?

[annevk]

Does that mean an opaque response's body property should be an opaque stream?

I think that's basically the suggestion, yes. It's not clear to me all browsers are on board with this plan, but that's the proposal from Google. At the moment though Fetch would return null for body on opaque responses.

[yutakahirano]

+@wanderview fyi

[wanderview]

Is there any definition of what an opaque stream is? Its hard to understand what this issue is asking for without at least a sketch of the opaque stream behavior.

Wouldn't an alternative be to allow MSE consume Response objects directly? You could then pass an opaque Response directly to MSE.

[yutakahirano]

@wolenetz, can you answer @wanderview's question?

[yutakahirano]

https://w3c.github.io/webappsec-credential-management/#monkey-patching-fetch-2 adds "opaque" flag to Request, but I think opaque streams can be used for such a case, too. @mikewest, what do you think about it?

[wanderview]

I would still like some definition of what an "opaque stream" is.

[wanderview]

From IRC:

12:19 PM wanderview: btw, opaque stream is basically a stream for all intents and purposes but only privileged code (read: the user agent) gets to see the bytes 12:21 PM wanderview: and thank you in advance for the review 12:24 PM → encryptdfractal joined (~encryptd@156.39.191.244) 12:24 PM annevk: this seems to create a whole new vector of opaque tainting to be implemented... 12:25 PM ⇐ yoav quit (~yoav@sdo26-1-78-245-148-181.fbx.proxad.net) Remote host closed the connection 12:25 PM annevk: pass an opaque stream to a DOM implemented decoder, and the decoder has to produce an opaque stream, right? 12:25 PM etc, etc 12:30 PM wanderview: what do you mean by "a DOM implemented decoder"? 12:31 PM yhirano: like if https://encoding.spec.whatwg.org/ grew support for streams... it would in theory be a transform 12:31 PM yhirano: yet, if it took an opaque stream it would now have to be smart enough to pass on the opaqueness, etc 12:32 PM yhirano: maybe we need this, but it seems to add some non-trivial complexity 12:33 PM yhirano: for example, var r = new Response(url, { body: anOpaqueBody }) gives a non-opaque Response with an opaque body... that has to be propagated through Cache, etc... 12:33 PM wanderview: Body.text() and so on don't recognize opaque streams. I thought it would be similar for other decoders 12:35 PM yhirano: well I guess thats the question... what recognizes these things and what doesn't... its hard to tell from the current spec issue 12:38 PM <yhirano> wanderview: agreed.

[wanderview]

If we allow opaque streams, then this would need to fail:

addEventHandler('fetch', function(evt) {
  if (evt.mode === 'navigate') {
    evt.respondWith(fetch(crossSiteURL, { mode: 'no-cors' }).then(function (response) {
      // Create a synthetic non-opaque response with an opaque body?
      var laundered = new Response(response.body);
      return laundered;
    })
  }
});

And probably lots of other opaque tainting issues to be discovered.

[tyoshino]

wolenetz explained thoughts on appendRespones() vs. appendStream() in this post to public-webappsec: https://lists.w3.org/Archives/Public/public-webappsec/2015Apr/0105.html

[annevk]

If that is really the only use case, appendResponse() would be much simpler.

[tyoshino]

One point I'm not sure yet is, ... it seems to me that opaqueness for CORS and MIX (here) are slightly different. Opaqueness for CORS response tainting must be strict so that no one can uncover the contents which the server didn't allow to provide to cross-origin scripts. Opaqueness for MIX is protecting developers from easily using unreliable (possibly forged) subresource, and so the wall doesn't have to be high as one for CORS, right?

Can we really converge these different opaqueness concept and protect by single opaqueness flag (the requirement for CORS is stronger so we can use it for MIX too)? Won't we want to distinguish them in the future?

If that is really the only use case, appendResponse() would be much simpler.

Yea... I also agree with yhirano's point about possible composition of streams to process bytes without right to read the actual bytes.

We were also discussing how to implement equivalent of "img.src=URL.createObjectURL(blob)" for streams.

We initially sketched it like:

var writableStream = img.srcWithStream();
var writer = writableStream.getWriter();
...

But e.g. to make it able to use no-cors results, someone said maybe it should be img.setResponse(response) instead, IIRC.

[annevk]

I'm not sure what you mean by opaqueness for CORS. A "cors" response has very limited opaqueness (just around headers). If you mean "no-cors", i.e. cross-origin resources without CORS, those are equivalent to MIX, since both are cross-origin. MIX is worse of course, since it's a worse legacy security bug that is hard to get rid of.

(As for <img>, I think we want something similar to what we already have for <video> and such, srcObject, and have that accept responses, e.g., <img>.srcObject = response. It cannot be just a stream since <img> needs to know about the Content-Type header too, for SVG.)

[tyoshino]

I'm not sure what you mean by opaqueness for CORS. A "cors" response has very limited opaqueness (just around headers). If you mean "no-cors", i.e. cross-origin resources without CORS, those are equivalent to MIX,

Yes, no-cors. Sorry for being unclear.

since both are cross-origin. MIX is worse of course, since it's a worse legacy security bug that is hard to get rid of.

Ah! I see. I was unaware of that... Thanks

[tyoshino]

(As for , I think we want something similar to what we already have for

I see. When composing an image from byte stream and type provided separately, maybe we just create a Response just for holding them? Hmm, it sounds natural as I guess most of elements we may want to stream data into via Streams are currently taking a URL (src=) and "fetch"-ing it. OK.

[tyoshino]

BTW, as yhirano mentioned in https://github.com/yutakahirano/fetch-with-streams/issues/56#issuecomment-152893640, there is another opaqueness being developed for Credential Management.

I was comparing the opaqueness for response (discussed in this issue) and that for analyzing possible merge/confusion/etc. in the future. I found that the opaqueness for Credential Management is kinda weak than one for response, currently (in terms of e.g. https://github.com/w3c/webappsec/issues/241#issuecomment-136638613 and https://github.com/w3c/webappsec/issues/241#issuecomment-136639712). This is still subject to change if needed? E.g. making the opaqueness propagated to SW, aligning XHR/Fetch with toFormData() by requiring XHR/Fetch to comare environment's origin and FormData's origin (currently not embedded) before issuing a request.

[annevk]

https://github.com/whatwg/fetch/issues/49 is about adding srcObject to more elements and supporting assigning Response to it.

I think @mikewest has not settled on the exact design of credential management opaqueness and is probably open to changes.

[wolenetz]

Getting back to the MIX MSE need for opacity in some form of API, I'm not quite clear on the relative difficulties to FETCH, STREAMS, and any opaque-tainting details of doing opaqueStream vs opaqueResponse. IIUC, MSE MIX (https://github.com/w3c/media-source/issues/22) needs: 1) secure-origin'ed web app ability to fetch from insecure origin a stream with an optional limit on maximum number of bytes to fetch 2) prevention of that web app from seeing the contents of the response 2a) I'm not clear what other Response object attributes might also need to be hidden from the web app in this MIX fetch case. Headers? 3) ability for the web app to provide the stream (or response) to a SourceBuffer.append{Response or Stream} method. 4) Upon SourceBuffer.append{Response or Stream} invocation, the UA must be able to become the exclusive reader of the fetched media stream (this is really a requirement on the non-opaque STREAM + FETCH that is inherited by the opaque version).

I would like to find a route to enabling MIX MSE, and would appreciate your expert guidance here.

[annevk]

secure-origin'ed web app ability to fetch from insecure origin a stream with an optional limit on maximum number of bytes to fetch

That breaks the same-origin policy. I don't think that's acceptable. You can only have complete responses of which you expose close to nothing to the API user.

[yutakahirano]

Note: the "opaque" body I mentioned at https://github.com/yutakahirano/fetch-with-streams/issues/56#issuecomment-152893640 was removed.

[yutakahirano]

Regarding mixed-content , we can make a fetch for a mixed-content resource by specifying request's type to "video". Is it acceptable to create a request with a user-provided type, with some restrictions such as prohibiting cors mode?

[annevk]

You'd also need to restrict how the response ends up being used. I'm not entirely convinced this is something we should be adding though. It's a fair amount of complexity for something that in a couple of years will all use HTTPS anyway.

[yutakahirano]

You'd also need to restrict how the response ends up being used.

Even now, we can access a Response for a request with type = "video" and mode = "no-cors" in a serviceworker. So I thought we need restrictions only in the request-side. Am I missing something?

[annevk]

If you could do it on a page, you could circumvent CSP restrictions.

[yutakahirano]

I see, thank you.

[annevk]

This never really went anywhere. I doubt it would still be accepted as it's somewhat of a security regression in ways. Close?