Add CORS guidance to specification

[BigBlueHat]

CORS is in the way of this header's success. Here's a for instance...

Visit https://example.com/ and open the browser console and paste...

fetch('http://www.w3.org/ns/anno.jsonld', {
  headers: {
    'Accept': 'application/ld+json',
    'Upgrade-Insecure-Requests': 1
  }
})
.then(console.log.bind(console))
.catch(console.error.bind(console));

Console presents this error (in Chrome 56, in this case):

Mixed Content: The page at 'https://www.example.com/' was loaded over HTTPS, but requested an insecure resource 'http://www.w3.org/ns/anno.jsonld'. This request has been blocked; the content must be served over HTTPS.

and

TypeError: Failed to fetch

Network tab shows this request (gist has the HAR file of the above experience): https://gist.github.com/BigBlueHat/24fec2ceffb72aa8b13bb194ca1b5917

Given that scenario, the Upgrade-Insecure-Requests header (though correctly handled by that server--which sends a 307 to the secure variation if that header's included) never gets it's chance to be helpful because CORS is in the way.

It would be great if the spec included more guidance for the use of the headers in relation to CORS as well as CSP.

(aside: if I've misunderstood something, I'm more than happy to hear what I should change to make this work. thanks!)

[annevk]

The reason that fails is because you are fetching HTTP from an HTTPS page. Doesn't have anything to do with CORS or this specification. Also, Upgrade-Insecure-Requests is a header the browser sets, not the developer, although I guess you're not prevented from including it.

[BigBlueHat]

So, the Upgrade-Insecure-Requests header can't be used by developers in this way? My hope was precisely to fix the issue of "fetching HTTP from an HTTPS page" as I'm dealing with older content referenced with http:// prefixed names that may (and often are these days) available at https:// locations.

This header seems to handle exactly this scenario...but apparently its not for me?

[annevk]

If https://example.com/ is served with a Content-Security-Policy: upgrade-insecure-requests header you get what you want.

[mikewest]

Upgrade-Insecure-Requests: 1 is a request header that advertises support for the upgrade feature, as described in https://w3c.github.io/webappsec-upgrade-insecure-requests/#feature-detect. It doesn't in itself have any effect on the request, but instead enables the server to make a decision about whether or not to turn on the upgrade feature by sending a Content-Security-Policy: upgrade-insecure-requests response header, which applies to an entire document.

I hope the examples in https://w3c.github.io/webappsec-upgrade-insecure-requests/#example-nonnavigational help explain things.

[BigBlueHat]

@annevk so, I think that's where things got muddied. I didn't realize that the page in which I was running the request had to be itself served with Content-Security-Policy: upgrade-insecure-requests.

In which case, that means that Upgrade-Insecure-Requests: 1 sent with an XHR request is meaningless unless the page from which that request is made is also served with Content-Security-Policy: upgrade-insecure-requests.

Is that correct?

For instance, the upgrade-insecure-requests is absent from github.io sites--which serve this CSP line:

Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; img-src data:; connect-src 'self'

Which means Upgrade-Insecure-Requests: 1 will fail from those if I'm not mistaken.

That sound correct?

[annevk]

Yeah, again, Upgrade-Insecure-Requests is just the browser indicating support, it doesn't mean anything else and doesn't have any side effects either.

[BigBlueHat]

@annevk right. I realize all that. What isn't clear is what other hoops have to be jumped through for that header to ever reach the server and/or have "meaning" within the context of the existing page.

My expectation would be that Upgrade-Insecure-Requests: 1 could be sent via an XHR/Fetch request anytime anywhere and the server could Do The Right thing in response. The reality is that it never even gets to the server unless the page the request is made from allows the upgrade to happen via the Content-Security-Policy: upgrade-insecure-requests.

AFAICT, this means I'd need server-side header change access to enable the successful use of this request header--which I don't always have--despite wanting to upgrade requests to a secure connection and sending this request header to do just that.

This seems to limit the usefulness of this Upgrade-Insecure-Requests: 1 header to just scenarios where one already controls the server. Is that correct?

[annevk]

Upgrade-Insecure-Requests is a signal from the browser that it supports the feature the server can control through Content-Security-Policy. If you can set a header from XHR/Fetch, presumably you can also alter the URL passed to XHR/Fetch to not be insecure.

[BigBlueHat]

Yes. I can alter the URL, but that change is just a blind leap as there's nothing mandating that the new https URL is identical. Typically they are, but it's not a requirement and so shouldn't be depended on.

Also, does this have to remain a browser-only header? The use case presented here seems like a viable scenario for using it.

[davidlehn]

I'll add some background: one reason for this issue is the desire to run http://json-ld.org/playground/ over https or at least have more resources sent over https if running under http. Input data can specify context URLs which the browser will try to fetch. Due to the wide spread use of non-https contexts this often hits mixed content failures if used under https. One popular problematic context in particular is http://schema.org/. The code can't blindly rewrite http to https in general since there is no guarantee those are the same resources. The hope was that the resource fetching code could add some hints that might help the situation. URL rewrite whitelists and proxies are other solutions but more complex.

[annevk]

I'm not entirely sure we're on the same page at this point. Do you understand how this feature works?

[BigBlueHat]

I believe I understand how it does work. I don't believe we're in agreement on how it should work. 😃

Given the proliferation of...

1. http:// prefixed @context values as well as other IRI's and URI's that are also URLs and
2. HTTPS as the preferred "only"/default protocol for browsing/running JavaScript

...there needs to be a way to:

1. utilize exiting names of things (those IRI's and URI's that are also URLs)
2. not run into the "Mixed Content" glass ceiling when the server can tell the browser that there's a variation of this same resource available via HTTPS.

The W3C's header's seem to do just that--when looking at them from curl--but the CSP/CORS/single-origin situations tangle up the issue when used in Firefox, Chrome, etc.

There are http:// prefixed names on the Web that can't be changed (or at least won't be for some time) and they should still be usable to locate and serve representations of resources even if the browser and server need to do a dance to get them over a different protocol--in this case HTTPS.

If this is not the thing to solve for that, then there needs to be a method for dealing with this scenario.

Thoughts?

[annevk]

The W3C's header's seem to do just that--when looking at them from curl--but the CSP/CORS/single-origin situations tangle up the issue when used in Firefox, Chrome, etc.

I have no idea what you mean by this. I also don't understand why names that look like URLs have to be passed unmodified to fetch(). Just change the scheme if you're fetching from an HTTPS document, since you'll get a network error otherwise.

[BigBlueHat]

Meaning, this request looks to be doing The Right Things:

curl -v -X HEAD http://www.w3.org/ns/anno.jsonld
* Hostname was NOT found in DNS cache
*   Trying 128.30.52.100...
* Connected to www.w3.org (128.30.52.100) port 80 (#0)
> HEAD /ns/anno.jsonld HTTP/1.1
> User-Agent: curl/7.35.0
> Host: www.w3.org
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Tue, 21 Feb 2017 18:53:50 GMT
< Last-Modified: Wed, 23 Nov 2016 12:36:00 GMT
< ETag: "1706-541f72199ac00"
< Accept-Ranges: bytes
< Content-Length: 5894
< Cache-Control: max-age=21600
< Expires: Wed, 22 Feb 2017 00:53:50 GMT
< P3P: policyref="http://www.w3.org/2014/08/p3p.xml"
< Vary: upgrade-insecure-requests
< Access-Control-Allow-Origin: *
< Content-Type: application/ld+json

It's not safe to say, "just stick an s in that URL and all will be well" as--though that's typical--it's not required. These are two different protocols, and therefore the strings behind them may not result in the same representation of the same resource.

If I ask for that same URL via curl with the Upgrade-Insecure-Request, the W3C servers Do The Right thing and send me a 307 to the new correct location (which, since the server is driving the response could just have an extra s or could be something more dissimilar...my client shouldn't care because hypermedia):

HEAD http://www.w3.org/ns/anno.jsonld -H 'Upgrade-Insecure-Requests: 1'
* Hostname was NOT found in DNS cache
*   Trying 128.30.52.100...
* Connected to www.w3.org (128.30.52.100) port 80 (#0)
> HEAD /ns/anno.jsonld HTTP/1.1
> User-Agent: curl/7.35.0
> Host: www.w3.org
> Accept: */*
> Upgrade-Insecure-Requests: 1
>
< HTTP/1.1 307 Temporary Redirect
< Cache-Control: no-cache
< Content-length: 0
< Location: https://www.w3.org/ns/anno.jsonld
< Vary: Upgrade-Insecure-Requests
<
* Connection #0 to host www.w3.org left intact

This is what I expected the experience to be like in Firefox and Chrome, but it isn't--which we've explored here as being related not to this header per se but to CSP/CORS things more broadly.

[annevk]

The reason that doesn't work is because you'd open yourself up to malice-in-the-middle attacks. Once you go insecure you're lost, so that's not an option.

[BigBlueHat]

I'm guessing browser support for RFC 2817 (or something like it) isn't in the offing then?

I'm more or less wishing there was a "find me the secure version of ___" request that could be made without the MITM risk. Especially when the user agent knows the particular Host does traffic over a secure channel.

Obviously, this isn't that (nor what I thought it was), however, the situation still exists and isn't going away anytime soon. Rewriting URLs and setting up proxies are bad band-aids. They work, but their just making permanent what's being broken by these fixes... 😢

Is there anything else in the webappsec space being done to address this scenario (http:// prefixed preferred names for things that also have secure-channel delivery available...though quite possible with a different name)?

[annevk]

The main thing we're doing is aggressively deprecating HTTP, through efforts such as Secure Contexts. Any kind of insecure traffic is a problem that needs fixing and therefore indeed, opportunistic "encryption" is not a solution we want.

[BigBlueHat]

Give that HTTP requests from within a Secure Context will fail and need to be re-requested over HTTPS, would it be possible to just attempt that first (when within a Secure Context)? It makes the same equivalency assumption (that http://...something... === https://...something...), but removes the requirement to do a bulk rename of code and (harder to change) data already in the wild.

Thoughts?

(oh, and I greatly appreciate the time here...I'm standing in for a wide group of people facing--or about to face--this same issue)

[annevk]

If you want that, use Content-Security-Policy: upgrade-insecure-requests as a response header for the resource where you want that kind of behavior. I don't think we can just start doing it as that might actually break things (that expect failure). (And we also don't block all fetches just yet, e.g., images can still leak unfortunately and would end up broken if we enforced HTTPS.)

[BigBlueHat]

Gotcha. So if GitHub (among others) were to add Content-Security-Policy: upgrade-insecure-requests to it's github.io hosted sites, then using the Upgrade-Insecure-Requests: 1 (in modern-ish browsers) would Do The Right Thing. I'll start there at least. :smiley: I'll try to hammer through some other stub-server examples to finish getting my head around which request/response headers effect which thing at which point in the process. That, afaict, is where things went :pear: shaped. Thanks again @annevk (and @mikewest).

[annevk]

If GitHub did that, Upgrade-Insecure-Requests: 1 would just be noise, but they can use that as a hint that the browser supports it.

Closing this as this doesn't really relate to the standard.