search

openstreetmap / operations

OSMF Operations Working Group issue tracking
https://operations.osmfoundation.org/
98 stars 13 forks source link

HTTPS by default on openstreetmap.org #117

Closed grischard closed 3 years ago

grischard commented 7 years ago

This ticket is intended to be a central place to discuss the possibility of switching openstreetmap.org to be https by default.

This is a subject many people feel passionate about, one way or the other, and not a switch that can simply be flicked on and off. Everyone's concerns, issues on the way, hopes, desires, possible solutions, etc. should be discussed or linked here.

Wikipedia did the switch in 2015, The Guardian in 2016.

This isn't about making all content available over https - that's AFAIK already the case, and users of the HTTPS Everywhere will always use HTTPS when loading the website.

The questions that we should address are:

What advantages would a switch to HTTPS by default bring?

A few so far:

What issues would a switch to HTTPS by default cause?

Should we redirect any traffic to HTTPS, or only strongly encourage it?

Forcing and strongly encouraging are different things.

Sending HSTS headers, converting all links to https://, or both, would move most traffic to HTTPS while still replying to legacy HTTP requests. Sending 301 redirects would make sure every client is using https, but make it impossible to use http.

What issues would HSTS cause?

HSTS breaks any site that accesses the OAuth API with http URLs. @mojodna has offered to patch our OAuth library to mitigate the issue.

HSTS could also be selectively enabled, for example only for the http/2 enabled tile servers, which could make the loading of maps faster. Note that clients that are http/2 capable automatically upgrade to https when they connect to a http/2 capable server.

What would technically need to be done to make the switch?

Many pieces of our infrastructure would probably need to be changed or configured differently. How much work would this be for the OWG? Andy is saying it's doable.

Considering the pluses and minuses, is it worth it?

Many of us, myself included, have (strong) opinions on this. I would like us to reach a decision based on measurable facts, and hypothetically review that decision if those facts change over time.

This list of questions is not meant to be exhaustive. I'll try to update the ticket to maintain visibility as comments come in.

grischard commented 7 years ago

CC @tyll @gravitystorm @bhousel @rorym @tomhughes @genodeftest because you've all shown interest in this before.

tomhughes commented 7 years ago

Ignore IE6 and Java 6 as they are already unsupported - you can't login using them to start with as that already requires https.

pnorman commented 7 years ago

Easing the load on the servers if we can serve all content over HTTP/2?

Probably not significant. Tiles are over HTTP/2 when possible, and that's where the many is particularly useful.

What issues would HSTS cause

I consider HSTS an implementation detail to the question of requiring all osm.org traffic to use HTTPS. How to implement is best discussed after the policy decision is made.

A couple of years ago I'd have disagreed, but HTTP/2 and overall HTTPS support have reduced the costs to requiring HTTPS, and Chrome has made a big change:

image

This means the "Show my Location" button doesn't work in Chrome over HTTP. To be clear, I think what Chrome is doing with blocking the location API is wrong, not addressing a real thread, and doesn't follow their own proposal because it's not a new feature. But we're stuck with it. The fix for this is to redirect any page on openstreetmap.org that shows a map to HTTPS. This is most pages.

I think the advantage of getting geolocation working in Chrome outweighs any current disadvantages of requiring HTTPS.

gravitystorm commented 7 years ago

This means the "Show my Location" button doesn't work in Chrome over HTTP

This is also the case for Safari on iOS, and is a bit annoying and I doubt many people can figure it out.

Given that you can't log into the site without HTTPS, any use-case that involves mis-configured proxies etc aren't a high priority. However, I'm interested to hear more about slow connections and how it impacts, and to see if this is still a problem.

How much work would this be for the OWG?

It's not trivial, but I don't think it's excessively large. If we decide to make the policy for https-everywhere, then we can work through whatever needs doing at a steady rate.

grischard commented 7 years ago

Ignore IE6 and Java 6 as they are already unsupported - you can't login using them to start with as that already requires https.

You're right, no one edits with them anymore, but are they also insignificant for the tile server user agent statistics?

@simonpoole also expressed concern about Android users on version 2.2 or less. I don't know if you can get that from the user agents.

grischard commented 7 years ago

https://wikitech.wikimedia.org/wiki/HTTPS and https://wikitech.wikimedia.org/wiki/HTTPS/Future_work and https://phabricator.wikimedia.org/T104681 and https://phabricator.wikimedia.org/tag/https/ might be interesting references.

systemed commented 7 years ago

@systemed is concerned that HTTPS connections are less reliable over bad links

I'll opt out of this discussion, I'm afraid. I have consistently been in areas of poor mobile coverage where https:// sites wouldn't load but http:// would (in particular, the boatyard in Burton-on-Trent where I moored for several years).

However, the arguments used for https are sufficiently emotive that I have no great interest in wasting hours on this that could be more productively spent elsewhere. As long as there are other mapping sites still available over http:// which I can use in areas of poor reception, I no longer have the energy to argue as to what OSM might do. But thank you for the mention.

zerebubuth commented 7 years ago

You're right, no one edits with them anymore, but are they also insignificant for the tile server user agent statistics?

IE6 (or user-agents claiming to be it) accounted for 0.16% of tile hits yesterday.

@simonpoole also expressed concern about Android users on version 2.2 or less. I don't know if you can get that from the user agents.

As far as I can tell Android <= 2.2 accounts for about 0.01% of site visitors and 0.005% of tile views according to user-agent headers yesterday. It is likely that there are apps running on Android versions <= 2.2 which wouldn't show up in this analysis, as they don't report the OS version, but I'd be surprised if it were a large proportion.

On a different subject, the title of this ticket is "HTTPS by default", but much of the discussion has been around HSTS, "requiring all osm.org traffic to use HTTPS" or "converting all links to https://". To me, these are very different things. "By default" doesn't shut the door to people who make the choice to use HTTP, whereas HSTS or converting all links to https:// would.

As pointed out, it is already the situation that logging in or other secure actions require HTTPS. It is already the situation that anyone choosing to use HTTPS Everywhere uses the site in a fully secure manner.

The only real problem appears to be the fairly minor issue that "show my location" doesn't work for some recent browsers. Would it be possible to "un-break" Chrome and Safari by detecting the failure condition and reloading the page over HTTPS?

grischard commented 7 years ago

You're right @zerebubuth, these are different things, and identifying the possible tools we have is essential to decide which ones to use.

FWIW, http://caniuse.com/#search=hsts says that switching on HSTS would affect 90% of the users.

grischard commented 7 years ago

It would still be very interesting to know under what circumstances https://openstreetmap.org is broken while http:// isn't.

Something like https://github.com/jonocole/disable-hsts-chrome-extension could be useful to @systemed and other hypothetical affected users.

pnorman commented 7 years ago

On a different subject, the title of this ticket is "HTTPS by default", but much of the discussion has been around HSTS, "requiring all osm.org traffic to use HTTPS" or "converting all links to https://". To me, these are very different things. "By default" doesn't shut the door to people who make the choice to use HTTP, whereas HSTS or converting all links to https:// would.

Using HTTPS by default to me means that when someone types in openstreetmap.org they end up on the HTTPS site. The only ways to accomplish that I know of are direct all traffic requesting the front page to HTTPS. That could be by means of HSTS, 301 redirects, or similar. This will effectively make it impossible or very difficult to browse openstreetmap.org with HTTP.

The only real problem appears to be the fairly minor issue that "show my location" doesn't work for some recent browsers. Would it be possible to "un-break" Chrome and Safari by detecting the failure condition and reloading the page over HTTPS?

About 75% of browsers require HTTPS for geolocation, and I suspect we'd have to have them redirected to HTTPS on page load, or press the geolocate button twice.

tyll commented 7 years ago

On Sun, Nov 06, 2016 at 03:17:25PM -0800, Matt Amos wrote:

On a different subject, the title of this ticket is "HTTPS by default", but much of the discussion has been around HSTS, "requiring all osm.org traffic to use HTTPS" or "converting all links to https://". To me, these are very different things. "By default" doesn't shut the door to people who make the choice to use HTTP, whereas HSTS or converting all links to https:// would.

HSTS does not mean that it is impossible to use the site with HTTP. It only means that once someone accessed the site via HTTPS they will continue to use HTTPS for a certain amount of time. You can even use this to upgrade anyone to HTTPS if their browser supports HTTPS for example by including a hidden image via HTTPS and serving a HSTS header via HTTPS.

Also even if all links from the outside (e.g. in mails) are https, people who prefer http can still use it. However, the future is that everything will be https and that will not be able to use the big sites on the Internet without https.

As pointed out, it is already the situation that logging in or other secure actions require HTTPS. It is already the situation that anyone choosing to use HTTPS Everywhere uses the site in a fully secure manner.

IMHO it should be a project's resposibility to make sure that their users can and usually access a site in a secure manner. This protects everyone who does not have the expertise to know whether or not what they are doing is secure.

zerebubuth commented 7 years ago

About 75% of browsers require HTTPS for geolocation, and I suspect we'd have to have them redirected to HTTPS on page load, or press the geolocate button twice.

I was thinking that, if the failure can't be detected until the button is pushed, then we'd have to pop up some sort of div with a message saying "This feature is only available over HTTPS, please click this link to reload."

Alternatively, if it's possible to detect before the button is pressed, then the button could be greyed out with a tooltip explaining why it's greyed out, possibly also with a link to reload over HTTPS.

Using HTTPS by default ... will effectively make it impossible or very difficult to browse openstreetmap.org with HTTP.

I think so too. While it would technically be possible to copy each link by hand into the address bar and edit the scheme to "http"... that would be so much work that no one would do it.

In summary so far:

Pros:

Cons:

pnorman commented 7 years ago

we'd have to pop up some sort of div with a message

I'd rather redirect, I think it's a much better user experience. A div popup turns a one-click operation into a three-click operation.

In summary so far: ...

What about the API calls? These have different usage patterns and user-agents, and they get into OAuth signing. Or are we only looking at the HTML pages right now?

genodeftest commented 7 years ago

The only real problem appears to be the fairly minor issue that "show my location" doesn't work for some recent browsers. Would it be possible to "un-break" Chrome and Safari by detecting the failure condition and reloading the page over HTTPS?

http://caniuse.com/#search=geolocation says: HTTPS is required for Chrome, Safari, Opera, iOS Safari, Android Browser, Chrome for Android. Only major browsers without this requirement are IE, Edge and Firefox.

grinapo commented 7 years ago

Just a sidenote: a site got burned by lack of entropy in the entropy pool when using it up for all kinds of crypto at a changeover. If one's using blocking random access (rejecting the use of "bad" quality randoms when there isn't enough entropy available) it simply blocks for extended periods of time. Non-blocking randoms with no entropy mean deteriorated security (without the blocking, obviously :-)). Usually a non-issue for "normal" configs, but just in case keep an eye on the gauge.

tyll commented 7 years ago

On Mon, Nov 07, 2016 at 10:28:30AM -0800, grinapo wrote:

If one's using blocking random access (rejecting the use of "bad" quality randoms when there isn't enough entropy available) it simply blocks for extended periods of time. Non-blocking randoms with no entropy mean deteriorated security (without the blocking, obviously :-)).

Do you have any proof that /dev/urandom (aka the non-blocking random pool) is less secure than /dev/random? This talk explains why /dev/urandom is the right random source: https://www.youtube.com/watch?v=Q8JAlZ-HJQI&t=1337

Firefishy commented 7 years ago

We enabled haveged across all services a few years ago to avoid entropy pool starvation.

grischard commented 7 years ago

Although we could mitigate this with the secure attribute on cookies

If all map pages aren't redirected to https, this would be confusing. "Why can't I edit? I've just logged in!"

zerebubuth commented 7 years ago

If all map pages aren't redirected to https, this would be confusing. "Why can't I edit? I've just logged in!"

Logging in switches the scheme to HTTPS, so this wouldn't be a natural navigation flow for the user. Clicking the edit button could also redirect to HTTPS, as one needs to be logged in for that anyway.

grischard commented 7 years ago

HSTS does not mean that it is impossible to use the site with HTTP. It only means that once someone accessed the site via HTTPS they will continue to use HTTPS for a certain amount of time.

Section 7.2. of the HSTS RFC says: "An HSTS Host MUST NOT include the STS header field in HTTP responses conveyed over non-secure transport."

A while ago, I asked Jeff Hodges, one of the RFC's author, why that was. If you provide both HTTP and HTTPS for compatibility reasons, HSTS on HTTP responses would seem to be a neat way of encouraging HTTPS without forcing it.

Jeff replied:

The rationale for not including the STS header field in unsecured HTTP responses is that it (the sts header field) is a security directive and http clients must not heed such directives obtained over unsecured channels. So it is best if webapps do not issue the STS header over unsecured channels. We made it a "MUST NOT" to try to get folks' attention. it is much more critical for the clients to get this right, as stipulated in section 8.1.

grischard commented 7 years ago

Hitting the 'Edit' button to load the map in JOSM from an https page can apparently sometimes fail to work. (Can anyone reproduce this?)

I remember running into that and fixing it, although I don't remember how. Osmose had the same problem and uses a redirect hack.

SomeoneElseOSM commented 7 years ago

JOSM HTTPS remote control support depends on Java trusting a self-signed certificate for "localhost" doesn't it? Presumably there are going to be a bunch of people out there who don't have write access to the Java keystore (or who just are going to be put off by the scary message that you get when enabling remote control HTTPS support).

Obviously JOSM remote control isn't the only way to get to start editing in an area (FWIW I tend to start via "download object" or locating somewhere on the imagery).

tyll commented 7 years ago

On Tue, Nov 08, 2016 at 08:51:25AM -0800, Guillaume Rischard wrote:

A while ago, I asked Jeff Hodges, one of the RFC's author, why that was. If you provide both HTTP and HTTPS for compatibility reasons, HSTS on HTTP responses would seem to be a neat way of encouraging HTTPS without forcing it.

There is a different technique to notify that HTTPS is preferred:

http://stackoverflow.com/questions/31950470/what-is-the-upgrade-insecure-requests-http-header/32003517#32003517

Also including an (hidden) image via HTTPS and sending an HSTS header there can be used to upgrade clients to HTTPS if they support HTTPS and HSTS.

pnorman commented 7 years ago

There is a different technique to notify that HTTPS is preferred: ...

I think the techniques should wait until after the policy discussion.

Komzpa commented 7 years ago

HTTP 2.0 for tiles can move them away from [abcd].tile.osm.org scheme, keeping less connections, allowing browser to request all the tiles at once and get them in single round-trip. From my experience with Wargaming GlobalMap, this is especially good thing for slow / laggy connections. This requires HTTPS.

Ensuring HTTPS via HSTS for tiles will likely make sure browsers will send Referer header to the server (they skip it when loading http image into https page), which allows more precise traffic source tracking.

Klumbumbus commented 7 years ago

Hitting the 'Edit' button to load the map in JOSM from an https page can apparently sometimes fail to work. (Can anyone reproduce this?)

In Firefox remote control to JOSM via https is not working at all. Only http works. More info at

SomeoneElseOSM commented 7 years ago

Not something that affects the main map site, but I was trying to translate https://lists.openstreetmap.org/pipermail/talk-es/2016-November/014568.html from Spanish to English for the OSM weekly news. Google Translate screws up the formatting somewhat, so I thought I'd try http://www.bing.com/translator - and that says "Translation of secure pages is not supported". This is a problem we have now rather than a theoretical future one as OSM's pipermail site redirects from http to https already.

scaidermern commented 7 years ago

In Firefox remote control to JOSM via https is not working at all. Only http works.

Works fine here. I always use HTTPS with Firefox on openstreetmap.org (thanks to HTTPS Everywhere). To get JOSM's remote control to work via HTTPS you have to open https://127.0.0.1:8112 in your browser and permanently store an exception for this self-signed certificate. Afterwards the remote control should work.

Klumbumbus commented 7 years ago

OK, thanks. Works for me.

grischard commented 7 years ago

Relevant for discussion: The Guardian has moved to HTTPS today.

simonpoole commented 7 years ago

@grischard the guardian just happens to be the newspaper site that requires you to leak what you are reading to something roughly between 15 and 30 third party sites, always including at least facebook but most of the other usual suspects too. It is hard to imagine a site that has -less- relevant privacy policies than them.

grischard commented 7 years ago

Extensive write-up on moving to HTTPS only from the Stack Overflow operations team.

grischard commented 7 years ago

So where are we six months later?

More and more sites are enabling HTTPS by default, so HTTPS should be blocked in less places, and be more reliable. There will also be even less legacy browsers.

JOSM still doesn't reliably add its certificate to the certificate stores on all browsers and all platforms. The https remote isn't enabled by default in JOSM. These three tickets must be resolved before we can switch:

  1. Enable https support on remote by default
  2. Mention that self-signed certificate might have to be accepted for remote control to work
  3. Josm do not start in remote control from osm.org in https , which is about adding JOSM's certificate to the keystore/keychain/NSS.
grischard commented 7 years ago

Simon points out https://github.com/MarcusWolschon/osmeditor4android/issues/523 - Android versions up to at least 6.0 have issues with SNI. I don't know whether using a certificate with the right CN for the OAuth endpoint would be a good workaround.

Bisaloo commented 7 years ago

Just a heads up: from now on (Firefox 55.0), Firefox requires HTTPS for the geolocation API as well.

Firefox 55.0 release notes

Can I use... Geolocation

systemed commented 7 years ago

On the Remote Control side, it also looks possible that desktop Potlatch will support Remote Control over HTTP but not HTTPS, for Adobe AIR reasons. I guess there are potential ways around this, such as an openstreetmap-website user setting to "Launch Remote Control over unencrypted local connection" which would use a popup or redirect rather than the current appended iframe.

rugk commented 7 years ago

Does my site need HTTPS?

Still think you don't need HTTPS?

Google: Why you should always use HTTPS

Is TLS fast yet?

Breaking OAuth for HTTP is also a good thing, as any authentication is worth nothing when it is done using insecure protocols, such as HTTP.

Bisaloo commented 7 years ago

Blocking: https://github.com/hotosm/osm-tasking-manager2/issues/988

pnorman commented 7 years ago

Blocking: hotosm/osm-tasking-manager2#988

I don't see that blocks the OSMF servers from forcing HTTPS.

Bisaloo commented 7 years ago

Maybe blocking is a strong word but https on openstreetmap.org breaks stuff from hotosm (and similar sites) due to MCB.

Forcing https will make this problem worse. Unless task.hotosm.org and others get https support first.

See https://github.com/EFForg/https-everywhere/issues/13211#issuecomment-337623749 for steps for reproduce this issue. It will probably be clearer.

grischard commented 7 years ago

So the issue in general is that content that the iD editor embeds from other sites will have to be served over https. Solving hotosm/osm-tasking-manager2#988 and sending https gpx links is the way to solve that.

Https usage on openstreetmap.org isn't uncommon, and showing the gpx boundary from hot is already broken for all those users. I'm actually surprised that tasks.hotosm.org isn't available over https at all.

I've added this issue to the list.

SomeoneElseOSM commented 7 years ago

@grischard Apologies if I'm missing something here, but how is the fact that tasks.hotosm.org doesn't support https at all relevant to whether it's a good idea to force users of openstreetmap.org to use https whether they want to or not?

grischard commented 7 years ago

@SomeoneElseOSM openstreetmap.org doesn't exist in a vacuum; it'd be relevant if forcing https broke a lot of things for the mappers, just like hsts breaking oauth is. The fact that it took almost one year for this to come up makes me think that it won't, and the issue can easily be mitigated on hot's side anyway, but I want to be thorough in looking at the possible issues.

rugk commented 7 years ago

Maybe depreciate/announce that you are removing HTTP support and let sites some time to switch to HTTPS. Just note that OAUTH and any password/authentication over insecure channels (HTTP) is not secure. Everyone can sniff your password, so… In any case the aim should at some point be HTTPS-only. Before that you may redirect to HTTPS e.g. only on the main site or on most sites (e.g. login page) with some exceptions.

HolgerJeromin commented 7 years ago

Just note that OAUTH and any password/authentication over insecure channels (HTTP) is not secure. Everyone can sniff your password, so…

With oauth no passwort is transmitted

rugk commented 7 years ago

But session tokens, or some other identifier. Any MITM possibility is bad. And you still submit passwords over http, so we don't even need to argue about oauth.

simonpoole commented 7 years ago

@rugk please get your facts straight, if you are using OAuth as essentially all major editors do, you do NOT authorize (aka login once with login/password) over http, but over https.

bhousel commented 7 years ago

@simonpoole - @rugk is correct that OAuth secrets can be transmitted over http when embedding a browser editor like Potlatch or iD on the openstreetmap website, if the website was first loaded via http. There is no really secure way to do communication between the embedding site and the editor iframe.

simonpoole commented 7 years ago

@bhousel nobody was claiming anything else, the authorisation process however takes place over https and you don't send login/password over http for external apps, just as I wrote (iD is a very special case and not the norm).