Deprecation of HTTP Basic Auth and OAuth 1.0a

grischard commented 1 year ago

This is to track the discussion of the future deprecation of HTTP Basic Auth and OAuth 1.0a.

See pnorman's diary and https://github.com/zerebubuth/openstreetmap-cgimap/issues/286

See the deprecation announcement for dates and details:

This change will have three key dates:

March 1st, 2024: New OAuth 1.0a application registrations are disabled. Existing applications will not be impacted. HTTP Basic Auth will not be impacted.

May 1st, 2024: Sysadmins will start brownouts to find applications that are still using OAuth 1.0a or HTTP Basic Auth

June 1st, 2024: OAuth 1.0a and HTTP Basic Auth will be shut down.

pnorman commented 1 year ago

Over the last week 5% of API calls were HTTP Basic Auth, 23% OAuth 1.0a, and 73% OAuth 2.0. There's some daily jobs that are being run that call OAuth 1.0a starting at 01:15 UTC, 02:00 UTC, and one that calls HTTP Basic at 02:30 UTC, but even excluding those it's 18% OAuth 1.0a.

5 months ago, it was 70% OAuth 2.0, some traffic is moving to OAuth 2.0 but slowly.

mmd-osm commented 1 year ago

I think it would be helpful to have some stats highlighting the top OAuth 1.0a users. This way, we could raise follow up issue in the respective app repo. This would also help us to better track the overall progress.

The number of valid tokens per app id might be a good starting point:

SELECT app.id AS app_id,
       app.name AS app_name,
       count(tk.id) AS token_count
FROM client_applications AS app
LEFT JOIN oauth_tokens AS tk ON app.id = tk.client_application_id
AND tk.invalidated_at IS NULL
GROUP BY app.id,
         app.name
ORDER BY token_count DESC
LIMIT 100;

Possibly there are better ways to gather relevant insights on per-app OAuth 1.0a usage.

tomhughes commented 1 year ago

I doubt the token count is very useful because some apps save a token and use it persistently and some generate a new one on every use.

More useful might be the number of users with at least one token for an app though there is still the question of how recent it is.

tomhughes commented 1 year ago

Here's a complete list of UAs that used OAuth 1 yesterday, with a count of requests:

      1 FitoTrack
      2 CityZen
      2 Zend_Http_Client
      5 AtlasNavi
      5 viking
      6 Go-http-client
     14 Authlib
     17 maps.me
     20 Node
     22 PHPoAuthLib
     30 FireYak
     32 Locus
     43 OAuth
     50 OsmAnd
     57 MapRoulette
     73 Apache-HttpClient
    249 Python-urllib
    255 Python-httplib2
    301 GuzzleHttp
    338 PECL-OAuth
    370 osm-wikidata
    431 NewOSMContributorsInBelgium,
    450 Organic%20Maps
   1573 MAPS.ME
   2776 Vespucci
   3875 okhttp
   6254 Dalvik
  10015 StreetComplete_ee
  13827 python-requests
  17775 -
  45333 NewOSMContributorsInRussia,
  70304 Mozilla
  74752 Java
  87722 StreetComplete
 129793 JOSM

I've reduced it down to just the UA name to exclude version differences etc.

tomhughes commented 1 year ago

The number of generic frameworks in that list limits it's usefulness (:angry:) so here's some results from @mmd-osm's query but limited to tokens issued this year to try and exclude historic things, and everything over 1000 tokens as a start:

 app_id |                 app_name                 | token_count 
--------+------------------------------------------+-------------
   3998 | Geofabrik Downloads                      |      198430
    750 | uMap                                     |       88382
   1896 | Organic Maps                             |       43020
   3277 | HOT Export Tool                          |       22932
   2302 | New MapRoulette                          |       20081
  10793 | StreetComplete                           |       18758
   2985 | How did you contribute to OpenStreetMap? |       17070
   9790 | MR MKS                                   |       14896
   3817 | Osmcha                                   |       11493
   5742 | umap-de                                  |       10960
   8524 | Organic Maps                             |       10710
    946 | JOSM - Java OpenStreetMap Editor         |        9061
   5066 | OsmAnd                                   |        9050
   7526 | OSM-Boundaries                           |        7487
    821 | OsmHydrant                               |        5224
   1875 | uMap Framasoft                           |        5214
   8520 | OrganicMaps F-Droid                      |        4526
   9922 | tasking-manager-hyd.geo.apple.com        |        4478
   3175 | overpass turbo                           |        3583
   3551 | OSM Streak                               |        3489
   4188 | TeachOSM Tasking Manager                 |        3361
   5265 | OpenPlaceReviews                         |        3226
   2100 | KartaView                                |        3098
   7404 | MapComplete                              |        2888
  11156 | DEMR3                                    |        2773
  10210 | Vespucci 2022                            |        2621
  12240 | OSM Imagery                              |        2616
   1198 | umap.osm.ch                              |        1941
   1161 | Osmose                                   |        1586
   1917 | GNOME Maps                               |        1411
    952 | Vespucci                                 |        1376
   8490 | mr_mumbai                                |        1249
   3215 | Freemap Slovakia                         |        1200
   6020 | OSM Teams                                |        1070

I think there's plenty there we can target as a starting point...

tsmock commented 1 year ago

129793 JOSM

Do we have a better idea of when OAuth 1.0a/Basic Auth will be officially deprecated? Last I heard, it was still under consideration.

I've had a ticket open in JOSM Trac for hiding Basic Auth/OAuth 1.0a (see #22810) which I had scheduled to revisit in January. I don't (yet) want to remove OAuth 1.0/Basic Auth, since it still works, but I don't mind hiding it from new users if necessary.

tomhughes commented 1 year ago

I'm inclined to say they're deprecated right now - that's literally the point of this ticket that we're trying to get people to move to OAuth 2 instead.

The question is when we stop supporting them.

I wasn't actually bothered by the JOSM entries because I assumed it was just people that had configured it some time ago and that they could migrate when necessary.

pnorman commented 1 year ago

Yes, this ticket starts the deprecation process. We have no date for when we will shut down OAuth 1.0a or HTTP Basic Auth, but earlier stated that we did not plan to shut them off this year. I would expect sometime mid-next year but that depends entirely on how efforts to move people to OAuth 2.0 go.

pnorman commented 1 year ago

MapRoulette has moved, see https://github.com/maproulette/maproulette3/issues/1876

I reached out to the TeachOSM and Apple Tasking Managers.

mmd-osm commented 1 year ago

OSM-Boundaries: https://osmboundaries.userecho.com/communities/1/topics/66-move-from-oauth-10a-to-oauth-20-for-osm-logins

mmd-osm commented 1 year ago

I'm not aware of a repo for How did you contribute to OpenStreetMap? Let's try to ping @pa5cal instead.

pa5cal commented 1 year ago

I'm not aware of a repo for How did you contribute to OpenStreetMap? Let's try to ping @pa5cal instead.

Thanks, @mmd-osm ! I will try to change my HDYC webpage to OAuth2 in the next few months. The first tests today looked quite promising.

pietervdvn commented 1 year ago

For all (javascript) devs reading this https://www.npmjs.com/package/osm-auth has been updated to use OAuth2.0

simonpoole commented 11 months ago

To maybe point out the obvious:

turning off basic auth is mainly a question of improving security / control on osm.org. 3rd party installs will likely continue to use it and so support for it will need to remain in the rails-port.
turning off OAuth 1a provides a reduction in code and UI complexity for osm.org, but no other benefits. @gravitystorm hasn't expressed himself if he wants to drop the code completely or not, but I suspect there may be again 3rd party installs that will want to continue to use it.

tl;dr a hard requirement for OAuth 2 on osm.org just means that everybody that wants to maintain compatibility with as many installs as possible, that is at least the JOSM devs and myself, simply needs to add additional complexity with no benefits at all.

tomhughes commented 11 months ago

There are security concerns with OAuth 1 as well - even if the protocol is still good the fact that our implementation relies on unmaintained gems is not great.

tsmock commented 11 months ago

tl;dr a hard requirement for OAuth 2 on osm.org just means that everybody that wants to maintain compatibility with as many installs as possible, that is at least the JOSM devs and myself, simply needs to add additional complexity with no benefits at all.

I've been looking at removing support for Basic Auth and OAuth 1.0a from JOSM (see #22810; if we end up going that route, I intend to do an Upcoming changes section in the JOSM Startup notes specifically calling the change out a few months beforehand).

Any OSM server that uses the rails port/cgimap (is there any other?) should support OAuth 2 (OAuth 2 has been supported by the OSM website since June 2021 (>2 years)). If they are using the OSM website stack and they do not support OAuth 2, they should update.

I may reach out to projects that use the OSM website stack and see if they want to give me a set of OAuth2 clientIds to hardcode into JOSM. I'd like to avoid doing this, but a RFC7591 implementation for doorkeeper isn't on the horizon (or in their bug tracker).

mmd-osm commented 11 months ago

@simonpoole : to be honest, I don't have much visibility how people use Vespucci (or JOSM) with 3rd party installs. I can think of OpenHistoricalMap and OpenGeofiction maybe, and possibly some non public in-house use cases. I don't see why we also couldn't give them a heads up. We would need to know who they are, and how to reach out to them.

We should also encourage 3rd party installs to stay somewhat up-to-date with the upstream Rails repo, which would avoid the issue altogether.

simonpoole commented 11 months ago

I don't have much visibility

To illustrate the issue why it is very difficult/near impossible to know anything about how our systems are used:

There's an Indian project/utility/group of utilities that are using a fork of Vespucci to collect/edit data and likely upload it to a private rails-port instance. I've never communicated with them (and there is no obvious way to do so), the only reason I know they exist is that I've now and then seen crash dumps because they broke things and didn't change the crash dump URL.

tsmock commented 11 months ago

For OHM, I pinged them on slack; they've since made an issue: https://github.com/OpenHistoricalMap/issues/issues/610 . They apparently already support OAuth 2.

For OpenGeofiction, we'd probably have to email them -- see https://wiki.opengeofiction.net/index.php/OpenGeofiction:Contact . They may support OAuth 2 already, but I'd have to make an account to check.

For non-public in-house use cases, as noted in https://github.com/openstreetmap/operations/issues/867#issuecomment-1725865973, I was intending to add an Upcoming changes section to the JOSM Startup page specifically to inform users about this. I don't know if Vespucci has a similar "startup news" page that can be updated out-of-band.

I'll go ahead and add the upcoming deprecation/removal to the JOSM startup page now.

We should also encourage 3rd party installs to stay somewhat up-to-date with the upstream Rails repo

AKA update their site at least every 2 years.

EDIT: JOSM startup page edited

simonpoole commented 11 months ago

I don't know if Vespucci has a similar "startup news" page that can be updated out-of-band.

As a rule we don't phone home.

mmd-osm commented 11 months ago

There's an Indian project/utility/group of utilities that are using a fork of Vespucci to collect/edit data and likely upload it to a private rails-port instance.

@simonpoole : does Vespucci support 3rd party installs using OAuth 1.0a without forking the app? I remember having to do this many years ago for testing purposes, although things may have changed in the meantime.

In general, if someone is running their own stack, I would assume they have sufficient knowledge to handle upstream changes as well.

simonpoole commented 11 months ago

@simonpoole : does Vespucci support 3rd party installs using OAuth 1.0a without forking the app?

Yes, you can load additional client keys for a while (http://vespucci.io/help/en/17.0.0%20Release%20notes/#improved-key-handling-and-new-oauth-10a-keys). Matter of fact the standard key used now days is configured using that mechanism at build time, we do still have a (different) key in the repo, because F-Droid.

Now if somebody actually uses that is a different question, there is literally no way we can know. But as I pointed out above, the team here needs to decide if it will drop OAuth 1.0a support and then client devs can decide if they want to drop support some time in the future. Which is related to but not the same question as what is supported on osm.org, I'm very much for dropping basic auth support there -now-, but would consider dropping it from the code silly.

mmd-osm commented 11 months ago

As simple as basic auth may seem, it introduced a dependency on Argon2 both on Rails and CGImap, and was quite a pain to get working properly across implementations. In total, we're using 6 different implementations to authenticate users today. The authentication topic has accumulated some technical debt.

tsmock commented 11 months ago

we're using 6 different implementations

~~Basic Auth~~

~~OAuth 1.0a~~

OAuth 2.0 x2 since we have the rails port and cgimap. Do I have things right? It seems like there should only be one implementation that should matter for each of the oauth methods. I don't think cgimap has to implement anything to do with the authentication dance for the oauth methods anyway, just verify the given tokens. But I don't know if you are counting the verification in cgimap as an "implementation".

mmd-osm commented 11 months ago

Yes, exactly, CGImap needs to validate the Bearer token for OAuth 2.0 only. That's a single database SELECT to check the write_api permission, check for expired / revoked tokens, etc. All the other parts are handled by Doorkeeper on Rails.

Even for Basic auth and OAuth 1.0a, CGImap implements the validating part only, which is surprisingly complex in case of OAuth 1.0a. Overall about 2500 LOC, including test cases.

danieldegroot2 commented 11 months ago

Moved to this comment.

harry-wg commented 11 months ago

Could somebody say if, or how, this will affect ordinary end users rather than programmers?

In particular will I still be able to log in to the main OSM pages using my username and password?

there is absolutely no way that I would be willing to log in using a third party spyware company's account (ie google, facebook etc).
will I still be able to edit with JOSM, again without having to use a third party spyware site for my login?
less important, but I occasionally use streetcomplete, overpass turbo and a few other useful sites. Will end users of any of those be affected?

I also have a personal use program running leaflet.js. It uses a few APIs in read-only form:

the Notes API (which as far as I remember doesn't require me to be logged in to OSM read existing notes)
overpass turbo queries (again, read only and I'm not aware of it requiring an OSM login

Can I just assume that JOSM and any most other third parties will implement any necessary changes internally and I'll be able to continue doing all of the above with just my OSM username and password?

tomhughes commented 11 months ago

This is about how things authenticate to OpenStreetMap not about how you login - when you authorise an application with OAuth (whether 1 or 2) you login to OpenStreetMap in the normal way and the app is then given a token it can use to access OpenStreetMap without having to know your password.

simonpoole commented 11 months ago

I also have a personal use program running leaflet.js. It uses a few APIs in read-only form:

Read access (with the exception of user specific data) currently doesn't require authentication, so yes.

pnorman commented 11 months ago

The current load shows that >85% of normal requests come from oauth 2.0, and those coming with OAuth 1.0a and HTTP basic tend to come in batches which indicates some kind of scripted load.

gravitystorm commented 11 months ago

turning off basic auth is mainly a question of improving security / control on osm.org. 3rd party installs will likely continue to use it and so support for it will need to remain in the rails-port.

turning off OAuth 1a provides a reduction in code and UI complexity for osm.org, but no other benefits. @gravitystorm hasn't expressed himself if he wants to drop the code completely or not, but I suspect there may be again 3rd party installs that will want to continue to use it.

I know this is the operations tracker and not the openstreetmap-website tracker, but I'll chip in here since I was mentioned.

We already have configuration options to enable basic_auth_support and oauth1_support in the codebase, and both are currently true by default. I would expect the sequence of events to be as follows:

We leave these settings as defaulting to true for now, which makes life easier for us in tests and CI and ensures they keep working.
OSMF (OWG) sets these options to false on the osm.org production servers (not necessarily simultaneously, and perhaps following the famous brownout/rolling blackout approach - up to them). This requires no software changes, just configuration changes fully under OSMF control, and so can be done at any time.
At some point after they have been disabled on osm.org, we will switch them to default to false in the default configuration. This will alert anyone who is deploying new versions of the codebase but who hasn't already heard about the changes.
Downstream deployments will then have an opportunity to temporarily restore the configuration to true, while they update their client software, and restore the settings to false at their convenience. Most, perhaps all, off-the-shelf client software used by custom deployments will already have OAuth 2.0 support due to the client software already having been updated for osm.org.
We'll wait for any feedback from any custom deployments who have encountered problems.
At some point later, after a reasonable amount of time (6-12+ additional months?), we'll remove the settings and remove the unused code from the project.

I don't propose retaining the unused code for these auth methods indefinitely, but I'm happy to hear feedback from custom deployments if their removal would be unduly burdensome.

mmd-osm commented 11 months ago

Read access (with the exception of user specific data) currently doesn't require authentication, so yes.

I should mention that some read only endpoints consider authentication for rate limiting (details: https://lists.openstreetmap.org/pipermail/dev/2016-October/029544.html). I'm not aware of any rate limiting for read only access to the Notes API on Rails, so you can simply skip it in this case.

mmd-osm commented 11 months ago

@pa5cal : I've noticed this X thread where a user was reporting some issues with HDYC and OAuth2. Please keep us updated, in case you can't troubleshoot the issue on your end and need further support to see what's happening on Rails.

pa5cal commented 11 months ago

@mmd-osm Thank you very much. Yes, some assistance would be greatly appreciated.

Since I switched to OAuth2 at HDYC, several hundred users should have successfully logged in. So far, only one user has contacted me to report that it doesn't work (see the thread above). I then renamed my test account, but was still able to successfully authenticate and use HDYC.

If you have any tips from the Rails side, that would be great.

mmd-osm commented 11 months ago

@tomhughes : I'm not sure if there's an easy way to troubleshoot this. Apparently, user https://www.openstreetmap.org/user/Paul%20Berry is unable to log in to HDYC using OAuth2, while many other users, including myself, didn't experience any issue.

The user has recently logged on the OSM community forum, so some OAuth 2 based apps seem to work for them: https://community.openstreetmap.org/u/paul_berry

tomhughes commented 11 months ago

Without more specific information about what happens it looks like an HDYC side problem to me - he has loads of access grants for it so he's been through the login and authorised it but none of those have been turned into tokens.

That basically means that either the redirect back to HDYC isn't working for some reason, or HDYC is failing to convert the grant to a token but without knowing exactly what he sees when he tries it it's hard to say more.

tsmock commented 11 months ago

A couple of sanity checks:

Does the user have any browser addon that might be removing URL parameters?
Have they tried a different browser that they do not use normally?

Firefishy commented 11 months ago

I can confirm there is an error server side at the moment with the https://www.openstreetmap.org/oauth2/token endpoint.

Firefishy commented 11 months ago

To confirm, the 500 error issue from https://www.openstreetmap.org/oauth2/token issue has now been resolved.

tomhughes commented 11 months ago

Can I suggest we don't use this ticket as a dumping ground for every problem with authentication?

biodranik commented 8 months ago

What was the reason of producing not expiring OAuth2 tokens?
What are the chances that this policy changes? When it may happen?

pnorman commented 8 months ago

We should look at current usage numbers in the new year and see where we're at.

pnorman commented 8 months ago

Current avg traffic is 85% oauth2. There is some process running on oauth1a that starts at 2 AM UTC and takes 1h45m and 7.4 reqs/s, and another process running at about 2:10 AM UTC with 8.5 req/s on top of the first one. It could also be some other combination of processes.

Could we check the logs for what is using OAuth1a at 2 in the morning? It almost certainly shouldn't be as it's obviously not editing activity.

pnorman commented 8 months ago

OSMF (OWG) sets these options to false on the osm.org production servers (not necessarily simultaneously, and perhaps following the famous brownout/rolling blackout approach - up to them). This requires no software changes, just configuration changes fully under OSMF control, and so can be done at any time.

@mmd-osm is there a way to toggle OAuth1.0a with a setting in cgimap?

What I would suggest for a sequence of events is

turn off the ability to register new oauth1 apps
turn off the ability to authorize new applications with oauth1. It might be worth doing this one as a brownout
brown out using oauth1 and HTTP basic for a bit
after a wait, turn off oauth1 and HTTP basic.

Probably start in Jan and finish in a 3-5 months?

mmd-osm commented 8 months ago

What was the reason of producing not expiring OAuth2 tokens? What are the chances that this policy changes? When it may happen?

@biodranik : Tokens are currently non-expiring (see link), although that's a bit of an anti-pattern.

I'd love to see an option for app devs to manage the token lifetime per registered app. If you don't support token refresh, you could stick to the current "non-expiring" tokens. Others may find it more suitable to expire tokens after a few hours or days. Maybe @tomhughes could comment, if this might be an option for us in the future.

@mmd-osm is there a way to toggle OAuth1.0a with a setting in cgimap?

@pnorman : Yes, 0.8.x releases have settings for both OAuth 1.0a and Basic Auth. It was added in https://github.com/zerebubuth/openstreetmap-cgimap/pull/262 - in 0.9.x, OAuth 1.0a support will be removed from the code (https://github.com/zerebubuth/openstreetmap-cgimap/pull/354).

What I would suggest for a sequence of events is turn off the ability to register new oauth1 apps

I would add an info/warning message to the OAuth1.0a screen first, maybe also rename "OAuth 1 settings" to "Legacy OAuth 1 settings".

What exactly is "brownout" in this context? I can think of disabling it based on time, IP, app, or maybe something else?

Probably start in Jan and finish in a 3-5 months?

@Zverik mentioned that he's somewhat unhappy with the proposed timelines. We need to see what is feasible.

tomhughes commented 8 months ago

Current avg traffic is 85% oauth2. There is some process running on oauth1a that starts at 2 AM UTC and takes 1h45m and 7.4 reqs/s, and another process running at about 2:10 AM UTC with 8.5 req/s on top of the first one. It could also be some other combination of processes.

The main thing I can see using oauth1 that starts at 2am is @Zverik's NewOSMContributorsInRussia thing. I can't see anything obvious extra at 10 past but maybe it ups it's request rate after an initial phase?

pnorman commented 8 months ago

What exactly is "brownout" in this context? I can think of disabling it based on time, IP, app, or maybe something else?

Turning it off for a short time then turning it back on in order to raise notifications with any clients then give them time to migrate without their programs being broken until they migrate.

simonpoole commented 8 months ago

Isn't this all 'slightly' premature given that while there have been handwavy indications of support stoping in roughly a years time, an actual date hasn't been communicated and breaking contributors uploads before that point in time would seem to be completely unwarranted?

pnorman commented 8 months ago

In the diary post I said it was unlikely we would do anything this year, and mmd said Q4 2023 or Q1 2024 in his issue. I said later on sometime mid next year for when I thought they would be shut off.

Starting late Jan and taking 3-5 months means late April or June.

As for an announcement with wider reach, it's good to have a rough schedule before then, and that's what's being discussed here.

tomhughes commented 8 months ago

New list of applications that have issued more than 50 oauth 1 tokens in the last eight weeks:

 app_id |              app_name              | token_count 
--------+------------------------------------+-------------
   3998 | Geofabrik Downloads                |       34376
    750 | uMap                               |       18961
   3277 | HOT Export Tool                    |        6435
   1896 | Organic Maps                       |        5589
  10793 | StreetComplete                     |        2780
   8524 | Organic Maps                       |        2702
   3817 | Osmcha                             |        2281
   5742 | umap-de                            |        2206
   1875 | uMap Framasoft                     |        1470
   5066 | OsmAnd                             |        1454
    821 | OsmHydrant                         |        1410
   8520 | OrganicMaps F-Droid                |        1242
   4188 | TeachOSM Tasking Manager           |         953
    946 | JOSM - Java OpenStreetMap Editor   |         926
   3551 | OSM Streak                         |         819
   9922 | tasking-manager-hyd.geo.apple.com  |         712
  10210 | Vespucci 2022                      |         592
   7526 | OSM-Boundaries                     |         559
  12240 | OSM Imagery                        |         505
   2100 | KartaView                          |         493
   1161 | Osmose                             |         397
   9829 | Map builder                        |         328
    952 | Vespucci                           |         319
   1198 | umap.osm.ch                        |         277
   6020 | OSM Teams                          |         221
   5969 | OSM Calendar                       |         197
  11156 | DEMR3                              |         188
   3007 | Mapwarper.net Login                |         175
   1184 | Level0                             |         173
   3740 | Pic4Review                         |         141
   5628 | OsmInEdit                          |         133
    600 | OSMTracker for Android™            |         132
   2344 | RevertUI                           |         129
   3000 | osm.wikidata.link                  |         129
   1917 | GNOME Maps                         |         126
  10772 | Gestor de Tareas OSM-es            |         110
   9630 | osm.karlas.si                      |         110
   6359 | Level0 - private by mueschel       |          91
   8650 | To-Fix-production                  |          86
   8490 | mr_mumbai                          |          83
   2334 | MapContrib                         |          79
  10201 | aed-mapa                           |          72
   7003 | In-House iD Editor Production      |          71
    760 | iD                                 |          59
   5889 | Healthsites                        |          54
    610 | locus                              |          53
   6333 | Divide and map. Now.               |          53
   5200 | NotesReview                        |          52
   7805 | Mapas Nomadas                      |          51

openstreetmap / operations

Deprecation of HTTP Basic Auth and OAuth 1.0a #867