search

benbucksch / autoconfig-spec

Mail autoconfig protocol allows email clients to automatically configure email accounts. This report creates the RFC to standardize the protocol within the IETF.
Other
4 stars 2 forks source link

Backwards compatibility #7

Closed cketti closed 3 months ago

cketti commented 6 months ago

The document currently contains this text:

The purpose of this paper is to document and specify what is deployed in the wild.

However, that doesn't seem to be the case. For example, the main URL in the document is https://autoconfig.%EMAILDOMAIN%/.well-known/mail-v1.xml?emailaddress=%EMAILADDRESS% as opposed to https://autoconfig.%EMAILDOMAIN%/mail/config-v1.1.xml?emailaddress=%EMAILADDRESS% that is used by current implementations.

What is the purpose of this document? Create a new standard loosely based on what is currently used? Or document the status quo? I think either one is fine. But it should be stated clearly in the document.

benbucksch commented 6 months ago

The purpose was to only document what is deployed. Then people came and wanted all kinds of changes.

How about: "The purpose of this paper is to document and specify what is deployed in the wild, and in some cases adjust it to current best practices." ?

cketti commented 6 months ago

I don't think that clears things up. Changing the URLs is an incompatible change. So effectively this is creating a new standard. At that point you might as well change everything else people complain about (e.g. use JSON instead of XML).

If you want to document what's currently deployed, you could still add new things. But it has to be done in a backwards compatible way. E.g. use a different authentication identifier for mAuth instead of "OAuth2" which is already in use for something else.

benbucksch commented 6 months ago

The URL changes I made are backwards compatible. The new URLs are different, but the old ones are still in there for backwards compat.

If you want to document what's currently deployed, you could still add new things.

That's what I'm doing. Did you find a case where I removed an URL that is important in practice? If so, I'd like to know.

E.g. use a different authentication identifier for mAuth instead of "OAuth2" which is already in use for something else.

Nobody uses <oAuth2> yet, so that's not a backwards incompatible change.

cketti commented 6 months ago

Section 3.1:

<!-- Authentication methods:
     […]
     "OAuth2":
               mAuth. Should be added only as second alternative.
     […]
-->
<authentication>password-cleartext</authentication>

OAuth2 is currently used with providers like Gmail that require manual client registration before OAuth2 can be used. This is incompatible with the proposed mAuth mechanism. The value for mAuth could be mAuth or something like OAuth2-static.

Section 4.1:

  • 1.1. https://autoconfig.%EMAILDOMAIN%/.well-known/mail-v1.xml?emailaddress=%EMAILADDRESS% (Required)
  • 1.2. https://%EMAILDOMAIN%/.well-known/autoconfig/mail/config-v1.1.xml (Optional)
  • 1.3. http://autoconfig.%EMAILDOMAIN%/mail/config-v1.1.xml (Optional)

The list of URLs used by current implementations is:

Leaving out https://autoconfig.%EMAILDOMAIN%/mail/config-v1.1.xml?emailaddress=%EMAILADDRESS% is an incompatible change.

The …/.well-known/mail-v1.xml?… URL is repeated in section 4.3, 6.1, and 6.2.

benbucksch commented 6 months ago

Leaving out https://autoconfig.%EMAILDOMAIN%/mail/config-v1.1.xml?emailaddress=%EMAILADDRESS% is an incompatible change.

That was probably unintentionally removed. My bad. You're right, that URL should stay for backwards compat. Thanks for catching it.

(FWIW, for MX, we will put only the new URLs in the spec, because Thunderbird so far does not query that old URL for MX.)

OAuth2 is currently used with providers like Gmail

You mean for <authentication>OAuth</authentication> ? Note that the value there did not change. Only the description changed. So, this is a backwards compatible change.

If specific email clients (like Thunderbird) do special things for specific providers (like Gmail), that's their business and between Thunderbird and Google. The spec tries to get something working for all email clients and all providers (if they implement the spec) at the same time, without contracts between the parties.

cketti commented 6 months ago

Apologies. I had another look. And the URLs currently in use are:

I don't think it makes sense to add yet another URL. Clients already need to issue too many HTTP requests when trying to find server settings. We should work on a deprecation strategy to get rid of some of them, not add more.

What's the advantage of adding another one? How is it better for email providers? How is it better for email clients? How is it better for users?

IMO adding another URL would put a needless burden on the ecosystem. With no practical upside.

(FWIW, for MX, we will put only the new URLs in the spec, because Thunderbird so far does not query that old URL for MX.)

I don't think that's a good idea. At least Evolution and Nextcloud Mail already use the "old" URLs after an MX lookup.

Also, using the existing URLs gives us compatibility with services that already deploy autoconfig and support customer domains, but are not in a fallback database. Example: https://github.com/thunderbird/autoconfig/pull/86 Forcing email providers to make the config available under yet another URL to be able to support customer domains doesn't make sense.

You mean for OAuth ? Note that the value there did not change. Only the description changed. So, this is a backwards compatible change.

You keep the name, but change the meaning. How is that not an incompatible change?

If specific email clients (like Thunderbird) do special things for specific providers (like Gmail), that's their business and between Thunderbird and Google. The spec tries to get something working for all email clients and all providers (if they implement the spec) at the same time, without contracts between the parties.

There are currently configs using the "OAuth2" value in Thunderbird's fallback database. Thunderbird uses this value to decide whether to use OAuth2. Almost all of the clients also implementing Autoconfig use this database. Some of them also support OAuth2. With this change all currently existing configs using the value "OAuth2" will become invalid because the providers don't support mAuth. How should an email client handle that? Currently they (hopefully) ignore the "OAuth2" value when they don't support it with a particular provider. Once they add support for mAuth, they won't know if "OAuth2" in a config means "this is OAuth2 with a provider I don't support" or "this is mAuth, I can do this". This problem can be easily avoided by using a different value.

benbucksch commented 6 months ago

I don't think it makes sense to add yet another URL.

I concur with the goal to keep the amount of URLs that a client as to query as small as possible. I have the same goal.

I am incorporating feedback from multiple parties. Some IETF people told me to use the .well-known/ best-practice. Yes, of course that means changing the URL. Given that this should become an Internet standard, we should use best practices. If I don't incorporate the feedback now in some way, I'll just get the same feedback again later.

What you're asking is in direct contradiction to what other people asked me: it's impossible to change it and to not change it. As a compromise, I added the new URL, with the old ones as backwards compat.

My plan is to define the URLs as we think they are good, plus those for backwards compat. Then I'll make a server survey and see which backwards compat URLs are actually used. Then, I'll drop all old URLs that don't make any significant difference in the result, or can easily be replaced by talking to a handful ISPs.

value "OAuth2" ... How should an email client handle that?

If you have a complete OAuth2 config, either from the autoconfig under <mAuth>, or from hardcoded values in your app, you can use it.

For Thunderbird, it has no practical relevance. You have signed the contract with Google and Microsoft, you have a hardcoded client ID, and you handle their expiry times and all. Given that you can handle Gmail now, you can handle it then.

To put it in another way: To make systems work well, the sender needs to be very strict, and the receiver needs to be very liberal (Jon Postel). As mail client, you're not the audience for this particular angle. As a mail client, you need to deal with everything. The goal here is to make ISPs configure their OAuth2 servers in a way that will work properly. It's mostly a red flag for ISPs (other than Google, MS and Yahoo) that they need to configure their OAuth2 server in a specific way, if they want this to work. The goal is to make the system saner, so that mail clients don't have to deal with too much crazy nonsense from ISPs.

cketti commented 6 months ago

I am incorporating feedback from multiple parties. Some IETF people told me to use the .well-known/ best-practice. Yes, of course that means changing the URL. Given that this should become an Internet standard, we should use best practices. If I don't incorporate the feedback now in some way, I'll just get the same feedback again later.

That's a reasonable request for a new standard. I think it's quite unreasonable for an existing standard.

The goal of the .well-known standard is to avoid name conflicts. I don't think this has been a problem for autoconfig in the past, especially since autoconfig.{emaildomain} itself is a separate namespace already.

Adding another URL is a change that requires a lot of people to make changes with absolutely no benefit for them.

benbucksch commented 3 months ago

Hey cketti, after having thought about it, I agree with you. The autoconfig.domain URL is dedicated to this protocol and therefore there is no conflict with any existing page or deployment setup of live sites. If it was a new protocol, we'd use .well-known/ simply for consistency, but in this case, backwards compat is critical and using another URL has no advantage.

I changed the 1.1. URL back to https://autoconfig.%EMAILDOMAIN%/mail/config-v1.1.xml?emailaddress=%EMAILADDRESS%, as it was before and has always been used in practice. Thanks for your feedback.

I therefore consider this resolved now. Fixed in git commit d32028722.

Concerning the OAuth2 vs. mAuth, that is not a breaking change, because no ISP (other than the big 3-4) can do OAuth2 at all right now, and the only objective of mAuth is to do it properly and sanely before we deploy it widely to all ISPs. OAuth2 has never been properly specced for mail, and that's the objective now, before we deploy it to all kinds of small companies.