Remove the v1 API

[richvdh]

Blocked by matrix-org/synapse#9677

The deprecated v1 identity service API is a source of security issues and poor UX. We should set a date to remove it, and follow through with that.

[richvdh]

Some notable places this is still used:

• Synapse (see https://github.com/matrix-org/synapse/issues/9677)
• Element-web (for .well-known validation or something - see MSC2499, https://github.com/matrix-org/matrix-spec/issues/557)
• we see a lot of requests from mxisd user-agents. mxisd is unmaintained aiui, so it's hard to know how best to proceed here.

[turt2live]

mxisd is more likely to be ma1sd: https://github.com/ma1uta/ma1sd

[callahad]

Would be good to set a public deprecation timeline for this

[turt2live]

related: https://github.com/matrix-org/matrix-doc/pull/2713

[erikjohnston]

@turt2live suggests that there is other work that might need to be done before we can remove the v1 APIs, but he'd need to go and spend some time figuring out what that is. (I hope that is a fair representation)

[BillCarsonFr]

Notice that email invites are still using V1 api in order to sign the invitation. Also we cannot just change the email template to use the v2 API, because v2 API requires consent for the sign-ed25519 endpoint, this will lead to a pretty bad invite flow.

This API is defined as helper for client that cannot do crypto, but still element clients are using it. I tried to get more info on what clients are supposed to do but failed to find more info.

I guess that before removing this API we should:

• Update documentation to explain the invitation signing process (for client to do it without server)
• Update EW, EA, EI to do the signing themself from the invite link
• Modify the email template to not send a sign URL (as it wont be usable as is because in v2 params are in request body and not in query) but instead send the token, private_key as well as the identity server url as regular parameters

[richvdh]

I decided to try and pull some figures on where the v1 API is being used. Over the 7 days to 24 June:

• Lookup API:
  • 53719 calls to /v1/lookup (excluding those from our internal Prometheus :roll_eyes:). These are split:
  • 52538: mxisd (https://github.com/ma1uta/ma1sd/issues/114)
  • 1129: Synapse (https://github.com/matrix-org/synapse/issues/13206)
  • 10195 bulk lookups via /v1/bulk_lookup. Useragent splits:
  • 9602 Riot.im/0.9.1 on Android. Suspect this is a single user doing something weird
  • 435 Riot on iPhone, with weird version numbers. Forks/old versions? None from current versions of Riot-iOS.
  • 124 AOCIM/1.5. No real idea what this is.
• Due to https://github.com/matrix-org/synapse/issues/5881:
  • 6082 GET /_matrix/identity/api/v1/3pid/getValidated3pid (of which 468 were successful: the remainder were polls)
  • 200 POST /_matrix/identity/api/v1/validate/email/requestToken
  • 448 GET /_matrix/identity/api/v1/validate/email/submitToken
  • 604 POST /_matrix/identity/api/v1/validate/msisdn/requestToken
  • 405 POST /_matrix/identity/api/v1/validate/msisdn/submitToken
• Due to https://github.com/matrix-org/synapse/issues/13171:
  • 78 POST /_matrix/identity/api/v1/3pid/unbind
• Due to https://github.com/matrix-org/synapse/issues/13206:
  • the 1129 calls to /v1/lookup mentioned above
  • 1193 calls to POST /_matrix/identity/api/v1/store-invite from Synapse
    • 13 GET /_matrix/identity/api/v1/pubkey/isvalid
• Unclassified:

  365 POST /_matrix/identity/api/v1/sign-ed25519
   17 GET /_matrix/identity/api/v1/sign-ed25519
  288 GET /_matrix/identity/api/v1/pubkey/ephemeral/isvalid
    8 HEAD /_matrix/identity/api/v1/validate/email/submitToken
    6 OPTIONS /_matrix/identity/api/v1/account/register
    3 GET /_matrix/identity/api/v1/terms
    3 GET /_matrix/identity/api/v1/pubkey/ed25519:0
    3 GET /_matrix/identity/api/v1/hash_details

[t3chguy]

/_matrix/identity/api/v1/sign-ed25519 is a strange one.

3PID (email) invites include a link like https://app.element.io/#/room/%21cCpRBTFuXwPmOCUFRL%3Amatrix.org?email=michaelt%40element.io&signurl=https%3A%2F%2Fvector.im%2F_matrix%2Fidentity%2Fapi%2Fv1%2Fsign-ed25519%3Ftoken%3DcBeQOefWJXQrcgHavWuqjMtiazBrYMCvLbpGvOSWFLkitbMbuFGyEmgBbcPtfHhHxiKKsdEbEXSJezCbTXowCuFaQIKV%26private_key%3DRkzs-2BDCjkOOnS74LwLRO_Uw&room_name=&room_avatar_url=&inviter_name=Michael%20Telatynski&guest_access_token=&guest_user_id=&room_type=

Which passes a signurl using the aforementioned API, but encodes the params in query params rather than a POST body which the v2 API docs state the API demands. This means a simple v1->v2 port isn't so simple as now clients would need to receive the parameters outside of the URL and include them in a POST body themselves, with no backwards compatibility.

The API description being

To aid clients who may not be able to perform crypto themselves, the identity server offers some crypto functionality to help in accepting invitations. This is less secure than the client doing it itself, but may be useful where this isn’t possible.

when in reality clients can't really do the signing themselves without parsing the signurl in brittle ways given the URL params don't match any known documentation means that clients are forced to blindly just call a given URL in undocumented ways.

[richvdh]

For links, the endpoint in question used to be specced at https://matrix.org/docs/spec/identity_service/r0.3.0.html#deprecated-post-matrix-identity-api-v1-sign-ed25519; note that a GET variant was never part of the spec.

@t3chguy (and @BillCarsonFr, since I think you said the same thing): I'm failing to grok how this "signurl" is used in the 3pid flow. How is the URL generated? How is it used? I can't find it mentioned at https://spec.matrix.org/v1.5/client-server-api/#server-behaviour-7.

[t3chguy]

It is being used as POST but with no body and all params being query params, so still doesn't match that spec

The link comes from the email templates https://github.com/search?q=repo%3Amatrix-org%2Fsydent+sign-ed25519&type=code The client includes a third_party_signed in the /join/ request which is the object it got back from the sign-ed25519 API, the path & params to which it got via signurl in the link in the email template above

[richvdh]

It is being used as POST but with no body and all params being query params, so still doesn't match that spec

Right. Yes.

The link comes from the email templates https://github.com/search?q=repo%3Amatrix-org%2Fsydent+sign-ed25519&type=code

Is that the link you meant? It doesn't seem to give anything useful here.

[t3chguy]

Yup - all the templates have v1 references to the API.

[richvdh]

that link definitely doesn't show me those results, but thanks

[t3chguy]

Ah, might be because I'm on the Github Search Beta which makes the search quite usable

[turt2live]

I'm failing to grok how this "signurl" is used in the 3pid flow. How is the URL generated? How is it used? I can't find it mentioned at https://spec.matrix.org/v1.5/client-server-api/#server-behaviour-7.

@richvdh worth noting that there's two flows an email invite can take, and only one of those flows is documented in the spec. The other is completely missing (I couldn't find an issue at first glance, but know this to be true)

[richvdh]

@richvdh worth noting that there's two flows an email invite can take, and only one of those flows is documented in the spec. The other is completely missing (I couldn't find an issue at first glance, but know this to be true)

Opened https://github.com/matrix-org/matrix-spec/issues/1359 to track this, then.

Re the signurl parameter: based on the evidence of https://github.com/matrix-org/sydent/blob/main/res/matrix-org/invite_template.eml and friends, it looks like:

• The sign-ed25519 URI is a sydent URI generated by sydent, and as such we can consider it to be an implementation detail of sydent, and there is no need for it to be part of the spec.

  Following the example of similar endpoints in Synapse (such as the SSO callback), ideally it would not be on the /_matrix/... path and instead would be called something like /_sydent/get_stored_invite, to emphasise that it's not part of the spec. It would also ideally be documented in the Sydent docs.

  I have opened https://github.com/matrix-org/sydent/issues/533 to track this.

• The signurl parameter - indeed, the whole of the "single-click" link in the email, including the /#/room/<room_id> fragment, and the other pseudo-query-params - appears to be somewhat specific to Element, though since homeservers can change the target of the link via the unspecced org.matrix.web_client_location parameter, the whole thing seems somewhat dicey. I have opened https://github.com/matrix-org/matrix-spec/issues/1360 to track this.

  It has to be said that a good start would be for the format of that link to be documented somewhere - either in the element-web docs, or the sydent docs. But in any case that's entirely orthogonal to the removal of /v1/.

[t3chguy]

and as such we can consider it to be an implementation detail of sydent, and there is no need for it to be part of the spec.

Yes, and no. Element Web code has to append a &mxid=<my_mxid> to the URL, presumably removing it would make it stop working, this behaviour means we'd need to spec that you need to do such for signurl somewhere.

[richvdh]

Element Web code has to append a &mxid=<my_mxid> to the URL, presumably removing it would make it stop working

Ugh. But also, surely that would work just as well if the root of signurl was https://matrix.org/_sydent/get_stored_invite?... instead of https://matrix.org/_matrix/identity/api/v1/sign-ed25519?... ? In other words, the location of the endpoint is entirely internal to sydent, rather than being a matter for the spec, even if the parameters that should be passed to the endpoint have to be specced (which would be part of matrix-org/matrix-spec#1360).

[t3chguy]

Quite right - especially the ugh

[H-Shay]

Note that there is a config option to disable V1 bindings: https://github.com/matrix-org/sydent/pull/267 - although currently this is only applied to the bind endpoint: https://github.com/matrix-org/sydent/blob/c9980a9fb89d7fa00ef45601294312110f47c565/sydent/http/httpserver.py#L129-L130