Homeserver url issue - Githubissues

Dear team,

(domains, subdomains and IP addresses changed for example purposes)

I have set up a local (private, non-federated) dendrite server that is running behind an nginx proxy. So far so good; matrix is functioning properly and element-web connects to it perfectly.

The machine is only publicly accessible through SSH and VPN. The machine has access to an internal wildcard Let's Encrypt certificate. It is running Dnsmasq locally to provide easy to use subdomains such as git.example.com, element.example.com, matrix.example.com etc. The DNS server is the default for the VPS and is also pushed to clients once they are connected through either a VPN or SSH tunnel. It works perfectly for everything except Sydent. A number of services are running as rootless podman containers (including Dendrite, Postgres, Element, Gitea / Forgejo, Woodpecker, Vikunja, NGINX and Sydent).

Dendrite is running at matrix.example.com, while users are registered as @user:example.com through delegation. Element is configured with base url https://matrix.example.com with homeserver example.com

When attempting to use Sydent as the identity server I get error:

Error fetching https://example.com/.well-known/matrix/server: Expecting value: line 1 column 1 (char 0)
WARNING - Unable to contact the Matrix homeserver (NoRouteError)

Which makes sense as it should be attempting to make the connection to matrix.example.com - not example.com directly.

Is there a way to force Sydent to go to matrix.example.com instead of example.com ?

I did copy (and edit) the nginx file (matrix.example.com) to run as example.com simultaneously, but that confused element for some reason, as well as breaking support for accessing the publicly accessible website (example.com) when connected to the VPS via VPN or SSH tunnelling. (The public domain is accessible through Cloudflare (example.com, www.example.com) but when connected to the VPS and its internal dnsmasq server (which is authoritative internally) other subdomains become available). This also did not fix the problem; Sydent was able to connect but complained about receiving incorrect JSON instead. Either way; it would be best to force Sydent (somehow) to use matrix.example.com as the target URL for communication.


server {
    listen 443 ssl; # IPv4
    listen [::]:443 ssl; # IPv6
    server_name matrix.example.com;

    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

    proxy_set_header Host      $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_read_timeout         600;

    location /.well-known/matrix/server {
        return 200 '{ "m.server": "matrix.example.com:443" }';
    }

    location /.well-known/matrix/client {
        # If your sever_name here doesn't match your matrix homeserver URL
        # (e.g. hostname.com as server_name and matrix.hostname.com as homeserver URL)
        # add_header Access-Control-Allow-Origin '*';
        return 200 '{ "m.homeserver": { "base_url": "https://matrix.example.com" } }';
    }

    location /_matrix/identity {
        proxy_pass http://10.1.0.16:8090;
    }

    location /_matrix {
        proxy_pass http://10.1.0.15:8008;
    }

}
server {
    if ($host = matrix.example.com) {
        return 301 https://$host$request_uri;
    }

    listen 80;
    listen [::]:80;

    server_name matrix.example.com;
    return 404;
}

Sydent configuration (edited for example):


[DEFAULT]
server.name = matrix.example.com
log.path = 
log.level = INFO
pidfile.path = /data/sydent.pid
terms.path = 
address_lookup_limit = 10000
templates.path = res
brand.default = matrix.example.com
enable_v1_associations = true
delete_tokens_on_bind = true
ip.blacklist = 
ip.whitelist = 
db.file = /data/sydent.db
clientapi.http.bind_address = ::
clientapi.http.port = 8090
internalapi.http.bind_address = ::1
internalapi.http.port = 
replication.https.certfile = 
replication.https.cacert = 
replication.https.bind_address = ::
replication.https.port = 4434
obey_x_forwarded_for = False
federation.verifycerts = True
client_http_base = 
email.from = Sydent Validation <noreply@{hostname}>
email.subject = Your Validation Token
email.invite.subject = %(sender_display_name)s has invited you to chat
email.invite.subject_space = %(sender_display_name)s has invited you to a space
email.smtphost = localhost
email.smtpport = 25
email.smtpusername = 
email.smtppassword = 
email.hostname = 
email.tlsmode = 0
email.default_web_client_location = https://app.element.io
email.third_party_invite_username_obfuscate_characters = 3
email.third_party_invite_domain_obfuscate_characters = 3
bodytemplate = Your code is {token}
username = 
password = 
ed25519.signingkey = 

[general]
log.path = /data/log/sydent.log
server.name = matrix.example.com
ip.whitelist = 10.1.0.1

[db]

[http]
clientapi.http.bind_address = 10.1.0.16
clientapi.http.port = 8090
internalapi.http.bind_address = 10.1.0.16
internalapi.http.port = 8091

[email]
email.smtphost = mail.example.com
email.smtpport = 465
email.smtpusername = admin@example.com
email.smtppassword = changedforexample
email.tlsmode = 1

[sms]

[crypto]
ed25519.signingkey = changedforexample

Any help would be greatly appreciated. With the exception of Sydent the whole thing works like a charm.

Element is configured with base url https://matrix.example.com with homeserver example.com

Can you explain how this configuration is set?

When attempting to use Sydent as the identity server I get error:

Error fetching https://example.com/.well-known/matrix/server: Expecting value: line 1 column 1 (char 0)
WARNING - Unable to contact the Matrix homeserver (NoRouteError)

For completeness, this error comes from https://github.com/matrix-org/sydent/blob/main/sydent/http/servlets/registerservlet.py#L80-L83

Which makes sense as it should be attempting to make the connection to matrix.example.com - not example.com directly.

This doesn't sound right to me.

Sydent has been given nothing but the username @user:example.com. It needs to be told by example.com itself to contact matrix.example.com to actually reach your homeserver; it does this by trying to GET https://example.com/.well-known/matrix/server before making any federation requests. (There is a fallback mechanism to lookup an SRV record---see item 4 here---but my understanding is that .well-known responses are preferred and less painful to configure.)

Is the sydent container able to GET https://example.com/.well-known/matrix/server?

@DMRobertson - thank you for the thorough reply, much appreciated :)

The dendrite configuration is as follows; (relevant sections for brevity):

# Global Matrix configuration. This configuration applies to all components.
global:
  # The domain name of this homeserver.
  server_name: example.com

  # The server name to delegate server-server communications to, with optional port
  # e.g. localhost:443
  well_known_server_name: "matrix.example.com"

  # The server name to delegate client-server communications to, with optional port
  # e.g. localhost:443
  well_known_client_name: "matrix.example.com"

  # Lists of domains that the server will trust as identity servers to verify third
  # party identifiers such as phone numbers and email addresses.
  trusted_third_party_id_servers:
    - matrix.example.com
    # - matrix.org
    # - vector.im

The sydent server is unable to connect to example.com directly as there is no nginx configuration running for example.com - only for matrix.example.com. If I set up an nginx service for example.com it breaks compatibility with the publicly available example.com and www.example.com, as well as breaking the Element configuration for some reason that I am not yet aware of.

Element is configured as:

{
    "default_server_config": {
        "m.homeserver": {
            "base_url": "https://matrix.example.com",
            "server_name": "example.com"
        },
        "m.identity_server": {
            "base_url": "https://matrix.example.com"
        }
    },

I reckon that's why Element is able to handle users as @foo:example.com, being configured to contact matrix.example.com directly for data exchange.

I originally had sydent configured with server.name = example.com before changing it to matrix.example.com in hopes of having it target the matrix.example.com directly but it didn't. Thus I was wondering / hoping that Sydent offered a server name and base_url solution as element does.

To do some digging I deleted the Matrix database and reinitialised everything, and changed the server name to matrix.example.com in each component (dendrite, sydent, element). I do get connectivity (as well as now having to register users as @foo:matrix.example.com, which is a shame really as I really would prefer @foo:example.com) but I get a JSON error:

2023-03-07 17:43:55,556 - sydent.http.httpclient - 67 - WARNING - Error parsing JSON from matrix://matrix.example.com/_matrix/federation/v1/openid/userinfo?access_token=foo
2023-03-07 17:43:55,556 - sydent.http.servlets.registerservlet - 64 - WARNING - The Matrix homeserver returned invalid JSON

The android client of element accordingly refuses to use matrix.example.com as the identity service, stating that there are no terms of service available, followed by a posting of a java error. I guess I should look into that part as I realise that this error is not related to the original reason for having opened this issue.

The question remains - is it not - or dare I say should it not - be possible to register a server name and a base url for the server directly in sydent as is possible with element?

I originally had sydent configured with server.name = example.com before changing it to matrix.example.com in hopes of having it target the matrix.example.com directly but it didn't.

server.name tells Sydent where Sydent itself is being hosted; it doesn't correspond to any particular homeserver.

The question remains - is it not - or dare I say should it not - be possible to register a server name and a base url for the server directly in sydent as is possible with element?

No, because Sydent is designed to talk to multiple homeservers (as the matrix.org and vector.im Sydent deployments do). Element (and Matrix clients more generally) only ever talks to one homeserver at a time.

From your original post:

Is there a way to force Sydent to go to matrix.example.com instead of example.com ?

This situation—having a homeserver with domain example.com be served by a process on matrix.example.com—is called delegation and it's notoriously tricky to configure. There are some tips on https://matrix-org.github.io/synapse/latest/delegate.html for setting this up. (That page is from Synapse's docs, but most of the advice there is Synapse-agnostic.)

It sounds like the way your VPN is arranged is an obstacle for setting up delegation though:

If I set up an nginx service for example.com it breaks compatibility with the publicly available example.com and www.example.com, as well as breaking the Element configuration for some reason that I am not yet aware of.

With apologies for bluntness: this sounds like a reverse-proxy or VPN configuration problem that you'll need to solve for yourself, rather than a Matrix/Sydent problem. (Can nginx be configured to serve the .well-known/matrix response for example.com, and otherwise proxy through to the publicly available example.com?)

@DMRobertson Thank you for replying again, much appreciated.

Delegation: yes, I was aware of that as stated in my original and follow-up messages. With the exception of Sydent delegation works perfectly fine for Dendrite, Element-web and Element on Android (over VPN) as Dendrite can be configured to inform clients that the Matrix server and its server-server, server-client URLs are different.

VPN: impossible as the routing tests etc are all performed directly on the VPS, the failure to communicate occurs internally on the VPS.

Reverse-proxy; unlikely as the proxy works perfectly for complex communications for a large number of container sets such as Gitea, Woodpecker-CI and its agents, Vikunja, Penpot - and of course Dendrite and Element. I have in the interim set up an extra nginx service file for example.com (essentially a slightly edited copy of matrix.example.com) and fixed earlier issues. As with my earlier attempt communication now occurs, but again with complaints about json, but that seems to be a Dendrite issue, not a Sydent one. I'll have to look into that a little deeper I reckon.

It is unfortunate that Sydent does not support / the team will not adopt a more flexible approach to configuration for non-federated, internal Matrix deployment that are strictly deployed for LAN / Intranet use. It seems to me that a configuration mode where Sydent is running in a 'local' mode - where it is restricted from communicating with other identity servers, as well as having the option to individually configure a static URL for the Matrix server instance and the server name under which is operates should be technically possible.

I would explore that myself were it not that I'm not very familiar with the Python ecosystem beyond shell scripting and don't have time at the moment to dig more deeply into that. I'm more familiar with C, C++, Rust and Go.

Thanks for the pointer regarding Synapse delegation tweaking, I'll look into that and also have a look at what the Dendrite <-> Sydent json issue seems to be all about. If that doesn't work then I'll have to do without an identity server for the foreseeable future I guess.

Thanks for reading, helping and replying and thanks for the hard work on Matrix in general. It's amazing and exactly what the world needs to escape the yoke of proprietary horrors.

I'll close the issue now and keep digging.

Possibly what's confusing a lot here is that the identity server isn't designed to be designated for a single homeserver: it's a separate component for a reason. The identity server needs to be able to discover your homeserver (through delegation if necessary) and perform the usual signing key checks, like it would for any other homeserver.

I struggle to believe that it's not possible to set up either well-known or SRV delegation in your environment so that you can get these things to work, even if that would involve hardcoding a special entry in /etc/hosts and serving the well-known file on a special-purpose nginx route in some way. However your setup sounds quite complex so I can't just give you a direct solution.

If Sydent is really no good for your situation, you might be interested in https://github.com/ma1uta/ma1sd — which is not our project and I can't vouch for it — I've never used it myself — but it provides some functionality that particularly internal deployments typically miss out on.

@reivilibre thank you for reading and replying, much appreciated.

I did see a number of other identity server projects out in the wild, but most of them seem either retired or unmaintained.

I admit I'm new to Matrix and a bit limited in my understanding of the intricacies of the protocol and the architecture of its server components. I did not mean to offend with my previous comment regarding internal homeserver deployment support for Sydent; apologies if that is how it might have come across.

I'm not giving up yet, I'll just need to do some more digging. With the addition of the example.com nginx service file Sydent does communicate with the matrix.example.com instance, but it throws a number of json errors that I will need to look into. I suspect however that said errors are on the Dendrite side of the equation. I'll be a bit busy for the next few days, but I'll look deeper into it over the weekend and report back if I managed to get it to work.

Above all: thanks for reading and replying and all the hard work on the matrix project. The matrix community is truly a shining example of digital chivalry :))

matrix-org / sydent

Homeserver url issue #555