'404 page not found' after installation and migration old data

[pscriptos]

Playbook Configuration:

My vars.yml file looks like this:

#---
# The bare domain name which represents your Matrix identity.
# Matrix user ids for your server will be of the form (`@user:<matrix-domain>`).
#
# Note: this playbook does not touch the server referenced here.
# Installation happens on another server ("matrix.<matrix-domain>").
#
# If you've deployed using the wrong domain, you'll have to run the Uninstalling step,
# because you can't change the Domain after deployment.
#
# Example value: example.com
matrix_domain: media-techport.de

# The Matrix homeserver software to install.
# See:
#  - `roles/custom/matrix-base/defaults/main.yml` for valid options
# - the `docs/configuring-playbook-IMPLEMENTATION_NAME.md` documentation page, if one is available for your implementation choice
matrix_homeserver_implementation: synapse

# A secret used as a base, for generating various other secrets.
# You can put any string here, but generating a strong one is preferred (e.g. `pwgen -s 64 1`).
matrix_homeserver_generic_secret_key: '$censored'

# By default, the playbook manages its own Traefik (https://doc.traefik.io/traefik/) reverse-proxy server.
# It will retrieve SSL certificates for you on-demand and forward requests to all other components.
# For alternatives, see `docs/configuring-playbook-own-webserver.md`.
#matrix_playbook_reverse_proxy_type: playbook-managed-traefik

### use my own ReverseProxyServer
matrix_playbook_reverse_proxy_type: playbook-managed-traefik

# Ensure that public urls use https
matrix_playbook_ssl_enabled: true

# Disable the web-secure (port 443) endpoint, which also disables SSL certificate retrieval.
# This has the side-effect of also automatically disabling TLS for the matrix-federation entrypoint
# (by toggling `matrix_federation_traefik_entrypoint_tls`).
devture_traefik_config_entrypoint_web_secure_enabled: false

# If your reverse-proxy runs on another machine, consider using `0.0.0.0:81`, just `81` or `SOME_IP_ADDRESS_OF_THIS_MACHINE:81`
devture_traefik_container_web_host_bind_port: '0.0.0.0:81'

# We bind to `127.0.0.1` by default (see above), so trusting `X-Forwarded-*` headers from
# a reverse-proxy running on the local machine is safe enough.
# If you're publishing the port (`devture_traefik_container_web_host_bind_port` above) to a public network interface:
# - remove the `devture_traefik_config_entrypoint_web_forwardedHeaders_insecure` variable definition below
# - uncomment and adjust the `devture_traefik_config_entrypoint_web_forwardedHeaders_trustedIPs` line below
#devture_traefik_config_entrypoint_web_forwardedHeaders_insecure: true
devture_traefik_config_entrypoint_web_forwardedHeaders_trustedIPs: ['10.0.1.101']

# Expose the federation entrypoint on a custom port (other than port 8448, which is normally used publicly).
#
# We bind to `127.0.0.1` by default (see above), so trusting `X-Forwarded-*` headers from
# a reverse-proxy running on the local machine is safe enough.
#
# If your reverse-proxy runs on another machine, consider:
# - using `0.0.0.0:8449`, just `8449` or `SOME_IP_ADDRESS_OF_THIS_MACHINE:8449` below
# - adjusting `matrix_playbook_public_matrix_federation_api_traefik_entrypoint_config_custom` (below) - removing `insecure: true` and enabling/configuring `trustedIPs`
matrix_playbook_public_matrix_federation_api_traefik_entrypoint_host_bind_port: '0.0.0.0:8449'

# Depending on the value of `matrix_playbook_public_matrix_federation_api_traefik_entrypoint_host_bind_port` above,
# this may need to be reconfigured. See the comments above.
matrix_playbook_public_matrix_federation_api_traefik_entrypoint_config_custom:
  forwardedHeaders:
#    insecure: true
    trustedIPs: ['10.0.1.101']

# This is something which is provided to Let's Encrypt when retrieving SSL certificates for domains.
#
# In case SSL renewal fails at some point, you'll also get an email notification there.
#
# If you decide to use another method for managing SSL certificates (different than the default Let's Encrypt),
# you won't be required to define this variable (see `docs/configuring-playbook-ssl-certificates.md`).
#
# Example value: someone@example.com
devture_traefik_config_certificatesResolvers_acme_email: '$censored'

# A Postgres password to use for the superuser Postgres user (called `matrix` by default).
#
# The playbook creates additional Postgres users and databases (one for each enabled service)
# using this superuser account.
devture_postgres_connection_password: '$censored'

# By default, we configure Coturn's external IP address using the value specified for `ansible_host` in your `inventory/hosts` file.
# If this value is an external IP address, you can skip this section.
#
# If `ansible_host` is not the server's external IP address, you have 2 choices:
# 1. Uncomment the line below, to allow IP address auto-detection to happen (more on this below)
# 2. Uncomment and adjust the line below to specify an IP address manually
#
# By default, auto-detection will be attempted using the `https://ifconfig.co/json` API.
# Default values for this are specified in `matrix_coturn_turn_external_ip_address_auto_detection_*` variables in the Coturn role
# (see `roles/custom/matrix-coturn/defaults/main.yml`).
#
# If your server has multiple IP addresses, you may define them in another variable which allows a list of addresses.
# Example: `matrix_coturn_turn_external_ip_addresses: ['1.2.3.4', '4.5.6.7']`
#
# matrix_coturn_turn_external_ip_address: ''

### Jitsi
jitsi_enabled: true

# Uncomment and adjust if you need to use another hostname
# jitsi_hostname: "jitsi.{{ matrix_domain }}"

# Uncomment and possible adjust if you'd like to host under a subpath
# jitsi_path_prefix: /jitsi

### custom email-services
exim_relay_sender_address: "$censored"
exim_relay_relay_use: true
exim_relay_relay_host_name: "mail.your-server.de"
exim_relay_relay_host_port: 587
exim_relay_relay_auth: true
exim_relay_relay_auth_username: "$censored"
exim_relay_relay_auth_password: "$censored"

### mautrix-whatsapp
matrix_mautrix_whatsapp_enabled: true
matrix_mautrix_whatsapp_bridge_relay_enabled: true
matrix_mautrix_whatsapp_bridge_relay_admin_only: false

### Shared Secret Auth-Passwortanbietermoduls
matrix_synapse_ext_password_provider_shared_secret_auth_enabled: true
matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret: $censored

### Mautrix Telegram
matrix_mautrix_telegram_enabled: true
matrix_mautrix_telegram_api_id: $censored
matrix_mautrix_telegram_api_hash: $censored

### Bridges Verschlüsselung
matrix_bridges_encryption_enabled: true

Matrix Server:

• OS: Ubuntu 22.04
• Architecture: amd64

Ansible: I am running Ansible on an LXC container. This LXC has only been created for Ansible. This means that nothing else is installed on the container. I also installed Matrix from here. Matrix was installed on a different server. Ansible Version: ansible [core 2.16.4]

Problem description:

I have installed Matrix via Ansible (IP:10.0.1.102) on the Matrix server (IP: 10.0.1.110). I decided to let Traefik do its work, but in such a way that I can use my ReverseProxy in the network. You can see this quite well in the vars.yml. The dockers all start and I can also see the status of the dockers, but when I call the matrix server, I get the message "404 page not found". It doesn't matter whether I try to call the service internally via HTTP/S or externally via my DNS record.

Special features: I have migrated my data. This means that I have migrated my Postgresql database and I have also migrated my media files. This worked well after some initial difficulties.

I also have a ProxyReverse server on the network. It is a CloudPanel. In terms of configuration, it is almost the same as an NGINX server. I guess that the ReverseProxy is not a problem here, because I also get the error when I try to call the service internally. Of course, I can't say for sure.

My installation is behind a firewall (OPNsense), behind which I currently still have an old, classic OnPrem installation of Matrix running. This runs on the same DNS. Of course, I switched off the "old" servers during and after the installation.

My hosts file looks like this:

[matrix_servers]
matrix.media-techport.de ansible_host=10.0.1.110 ansible_ssh_user=root
matrix_coturn_turn_external_ip_address: "$censored"

I have discovered the following. Maybe this will help.

➜  ~ systemctl status matrix-synapse.service
● matrix-synapse.service - Synapse server
     Loaded: loaded (/etc/systemd/system/matrix-synapse.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-03-07 19:30:13 CET; 2h 48min ago
   Main PID: 1020 (docker)
      Tasks: 9 (limit: 154380)
     Memory: 8.7M
        CPU: 213ms
     CGroup: /system.slice/matrix-synapse.service
             └─1020 docker start --attach matrix-synapse

Mär 07 20:36:50 sv10-matrix matrix-synapse[1020]: 2024-03-07 19:36:50,984 - synapse.federation.transport.server._base - 319 - WARNING - PUT-1987 - authenticate_request failed: 401: Destination mismatch in auth header
Mär 07 20:38:12 sv10-matrix matrix-synapse[1020]: 2024-03-07 19:38:12,806 - synapse.federation.transport.server._base - 319 - WARNING - PUT-2055 - authenticate_request failed: 401: Destination mismatch in auth header
Mär 07 20:38:36 sv10-matrix matrix-synapse[1020]: 2024-03-07 19:38:36,841 - synapse.federation.transport.server._base - 319 - WARNING - PUT-2072 - authenticate_request failed: 401: Destination mismatch in auth header
Mär 07 20:39:26 sv10-matrix matrix-synapse[1020]: 2024-03-07 19:39:26,901 - synapse.federation.transport.server._base - 319 - WARNING - PUT-2113 - authenticate_request failed: 401: Destination mismatch in auth header
Mär 07 20:40:08 sv10-matrix matrix-synapse[1020]: 2024-03-07 19:40:08,685 - synapse.federation.transport.server._base - 319 - WARNING - PUT-2150 - authenticate_request failed: 401: Destination mismatch in auth header
Mär 07 20:40:32 sv10-matrix matrix-synapse[1020]: 2024-03-07 19:40:32,069 - synapse.federation.transport.server._base - 319 - WARNING - PUT-2168 - authenticate_request failed: 401: Destination mismatch in auth header
Mär 07 20:41:07 sv10-matrix matrix-synapse[1020]: 2024-03-07 19:41:07,611 - synapse.federation.transport.server._base - 319 - WARNING - PUT-2198 - authenticate_request failed: 401: Destination mismatch in auth header
Mär 07 20:43:25 sv10-matrix matrix-synapse[1020]: 2024-03-07 19:43:25,155 - synapse.federation.transport.server._base - 319 - WARNING - PUT-2321 - authenticate_request failed: 401: Destination mismatch in auth header
Mär 07 20:44:38 sv10-matrix matrix-synapse[1020]: 2024-03-07 19:44:38,431 - synapse.federation.transport.server._base - 319 - WARNING - PUT-2383 - authenticate_request failed: 401: Destination mismatch in auth header
Mär 07 20:45:43 sv10-matrix matrix-synapse[1020]: 2024-03-07 19:45:43,845 - synapse.federation.transport.server._base - 319 - WARNING - PUT-2441 - authenticate_request failed: 401: Destination mismatch in auth header
➜ 

Client (please complete the following information):

• Device: Computer
• OS: Windows 10 x64
• Browser Chrome 122.0.6261.111, Firefox 123.0 (64-Bit)

I hope I have given you enough information. If there is anything else you would like me to provide, please let me know.

Thank you. Best regards, Patrick

[spantaleev]

Your reverse proxy is likely not sending a Host HTTP header or is sending an unexpected value.

The request seems to be reaching Traefik, but it responding with "404 not found" is an indication that you're either not specifying a Host or you're specifying something that it doesn't recognize.

You can refeer to tour nginx example reverse-proxy configuration: https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/7c1e5df3e79589c1288f2c5d0d8f406072866007/examples/reverse-proxies/nginx/matrix.conf#L18

[pscriptos]

Good evening,

Thank you very much for your feedback. So far I have limited myself to editing the two files. Do I have to carry out all other configurations directly on the matrix server or am I wrong? If so, where can I start? Because I have specified in the vars.yml that I use my own reverse proxy and I have specified an IP address. Can I change the configurations on the matrix server directly without them being undone, for example, if I make further configurations on the Ansible side?

Thank you for your feedback and have a nice weekend, Patrick

[spantaleev]

Alll your modifications go to inventory/hosts or inventory/matrix.DOMAIN/vars.yml.

The matrix.conf file I've linked to above is an example file that you can copy into your nginx configuration. You're not really using regular nginx though, so it doesn't really apply to you. That said, it can serve as inspiration.

I'd say there's nothing left for you to do on the Ansible side. Services should be configured correctly. You just need to fix your CloudPanel reverse-proxy server to send the appropriate Host header when sending requests to the Traefik server.

[pscriptos]

Another question:

Shouldn't I also be able to call the Matrix Server internally? For example; http://10.0.1.110:8449.

Here I get the same error message as shown above. if I call it internally, I do not go via the ReverseProxy but directly to the server. why do I get the same error message 404 page not found?

[spantaleev]

Calling via the IP does not pass a Host header, so it won't work.

If you modify your hosts file (/etc/hosts on UNIX, C:\Windows\something\something\hosts on Windws) and hardcode matrix.DOMAIN to go to 10.0.1.110, you may be able to open http://matrix.DOMAIN:PORT and reach it locally.

[pscriptos]

I have the same behaviour when I do it the way you just described:

I think there is something wrong with the configuration. I have certainly done something wrong, but what have I done wrong? :/

[ZhenyaPav]

Having the same issue. Did you manage to fix it?

[ZhenyaPav]

I am using nginx with the config similar to what @spantaleev linked above, but when I try to connect through that reverse proxy, I get 504 Gate Time-Out. When I try to use curl, I get 404. In my case, the matrix server and the NGINx are on separate machines in a VPN. I have added '10.10.0.1' (the IP of the reverse proxy) to devture_traefik_config_entrypoint_web_forwardedHeaders_trustedIPs, and NGINx routes its requests to 10.10.0.3 (the matrix server)

EDIT: I am able to get through with curl --header 'Host: matrix.domain.tld' http://10.10.0.3:8449/_matrix/federation/v1/version, but it only works from the matrix server, not from the reverse proxy machine.

journalctl --since "now" -xfeu matrix-traefik.service does not show any logs when making a request from the proxy machine.

EDIT2: My issue was caused by the (initial) lack of proxy_set_header Host $host; in my NGINx config, and later by incorrect firewall settings (the ports were not open)

[jmtsantos]

fixed for me by adding matrix_playbook_reverse_proxy_traefik_middleware_compression_enabled: false