Matrix-nginx-proxy.service fails to start when using additional config blocks

[zorlaski]

Describe the bug A clear and concise description of what the bug is.

Using multiple nginx additional configuration blocks in the vars.yml file causes matrix-nginx-proxy.service to fail to start. removing these blocks allows the proxy to start. These two blocks were previously working and had been untouched until a recent update caused them to stop working. Service logs report the following error:

-- Journal begins at Wed 2023-07-19 13:45:19 EDT. --
Jul 24 08:21:32 BuicBase matrix-nginx-proxy[3181103]: 2023/07/24 12:21:32 [warn] 1#1: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-jitsi.conf:25
Jul 24 08:21:32 BuicBase matrix-nginx-proxy[3181103]: nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-jitsi.conf:25
Jul 24 08:21:32 BuicBase matrix-nginx-proxy[3181103]: 2023/07/24 12:21:32 [warn] 1#1: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/nginx-http.conf:16
Jul 24 08:21:32 BuicBase matrix-nginx-proxy[3181103]: nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/nginx-http.conf:16
Jul 24 08:21:32 BuicBase matrix-nginx-proxy[3181103]: 2023/07/24 12:21:32 [warn] 1#1: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/nginx-http.conf:17
Jul 24 08:21:32 BuicBase matrix-nginx-proxy[3181103]: nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/nginx-http.conf:17
Jul 24 08:21:32 BuicBase matrix-nginx-proxy[3181103]: 2023/07/24 12:21:32 [emerg] 1#1: unknown directive "ssl" in /etc/nginx/conf.d/nginx-http.conf:22
Jul 24 08:21:32 BuicBase matrix-nginx-proxy[3181103]: nginx: [emerg] unknown directive "ssl" in /etc/nginx/conf.d/nginx-http.conf:22
Jul 24 08:21:33 BuicBase systemd[1]: matrix-nginx-proxy.service: Main process exited, code=exited, status=1/FAILURE
Jul 24 08:21:33 BuicBase systemd[1]: matrix-nginx-proxy.service: Failed with result 'exit-code'.
Jul 24 08:22:03 BuicBase systemd[1]: matrix-nginx-proxy.service: Scheduled restart job, restart counter is at 3.
Jul 24 08:22:03 BuicBase systemd[1]: Stopped Matrix nginx-proxy server.
Jul 24 08:22:03 BuicBase systemd[1]: Starting Matrix nginx-proxy server...
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182229]: 4378e1fb40615bb27f02edf550addb1641f9575dd9ff7c0bdb4d26a34342d2a4
Jul 24 08:22:04 BuicBase systemd[1]: Started Matrix nginx-proxy server.
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: 10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: /docker-entrypoint.sh: Configuration complete; ready for start up
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: 2023/07/24 12:22:04 [warn] 1#1: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-base-domain.conf:22
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-base-domain.conf:22
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: 2023/07/24 12:22:04 [warn] 1#1: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-base-domain.conf:23
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-base-domain.conf:23
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: 2023/07/24 12:22:04 [warn] 1#1: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-client-element.conf:25
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-client-element.conf:25
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: 2023/07/24 12:22:04 [warn] 1#1: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-client-element.conf:26
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-client-element.conf:26
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: 2023/07/24 12:22:04 [warn] 1#1: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-domain.conf:26
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-domain.conf:26
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: 2023/07/24 12:22:04 [warn] 1#1: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-domain.conf:27
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-domain.conf:27
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: 2023/07/24 12:22:04 [warn] 1#1: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-domain.conf:115
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-domain.conf:115
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: 2023/07/24 12:22:04 [warn] 1#1: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-domain.conf:116
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-domain.conf:116
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: 2023/07/24 12:22:04 [warn] 1#1: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-grafana.conf:25
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-grafana.conf:25
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: 2023/07/24 12:22:04 [warn] 1#1: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-grafana.conf:26
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-grafana.conf:26
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: 2023/07/24 12:22:04 [warn] 1#1: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-jitsi.conf:24
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-jitsi.conf:24
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: 2023/07/24 12:22:04 [warn] 1#1: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-jitsi.conf:25
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/matrix-jitsi.conf:25
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: 2023/07/24 12:22:04 [warn] 1#1: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/nginx-http.conf:16
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/nginx-http.conf:16
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: 2023/07/24 12:22:04 [warn] 1#1: the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/nginx-http.conf:17
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/nginx-http.conf:17
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: 2023/07/24 12:22:04 [emerg] 1#1: unknown directive "ssl" in /etc/nginx/conf.d/nginx-http.conf:22
Jul 24 08:22:04 BuicBase matrix-nginx-proxy[3182248]: nginx: [emerg] unknown directive "ssl" in /etc/nginx/conf.d/nginx-http.conf:22
Jul 24 08:22:05 BuicBase systemd[1]: matrix-nginx-proxy.service: Main process exited, code=exited, status=1/FAILURE
Jul 24 08:22:05 BuicBase systemd[1]: matrix-nginx-proxy.service: Failed with result 'exit-code'.

To Reproduce My vars.yml file looks like this:

# The bare domain name which represents your Matrix identity.
# Matrix user ids for your server will be of the form (`@user:<matrix-domain>`).
#
# Note: this playbook does not touch the server referenced here.
# Installation happens on another server ("matrix.<matrix-domain>").
#
# If you've deployed using the wrong domain, you'll have to run the Uninstalling step,
# because you can't change the Domain after deployment.
#
# Example value: example.com
matrix_domain: buic.xyz

# The Matrix homeserver software to install.
# See `roles/matrix-base/defaults/main.yml` for valid options.
matrix_homeserver_implementation: synapse

# A secret used as a base, for generating various other secrets.
# You can put any string here, but generating a strong one is preferred (e.g. `pwgen -s 64 1`).
matrix_homeserver_generic_secret_key: 'xxxxxxxxxxxxxxxxxxxxxxx'

# This is something which is provided to Let's Encrypt when retrieving SSL certificates for domains.
#
# In case SSL renewal fails at some point, you'll also get an email notification there.
#
# If you decide to use another method for managing SSL certificates (different than the default Let's Encrypt),
# you won't be required to define this variable (see `docs/configuring-playbook-ssl-certificates.md`).
#
# Example value: someone@example.com
matrix_ssl_lets_encrypt_support_email: 'XXXXXXXXXX'

############################
# ELEMENT CONFIG OVERRIDES #
############################
matrix_client_element_integrations_ui_url: "https://dimension.XXX.XXX/element"
matrix_client_element_integrations_rest_url: "https://dimension.XXX.XXX/api/v1/scalar"
matrix_client_element_integrations_widgets_urls: ["https://dimension.XXX.XXX/widgets"]
matrix_client_element_integrations_jitsi_widget_url: "https://dimension.XXX.XXX/widgets/jitsi"

# Controls whether custom Element themes will be installed.
# When enabled, all themes found in the `matrix_client_element_themes_repository_url` repository
# will be installed and enabled automatically.
matrix_client_element_themes_enabled: true
matrix_client_element_themes_repository_url: https://github.com/aaronraimist/element-themes
# Controls the default theme
matrix_client_element_default_theme: 'dark'

#########
##JITSI##
#########

# A Postgres password to use for the superuser Postgres user (called `matrix` by default).
#
# The playbook creates additional Postgres users and databases (one for each enabled service)
# using this superuser account.
devture_postgres_connection_password: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
jitsi_enabled: true

# Run `bash inventory/scripts/jitsi-generate-passwords.sh` to generate these passwords,
# or define your own strong passwords manually.
jitsi_jicofo_auth_password: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
jitsi_jvb_auth_password: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
jitsi_jibri_recorder_password: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
jitsi_jibri_xmpp_password:  XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
jitsi_jvb_container_extra_arguments:
  - '--env "DOCKER_HOST_ADDRESS=192.168.1.132"'
jitsi_web_custom_config_extension: |
  config.enableLayerSuspension = true;

  config.disableAudioLevels = true;

  // Limit the number of video feeds forwarded to each client
  config.channelLastN = 10;

############
# GRAFANA #
############
prometheus_enabled: true

prometheus_node_exporter_enabled: true

grafana_enabled: true

grafana_anonymous_access: false

# This has no relation to your Matrix user id. It can be any username you'd like.
# Changing the username subsequently won't work.
grafana_default_admin_user: "XXXXX"

# Changing the password subsequently won't work.
grafana_default_admin_password: "XXXXXXXXX"

###########
# TRAEFIK #
###########

matrix_playbook_reverse_proxy_type: playbook-managed-nginx

devture_traefik_config_certificatesResolvers_acme_email: XXXXXXXXX

#########
# NGINX #
#########
matrix_nginx_proxy_access_log_enabled: true
matrix_nginx_proxy_base_domain_serving_enabled: true
matrix_ssl_additional_domains_to_obtain_certificates_for:
  - jellyfin.XXX.XXX
matrix_nginx_proxy_proxy_http_additional_server_configuration_blocks:
 - |
    server {
      listen {{ 8080 if matrix_nginx_proxy_enabled else 80 }};
      server_name jellyfin.XXX.XXX;
      return 301 https://jellyfin.XXX.XXX$request_uri;
    }
 - |
    server {
      listen {{ 8443 if matrix_nginx_proxy_enabled else 443 }} ssl http2;
      listen [::]:{{ 8443 if matrix_nginx_proxy_enabled else 443 }} ssl http2;
      server_name jellyfin.XXX.XXX;
      server_tokens off;
      root /dev/null;
      ssl on;
      ssl_certificate {{ matrix_ssl_config_dir_path }}/live/jellyfin.XXX.XXX/fullchain.pem;
      ssl_certificate_key {{ matrix_ssl_config_dir_path }}/live/jellyfin.XXX.XXX/privkey.pem;
      ssl_protocols {{ matrix_nginx_proxy_ssl_protocols }};
      ssl_prefer_server_ciphers on;
      ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
      location / {
          proxy_pass http://jellyfin:8096/;
          proxy_set_header Host $host;
         proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $scheme;
          add_header Front-End-Https on;
      }
     location /.well-known/acme-challenge {
         resolver 127.0.0.11 valid=5s;
         set $backend "matrix-certbot:8080";
         proxy_pass http://$backend;
     }

    }
#############
# DIMENSION #
#############

matrix_dimension_enabled: false
matrix_dimension_admins:
  - "@XXXXXXXXXXXXX"
matrix_dimension_access_token: XXXXXXXXXXXXXXXXXXXXXXXXXXX

############
# POSTGRES #
############
devture_postgres_process_extra_arguments: [
  "-c 'max_connections=200'",
  "-c 'shared_buffers=12GB'",
  "-c 'effective_cache_size=36GB'",
  "-c 'maintenance_work_mem=2GB'",
  "-c 'checkpoint_completion_target=0.9'",
  "-c 'wal_buffers=16MB'",
  "-c 'default_statistics_target=100'",
  "-c 'random_page_cost=4'",
  "-c 'effective_io_concurrency=2'",
  "-c 'work_mem=31457kB'",
  "-c 'min_wal_size=1GB'",
  "-c 'max_wal_size=4GB'",
  "-c 'max_worker_processes=20'",
  "-c 'max_parallel_workers_per_gather=4'",
  "-c 'max_parallel_workers=20'",
  "-c 'max_parallel_maintenance_workers=4'",
]

#########
# OTHER #
#########
matrix_synapse_admin_enabled: true
matrix_registration_enabled: true

# Generate a strong secret using: `pwgen -s 64 1`.
matrix_registration_admin_secret: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

matrix_synapse_configuration_extension_yaml: |
  retention:
    enabled: true
    purge_jobs:
      - longest_max_lifetime: 3d
        shortest_max_lifetime: 1d
        interval: 4h
    default_policy:
      min_lifetime: 1d
      max_lifetime: 60h
    allowed_lifetime_max: 3d

matrix_ssl_lets_encrypt_key_type: ecdsa
# A list of additional loggers to register in synapse.log.config.
# This list gets populated dynamically based on Synapse extensions that have been enabled.
# Contains definition objects like this: `{"name": "..", "level": "DEBUG"}
matrix_synapse_additional_loggers: [{"name": "synapse", "level": "INFO"}]

Expected behavior matrix-nginx-proxy.service should start normally, with the additional configured nginx proxy redirects taking effect. Matrix Server:

• OS: Debian 11
• Architecture: amd64

[spantaleev]

Are you previous adding server blocks into a server block?

I'm any case, it's better to add container labels to your Jellyfin container and let Traefik handle things instead of going into the dead matrix-nginx-proxy

[zorlaski]

Thanks for helping out. Looking here to setup traefik, but how can I add the additional traefik config blocks for the other servers I am using? Which variable should I modify? I dont know too much about traefik, an example of how to add our own service with the additional configuration option might be useful to alot of people.

[rltas]

The log you posted already tells you what the problem is: nginx: [emerg] unknown directive "ssl" in /etc/nginx/conf.d/nginx-http.conf:22 The ssl on directive has been obsolete since nginx 1.5.0 and finally removed in 1.25.1: http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl

You should still migrate to Traefik, once you get it it's easier and more logical. nginx-proxy will just go away at some point and then your config would break anyway. Traefik configuration (mostly) happens on the container side via labels, so that's more of a Jellyfin question. They even have Traefik related docs: https://jellyfin.org/docs/general/networking/traefik2/