Use external proxy in another container

[Samonitari]

Hi again!

Sorry for asking but I am stucked because of the lack of my knowledge. Title says it mostly, I am not successful in using the external nginx (located in a container)

My matrix.conf file from sites enabled looks like this:

server {
        listen 8000;
        server_name matrix.xyx.cloud;

        server_tokens off;
        root /dev/null;

                location /.well-known/acme-challenge/ {
                        alias /etc/letsencrypt/acme-challenge/.well-known/acme-challenge/;
                }

                location / {
                        return 301 https://$http_host$request_uri;
                }
}

server {
        listen 4430 ssl http2;
        listen [::]:4430 ssl http2;

        server_name matrix.xyx.cloud;

        server_tokens off;
        root /dev/null;

        ssl_certificate /certs/live/matrix.xyx.cloud/fullchain.pem;
        ssl_certificate_key /certs/live/matrix.xyx.cloud/privkey.pem;
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_prefer_server_ciphers on;
        ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";

        gzip on;
        gzip_types text/plain application/json;

        location /.well-known/matrix {
                root /matrix/static-files;
                expires 4h;
                default_type application/json;
                add_header Access-Control-Allow-Origin *;
        }

        location /_matrix/identity {
                proxy_pass http://127.0.0.1:8090;

                proxy_set_header Host $host;
                proxy_set_header X-Forwarded-For $remote_addr;
        }

        location /_matrix/client/r0/user_directory/search {
                proxy_pass http://127.0.0.1:8090;

                proxy_set_header Host $host;
                proxy_set_header X-Forwarded-For $remote_addr;
        }

        location /_matrix {
                proxy_pass http://127.0.0.1:8008;

                proxy_set_header Host $host;
                proxy_set_header X-Forwarded-For $remote_addr;

                client_body_buffer_size 25M;
                client_max_body_size 10M;
                proxy_max_temp_file_size 0;
        }

        location /_synapse/admin {
                proxy_pass http://127.0.0.1:8008;

                proxy_set_header Host $host;
                proxy_set_header X-Forwarded-For $remote_addr;

                client_body_buffer_size 25M;
                client_max_body_size 10M;
                proxy_max_temp_file_size 0;
        }

        location / {
                rewrite ^/$ /_matrix/static/ last;
        }

}

server {
        listen 8448 ssl http2;
        listen [::]:8448 ssl http2;

        server_name matrix.xyx.cloud;
        server_tokens off;

        root /dev/null;

        gzip on;
        gzip_types text/plain application/json;

        ssl_certificate /certs/live/matrix.xyx.cloud/fullchain.pem;
        ssl_certificate_key /certs/live/matrix.xyx.cloud/privkey.pem;
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_prefer_server_ciphers on;
        ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";

        location / {
                proxy_pass http://127.0.0.1:8048;

                proxy_set_header Host $host;
                proxy_set_header X-Forwarded-For $remote_addr;

                client_body_buffer_size 25M;
                client_max_body_size 30M;
                proxy_max_temp_file_size 0;
        }
}

I made few modifications to the autogenerated one:

• The 80 and 443 ports of my host are mapped to 8000 and 4430 (of the nginx container) respectively, straight substitution
• location /.well-known/acme-challege/ made to resembe the other sites
• Certificate location changed

Furthermore, a configured the playbook so that matrix_docker_network points to the same network that nginx's container uses. Checked with docker network inspect, and config works as intended

All i manage is bad gateway response from nginx when self-check hits TASK [matrix-synapse : Check Matrix Client API]

root and www-data users are added to matrix group. Where should I look?

[Samonitari]

I keep looking into it and learning, but no success so far.

However, I observed that even though I set matrix_ssl_retrieval_method to none, I still have these lines in /matrix/coturn/turnserver.conf:

cert=/matrix/ssl/config/live/matrix.meszaros-szegi.cloud/fullchain.pem
pkey=/matrix/ssl/config/live/matrix.meszaros-szegi.cloud/privkey.pem

I couldn't find an option to redirect it, symlinking to the separately managed files doesn't sound robust.

[spantaleev]

group_vars/matrix_servers defines this defaults regardless:

matrix_coturn_tls_enabled: true
matrix_coturn_tls_cert_path: "{{ matrix_ssl_config_dir_path }}/live/{{ matrix_server_fqn_matrix }}/fullchain.pem"
matrix_coturn_tls_key_path: "{{ matrix_ssl_config_dir_path }}/live/{{ matrix_server_fqn_matrix }}/privkey.pem"

You can either overwrite the paths with something else, or flip matrix_coturn_tls_enabled to false to disable TLS support. If you need to mount additional paths in the Coturn container, take a look at the matrix_coturn_container_additional_volumes setting (see roles/matrix-coturn/defaults/main.yml).

[Samonitari]

Thanks for the coturn settings.

I made some progress.

At least, .well-known hosting works now. I had like, min 2 problems with my setup:

• base domain's nginx conf file had a regex'd location, the pattern including anything starting with . containing a single deny all rule. I don't know how carddav and other well-known worked with that, maybe because those aren't proxies, but it doesn't matter...
• The most of the docker-composed containers of my site use www-data as user, but the nginx container used nginx, of cource. So no, I did not had the permissions on two levels.

The 8448 port should be published by the nginx conainer in my case, or not? I made it so, so at least curl localhost:8448 give me something, but external self-check still fails with bad gateway.

Here is the output of docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Ports}}"

CONTAINER ID        NAMES                     PORTS
e26b2a4382f2        matrix-mautrix-facebook
5b30193cd424        matrix-synapse            127.0.0.1:8008->8008/tcp, 8009/tcp, 127.0.0.1:8048->8048/tcp, 8448/tcp
64acdbebca43        matrix-postgres           5432/tcp
1701ea34b889        matrix-coturn             0.0.0.0:3478->3478/tcp, 0.0.0.0:3478->3478/udp, 0.0.0.0:5349->5349/udp, 0.0.0.0:5349->5349/tcp, 0.0.0.0:49152-49172->49152-49172/udp
d7150e25e5ec        matrix-mxisd              127.0.0.1:8090->8090/tcp
76c52ebfe873        matrix-mailer             8025/tcp
2f2590fac60c        nginx                     80/tcp, 0.0.0.0:8448->8448/tcp, 0.0.0.0:443->4430/tcp, 0.0.0.0:80->8000/tcp
44204bf4d8d1        create-account
9898b1aea19b        postfixadmin              8888/tcp
29d9d8e53cb7        welcome                   80/tcp
35606d91841a        eelomailserver            0.0.0.0:25->25/tcp, 0.0.0.0:110->110/tcp, 0.0.0.0:143->143/tcp, 0.0.0.0:587->587/tcp, 0.0.0.0:993->993/tcp, 0.0.0.0:995->995/tcp, 465/tcp, 0.0.0.0:4190->4190/tcp, 11334/tcp
aa9e3380690d        nextcloud                 9000/tcp
06980c195d30        automx                    80/tcp
4c40ed8f8c4f        mariadb                   3306/tcp
4c0eb60afd77        redis                     6379/tcp

[Samonitari]

As discussed in the support room, existing flag matrix_nginx_proxy_enabled is currenttly used for 3 'roles':

1. decide if matrix-nginx-proxy service/docker-image gets installed, as its name suggest.
2. what port to expose/publish of the containers, most importantly matrix-synapse
3. content of the config file matrix-synapse.conf for nginx setup:
   • add a resolver or not to the location entries
   • what ports to listen on the matrix.<domain_name>

To keep backwards compatibility solution should possibly keep this flag "as-is", including the effect of it of course.

Solution 1

• Keep the existing matrix_nginx_proxy_enabled flag, and use it for role:1
• Add a new flag, (e.g. matrix_proxy_in_container - nginx from name is intentionally missing!) which will be:
  • used for roles:[2,3], and consequenctly replace the existing one in the YAML conditionals affected
  • initialized to true if matrix_nginx_proxy_enabled is true
  • can be set to true in inventory/host_vars/matrix.<doamain_name>/vars.yml
  • false or uninitialized otherwise

There is a hidden "invalid" option with this solution that should be prohibied, and the initialization rules outlined above would take care of:

matrix_nginx_proxy_enabled: true
matrix_proxy_in_container: false

Solution 2

• Keep the existing matrix_nginx_proxy_enabled flag, use it for roles:[2,3]
• Add a new flag, (e.g. matrix_nginx_proxy_service_enabled) which will be:
  • used for role:1, and consequenctly replace the existing one in the YAML conditionals affected It needs less replacement as I see
  • initialized to true if matrix_nginx_proxy_enabled is true, uninitialized or false otherwise
  • can be set to false, which overrides even the previous point point occurs

This also has an "invalid" option out of the four possibilities:

matrix_nginx_proxy_enabled: false
matrix_proxy_service_enabled: true

Solution 3

• Keep the existing matrix_nginx_proxy_enabled flag, use it for all roles:[1,2,3]
• Make it tri state, e.g.: true|false|existing_container and
  • roles:[2,3] conditionals handle existing_container the same as true
  • role:1 handles existing_container the same as false

Solution: 1 is the most logical, out of these at least, but there could be a better one Solution: 2 needs the least source modification, but is a bit hacky, and the existing name matrix_nginx_proxy_enabled is not a good match that way (external container could be any reverse proxy) Solution: 3 is mainly here for a weak try on being detailed in making the suggestion. Name is also wrong here, but it is keeps backward compatibility. Should be renamed to be fair, like matrix_proxy: internal_nginx|external_container|external, and that way it is a good non-backward-compatible alternative

What do you think @spantaleev?

[spantaleev]

I'm kind of pulled toward Solution 1 - introducing a new flag which controls whether services expose their ports outside containers or not.

Instead of calling this new flag matrix_proxy_in_container, we could also call it matrix_containers_expose_ports. We can define it with a false default in matrix-base, and in terms of matrix_nginx_proxy_enabled in group_vars/matrix_servers:

matrix_containers_expose_ports: "{{ matrix_nginx_proxy_enabled }}"

We then need to replace most (all?) references to matrix_nginx_proxy_enabled in group_vars/matrix_servers (well, besides this one above) with matrix_containers_expose_ports.

---

In the matrix-nginx-proxy role, we probably have to introduce some new matrix_nginx_proxy_services_expose_ports variable as well, to affect the roles/matrix-nginx-proxy/templates/nginx/conf.d templates. Right now the proxy_pass value is affected based on matrix_nginx_proxy_enabled.

Actually, I'd redo this part. Perhaps it should be redone separately, before we address this issue even.

It currently sucks that nginx server configuration templates hardcode address like this:

{% if matrix_nginx_proxy_enabled %}
    {# Use the embedded DNS resolver in Docker containers to discover the service #}
    resolver 127.0.0.11 valid=5s;
    set $backend "matrix-dimension:8184";
    proxy_pass http://$backend;
{% else %}
    {# Generic configuration for use outside of our container setup #}
    proxy_pass http://127.0.0.1:8184;
{% endif %}

Why would matrix-nginx-proxy assume that Dimension is at matrix-dimension:8184 in one case and at http://127.0.0.1:8184 in the other? That's playbook-level knowledge.

I think we can define a variable (e.g. matrix_nginx_proxy_proxy_dimension_proxy_pass_address), some default value, and then wire things correctly from group_vars/matrix_servers. Since we need to somehow know whether we should do the resolver 127.0.0.11 valid=5s; and set $backend hacks, we may need one more variable there (e.g. matrix_nginx_proxy_proxy_dimension_proxy_pass_with_docker_resolver).

group_vars/matrix_servers could then define things like this then:

matrix_nginx_proxy_proxy_dimension_proxy_pass_address: "{{ '127.0.0.1:8184' if matrix_nginx_proxy_services_expose_ports else 'matrix-dimension:8184' }}"

matrix_nginx_proxy_proxy_dimension_proxy_pass_with_docker_resolver: "{{ not matrix_nginx_proxy_services_expose_ports }}"

---

I'm also thinking of something else.. matrix_containers_expose_ports may be more helpful as a tri-state variable. Example matrix_containers_port_exposure_mode: "none|loopback|everywhere".

The default would be "none". "loopback" is what currently happens when the nginx proxy is disabled. "everywhere" is some new value that would be useful to people who want to reverse-proxy from another VM on the network - they'd benefit from being able to easily expose services on 0.0.0.0, instead of 127.0.0.1.

Hmm.. I guess this would really complicate things though..

---

Hehe.. I think I may have went too far.. This is a hairy issue..

[Samonitari]

Thanks, for the detailed answer!

I am not in a hurry, but if you aren't either, I'll give this a try. Be warned though, I haven't dig into ansible much before.

Part three about the tri-state matrix_containers_expose_ports seems legit. I'll add this at a second part, but that would create a point from where final solution would be non-backward-compatible. So it should be merged only when "final", in one fell swoop

[Olawale1]

Hi @Samonitari

I encountered a very similar issue. Not sure how you resolved yours but I got around it by inspecting the Docker Network (to get container ip address) and replacing 127.0.0.1 in the nginx conf file with the network ip address of the matrix container

Cheers

[Olawale1]

Updating this just in case someone else finds it helpful...

I switched to using container names instead of ip address. I found this to be less fragile as ip addresses could change after rebooting the host machine. Containers can be located with their names provided they're on the same network