Add this behind an Nginx Reverse Proxy (different server)?

[enoch85]

Hi,

Gave it 30 minutes to try different configs to get this to work behind an Nginx Reverse Proxy, but I failed.

So, I'm wondering how this would be achieved?

Thanks!

[whisperdancer]

I'm unsure if this is correct. As there is a lack of proper documentation, I got to this using trial & error. Eventually, this worked for me:

On reverse proxy, add:

location /push/ { proxy_pass http://ip_of_nc_server:7867/; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "Upgrade"; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; }

If your nc server is separate, ensure the port is open: sudo ufw allow 7867/tcp

Then also add the location entry in the vhost on your nc server using localhost. Same as above except for this line, which uses localhost instead of the ip of your nc server.

proxy_pass http://127.0.0.1:7867/;

Note, self-test works. Metrics shows connections. Debug log shows tons of activity, yet no clients are receiving notifications.

[enoch85]

Thanks! Not tested, but looks promising!

Anyone else that can confirm?

[Whissi]

Note:

proxy_set_header Upgrade $http_upgrade;

will have zero effect because $http_upgrade is not a global nginx variable. You find it in so many tutorials but all of them are copying stuff without understanding/reading to the end.

tl;dr You will also need

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

Read https://nginx.org/en/docs/http/websocket.html for more details.

[enoch85]

So to recap:

On Nginx

# Nextcloud Notify daemon

location /push/ {
proxy_pass http://ip_of_nc_server:7867/;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }
}

On the backend:

proxy_pass http://127.0.0.1:7867/;

Correct?

[Whissi]

What do you mean with "backend"? You add the location block for /push to the server block of your nextcloud instance. Note that the map block must be in the http block, not the server block like the location. Are you talking about using Apache for example for Nextcloud but you put an additional nginx instance in front of it as reverse proxy?

[enoch85]

Are you talking about using Apache for example for Nextcloud but you put an additional nginx instance in front of it as reverse proxy?

Exactly!

Internet --> 80|443 --> Nginx Reverse Proxy --> 80|443 --> Apache2 (Nextcloud)

[chrissi55]

Note:

proxy_set_header Upgrade $http_upgrade;

will have zero effect ....

... You will also need

map $http_upgrade $connection_upgrade {
  default upgrade;
  ''      close;
}

Read https://nginx.org/en/docs/http/websocket.html for more details.

In my reverse proxy (nginx) on a different server, than the nextcloud server i put the following lines in

http    {
        map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
        } 

server {
        listen *:80;
        server_name MY-FQDN;
        server_name_in_redirect off;
# Redirect non-https traffic to https
        if ($scheme != "https") {
                return 301 https://$host$request_uri;
        }
}

server {
        listen *:443 ssl;
       .....
       ......

After sudo nginx -t i get

nginx: [emerg] "http" directive is not allowed here in /etc/nginx/conf.d/cloud.conf:1
nginx: configuration file /etc/nginx/nginx.conf test failed

Made a mistake, put it above the ../conf.d/nc.config and not in the ../nginx/nginx.conf file in the http block. After changing nginx -t went ok.

In my nextcloud server i tried sudo -u www-data php /var/www/nextcloud/occ notify_push:setup my.domain

and got

✓ redis is configured
🗴 push server is not receiving redis messages (received 123863549, got 0)

Using sudo -u www-data php /var/www/nextcloud/occ notify_push:setup my.domain/push

delivers

✓ redis is configured
🗴 can't connect to push server: Server error: `GET https://my.domain/push/test/cookie` resulted in a `502 Bad Gateway` response:
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>ngin (truncated...)

Don't know how to solve this ...

[Whissi]

@enoch85 : First things first: Let's recap what WebSockets are. From Wikipedia:

WebSocket is a computer communications protocol, providing full-duplex communication channels over a single TCP connection. The WebSocket protocol was standardized by the IETF as RFC 6455 in 2011, and the WebSocket API in Web IDL is being standardized by the W3C. WebSocket is a different protocol from HTTP. Both protocols are located at layer 7 in the OSI model and, as such, depend on TCP at layer 4. Although they are different, RFC 6455 states that WebSocket "is designed to work over HTTP ports...

So you have two options:

• If you want to make it fully transparent, create a fully working Apache configuration like you wouldn't use a reverse proxy. I.e. tell Apache to do web sockets for /push like described in notify_push documentation. However, because WebSockets aren't HTTP requests, you need to tell the edge reverse proxy, nginx in your example, to do something like

  location /push {
    include proxy_params;
  
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
  
    ...
  
    proxy_pass http://your-apache-instance/;
  }

  I.e. proxy_set_header Connection "Upgrade"; is important setting on the edge reverse proxy because without that the edge reverse proxy would try to create the web socket connection to the apache server which you don't want.

• If the reverse proxy can access notify_push server directly you can tell edge reverse proxy to do that instead of proxying WebSocket connection. You will basically follow the notify_push documentation for nginx.

Keep in mind: When using $http_upgrade variable, you always need to set it. For example via shown map block above. If you don't do that you would always send empty Upgrade header because like said, $http_upgrade is not a default nginx variable and therefore empty if not defined.

@chrissi55 You need to check https://my.domain/push configuration. The proxy_pass setting is either wrong or set proxy destination isn't work. nginx error log will have more information.

[chrissi55]

@chrissi55 You need to check https://my.domain/push configuration. The proxy_pass setting is either wrong or set proxy destination isn't work. nginx error log will have more information.

In my error log on NC Server i see these entries

2021/04/11 09:13:08 [error] 88645#88645: *15844 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 127.0.0.1, server: my.domain, request: "GET /push/test/cookie HTTP/1.1", upstream: "https://127.0.0.1:7867/test/cookie", host: "my.domain"
2021/04/11 09:17:17 [error] 88645#88645: *15983 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 127.0.0.1, server: my.domain, request: "GET /push/test/cookie HTTP/1.1", upstream: "https://127.0.0.1:7867/test/cookie", host: "my.domain"
2021/04/11 09:22:27 [error] 161319#161319: *1 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 127.0.0.1, server: my.domain, request: "GET /push/test/cookie HTTP/1.1", upstream: "https://127.0.0.1:7867/test/cookie", host: "my.domain"
2021/04/11 09:27:25 [error] 161319#161319: *161 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 127.0.0.1, server: my.domain, request: "GET /push/test/cookie HTTP/1.1", upstream: "https://127.0.0.1:7867/test/cookie", host: "my.domain"
2021/04/11 09:53:04 [error] 161319#161319: *962 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 127.0.0.1, server: my.domain, request: "GET /push/test/cookie HTTP/1.1", upstream: "https://127.0.0.1:7867/test/cookie", host: "my.domain"
2021/04/11 09:55:11 [error] 1118#1118: *52 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 127.0.0.1, server: my.domain, request: "GET /push/test/cookie HTTP/1.1", upstream: "https://127.0.0.1:7867/test/cookie", host: "my.domain"
2021/04/11 09:55:15 [error] 1118#1118: *52 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 127.0.0.1, server: my.domain, request: "GET /push/test/cookie HTTP/1.1", upstream: "https://127.0.0.1:7867/test/cookie", host: "my.domain"
2021/04/11 09:59:51 [error] 1118#1118: *152 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 127.0.0.1, server: my.domain, request: "GET /push/test/cookie HTTP/1.1", upstream: "https://127.0.0.1:7867/test/cookie", host: "my.domain"
2021/04/11 09:59:51 [error] 1118#1118: *152 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 127.0.0.1, server: my.domain, request: "GET /push/test/cookie HTTP/1.1", upstream: "https://127.0.0.1:7867/test/cookie", host: "my.domain"

how can i shut down SSL to communicate with push because it's between proxy and NC Server isn't it?

[Whissi]

notify_push end point doesn't support TLS. Set your proxy_pass directive to http://127.0.0.1...

[chrissi55]

notify_push end point doesn't support TLS. Set your proxy_pass directive to http://127.0.0.1...

Thanks that did it for me.

On the reverse proxy i had to modify my /etc/nginx/conf.d/cloud.conf as follows

 location /push/ {
        proxy_pass      http://$upstream:7876;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host                   $host;
        proxy_set_header X-Forwarded-Host       $host;
        proxy_set_header X-Forwarded-Proto      $scheme;
        proxy_set_header X-Real-IP              $remote_addr;
        proxy_set_header X-Forwarded-For        $proxy_add_x_forwarded_for;
        }

$upstream = LOCAL-IP-OF-THE-NC-SERVER added

proxy_set_header X-Forwarded-Host       $host;
proxy_set_header X-Forwarded-Proto      $scheme;

as well

With the tip above map $http_upgrade $connection_upgrade i putted into my http conf of the NC-SERVER it is working now.

[enoch85]

OK, I'm starting to feel stupid now, but I can't seem to fix this...

On the Apache2 Backend server

root@cloud:/etc/apache2/sites-available# curl -v https://cloud.danielhansson.nu
*   Trying 127.0.1.1:443...
* TCP_NODELAY set
* Connected to cloud.danielhansson.nu (127.0.1.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

root@cloud:/etc/apache2/sites-available# nextcloud_occ notify_push:setup https://192.168.11.111/push
✓ redis is configured
✓ push server is receiving redis messages
✓ push server can load mount info from database
🗴 push server can't connect to the Nextcloud server
  error sending request for url (https://cloud.danielhansson.nu/index.php/apps/notify_push/test/cookie): error trying to connect: received corrupt message: error trying to connect: received corrupt message: received corrupt message

root@cloud:/etc/apache2/sites-available# nextcloud_occ notify_push:setup https://cloud.danielhansson.nu/push
✓ redis is configured
🗴 can't connect to push server: cURL error 35: error:1408F10B:SSL routines:ssl3_get_record:wrong version number (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://cloud.danielhansson.nu/push/test/cookie

    ProxyPass /push/ws ws://192.168.11.111:7867/ws
    ProxyPass /push/ http://192.168.11.111:7867/
    ProxyPassReverse /push/ http://192.168.11.111:7867/

    # Intermediate configuration
    SSLEngine               on
    SSLCompression          off
    SSLProtocol             -all +TLSv1.2 +TLSv1.3
    SSLCipherSuite          ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AE>
    SSLHonorCipherOrder     off
    SSLSessionTickets       off
    ServerSignature         off

I also have the trusted proxies stuff in the config already:

  'trusted_proxies' => 
  array (
    0 => '192.168.2.201',
    1 => '192.168.11.111', (added just for testing)
  ),
  'forwarded_for_headers' => 
  array (
    0 => 'HTTP_X_FORWARDED',
    1 => 'HTTP_FORWARDED_FOR',
  ),

On the Nginx Frontend

nginx.conf

.......
http {

        map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
        }

.......

        ssl_early_data off;
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256$
        ssl_prefer_server_ciphers on;
        ssl_ecdh_curve secp384r1;
        #ssl_session_cache shared:SSL:10m;
        #ssl_session_timeout 1h;
        #ssl_buffer_size 4k;

.......

nc-config.conf

        location = /push/ {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host                   $host;
        proxy_set_header X-Forwarded-Host       $host;
        proxy_set_header X-Forwarded-Proto      $scheme;
        proxy_set_header X-Real-IP              $remote_addr;
        proxy_set_header X-Forwarded-For        $proxy_add_x_forwarded_for;
        proxy_pass      http://$upstream:7876;
        }

[solracsf]

Note:

proxy_set_header Upgrade $http_upgrade;

will have zero effect because $http_upgrade is not a global nginx variable. You find it in so many tutorials but all of them are copying stuff without understanding/reading to the end.

tl;dr You will also need

map $http_upgrade $connection_upgrade {
  default upgrade;
  ''      close;
}

Read https://nginx.org/en/docs/http/websocket.html for more details.

This is not true.

$http_upgrade is a very valid header. Nginx uses the $http_NAME to read the header (here, Upgrade, in lower case). You can pass upgrade or any valid header value, like $http_cookie or $http_forwarded

https://nginx.org/en/docs/http/ngx_http_core_module.html#varhttp

When using

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

You're mapping $http_upgrade to $connection_upgrade, nothing else, so you can use $connection_upgrade later, like :

proxy_set_header Connection $connection_upgrade;

If you don't reuse $connection_upgrade later, your map is useless.

[thstyl2000]

Using the block in the readme only in the reverse proxy configuration for nextcloud, changing 127.0.0.1 to ip_of_nextcloud_instance connects, after running the binary with --bind 127.0.0.1 option.

[enoch85]

Given all the suggestions here, I'll close this.