Replication error behind Nginx

[shivam-880]

So I have deployed my app on a server behind nginx reverse proxy server with configurations which look like this:

upstream myapp {
    server localhost:3000;
    keepalive 16;
}

upstream couchdb {
    server localhost:6985;
    keepalive 16;
}

server {
    server_name myapp.shivamkapoor.com;

    location ^~ / {
        proxy_pass http://myapp;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/www.shivamkapoor.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/www.shivamkapoor.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

server {
    listen 6984 ssl;
    server_name myapp.shivamkapoor.com;

    ssl_certificate /opt/couchdb/etc/cert/fullchain.pem;
    ssl_certificate_key /opt/couchdb/etc/cert/privkey.pem;

    location / {
        proxy_pass https://couchdb;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    }
}

I am facing replication error on the browser:

The nignx error logs at /var/log/nginx/error.log look like this:

2024/04/26 19:44:25 [error] 8531#8531: *17 upstream timed out (110: Connection timed out) while reading upstream, 
client: 90.15.235.124, 
server: myapp.shivamkapoor.com, 
request: "GET /contribution_graph_106566519089485180314/_changes?style=all_docs&feed=longpoll&since=now&include_docs=true&heartbeat=60000&limit=20&seq_interval=20 HTTP/1.1", 
upstream: "https://127.0.0.1:6985/contribution_graph_106566519089485180314/_changes?style=all_docs&feed=longpoll&since=now&include_docs=true&heartbeat=60000&limit=20&seq_interval=20", 
host: "myapp.shivamkapoor.com:6984", 
referrer: "https://myapp.shivamkapoor.com/"

Thought may its an issue with the couchdb but following command return Ok:

# curl -k -I -X GET https://localhost:6985/contribution_graph_106566519089485180314/ -H "Authorization: Basic YWRtaW46YWRtaW4="
HTTP/1.1 200 OK
Cache-Control: must-revalidate
Content-Length: 514
Content-Type: application/json
Date: Fri, 26 Apr 2024 20:08:17 GMT
Server: CouchDB/3.3.3 (Erlang OTP/24)
X-Couch-Request-ID: 45e0360465
X-CouchDB-Body-Time: 0

# curl -k -I -X GET https://127.0.0.1:6985/contribution_graph_106566519089485180314/ -H "Authorization: Basic YWRtaW46YWRtaW4="
HTTP/1.1 200 OK
Cache-Control: must-revalidate
Content-Length: 514
Content-Type: application/json
Date: Fri, 26 Apr 2024 20:08:32 GMT
Server: CouchDB/3.3.3 (Erlang OTP/24)
X-Couch-Request-ID: 3c8f9ce314
X-CouchDB-Body-Time: 0

Tried a suggestion from stackoverflow proposing to clear connection header. This doesn't seem to take any effect.

proxy_http_version 1.1;
proxy_set_header Connection "";

What seems to stall the replication error was this suggestion:

proxy_read_timeout 3600;

However, this leaves requests in pending state! It clearly looks like it is stalling the replication error to sometime in the future.

Not really sure what's causing rxdb replication to fail like this!

Although I would like to point out that data does get writing to the couchdb collection. But this error certainly suggest the replication needs a fix!

Please suggest what can be done to resolve this issue! What am I doing wrong?

[pubkey]

It looks like you hit the 6-connection-per-origin limit that is hardcoded into the browsers. Please read this to learn about the workarounds: https://rxdb.info/replication-couchdb.html#limitations

[shivam-880]

Unfortunately this workaround doesn't work!

2024/04/27 17:59:39 [error] 29306#29306: *1 upstream timed out (110: Connection timed out) while reading upstream, 
client: 91.195.215.124, server: myapp.shivamkapoor.com, 
request: "GET /timetable_106566519089485180314/_changes?style=all_docs&feed=longpoll&since=now&include_docs=true&heartbeat=60000&limit=20&seq_interval=20 HTTP/2.0", 
upstream: "https://127.0.0.1:6985/timetable_106566519089485180314/_changes?style=all_docs&feed=longpoll&since=now&include_docs=true&heartbeat=60000&limit=20&seq_interval=20", 
host: "myapp.shivamkapoor.com:6984", 
referrer: "https://myapp.shivamkapoor.com/"

As you can see request does have HTTP/2.0 added this time unlike HTTP/1.1 before changes to the nginx conf:

request: "GET /timetable_106566519089485180314/_changes?
style=all_docs&
feed=longpoll&
since=now&
include_docs=true&
heartbeat=60000&
limit=20&
seq_interval=20 
HTTP/2.0"

This is what my nginx conf looks like now:

upstream couchdb {
    server localhost:6985;
    keepalive 16;
}

server {
    listen 6984 ssl http2;
    server_name myapp.shivamkapoor.com;

    ssl_certificate /opt/couchdb/etc/cert/fullchain.pem;
    ssl_certificate_key /opt/couchdb/etc/cert/privkey.pem;

    location / {
        proxy_pass https://couchdb;
        proxy_redirect off;
        proxy_buffering off;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Connection "";
    }
}

Nginx Version

# nginx -v
nginx version: nginx/1.14.0 (Ubuntu)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed soon. If you still have a problem, make a PR with a test case or to prove that you have tried to fix the problem. Notice that only bugs in the rxdb premium plugins are ensured to be fixed by the maintainer. Everything else is expected to be fixed by the community, likely you must fix it by yourself.

[stale[bot]]

Issues are autoclosed after some time. If you still have a problem, make a PR with a test case or to prove that you have tried to fix the problem.