On first install: Error response from daemon. Error starting userland proxy: listen tcp4 0.0.0.0:9000: bind: address already in use (need to change port?)

[MaximilianKohler]

Version:

• listmonk: latest 2.5.1 I think
• OS: CentOS 7

EDIT: it's a port conflict with php fpm. Is it simple to assign a different port to listmonk? Can I just pick any random number (3876) and replace "9000" in docker-compose.yml and config.toml with "3876"? And then just run docker compose up -d app db again?

---

First install. Ran:

cd /home/nginx/domains/test1.mysite.com/public/
mkdir listmonk
cd listmonk
sh -c "$(curl -fsSL https://raw.githubusercontent.com/knadh/listmonk/master/install-prod.sh)"

>  checking for an existing docker db volume
>  fetching config.toml from listmonk repo
>  fetching docker-compose.yml from listmonk repo
>  generating a random password
>  modifying config.toml
>  modifying docker-compose.yml
>  running migrations
[+] Running 14/14
 ✔ db 13 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                                                                     4.9s
   ✔ a378f10b3218 Pull complete                                                                                                                            0.2s
   ✔ 3ae46626af9d Pull complete                                                                                                                            0.0s
   ✔ 3572f9d0180e Pull complete                                                                                                                            0.1s
   ✔ 8790dde5065e Pull complete                                                                                                                            0.1s
   ✔ 86b9ebea34dc Pull complete                                                                                                                            0.2s
   ✔ f12c6ef3a820 Pull complete                                                                                                                            0.2s
   ✔ e2e0fad8f22e Pull complete                                                                                                                            0.3s
   ✔ 3519a24f40df Pull complete                                                                                                                            0.3s
   ✔ a54792b78270 Pull complete                                                                                                                            0.9s
   ✔ 20c4a1dd56b3 Pull complete                                                                                                                            0.3s
   ✔ 7df6b57887e4 Pull complete                                                                                                                            0.4s
   ✔ 90d4e4e64153 Pull complete                                                                                                                            0.4s
   ✔ 053f198d4c38 Pull complete                                                                                                                            0.4s
[+] Running 3/3
 ✔ Network listmonk_listmonk        Created                                                                                                                0.0s
 ✔ Volume "listmonk_listmonk-data"  Created                                                                                                                0.0s
 ✔ Container listmonk_db            Started                                                                                                                0.1s
>  waiting for db container to be up. retrying in 3s
>  waiting for db container to be up. retrying in 3s
>  waiting for db container to be up. retrying in 3s
>  waiting for db container to be up. retrying in 3s
>  waiting for db container to be up. retrying in 3s
[+] Creating 1/0
 ✔ Container listmonk_db  Running                                                                                                                          0.0s
[+] Running 7/7
 ✔ app 6 layers [⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                                                                            0.9s
   ✔ 7264a8db6415 Pull complete                                                                                                                            0.1s
   ✔ 018e408a4e93 Pull complete                                                                                                                            0.1s
   ✔ d53dd0513dc1 Pull complete                                                                                                                            0.1s
   ✔ ad15b75486f1 Pull complete                                                                                                                            0.2s
   ✔ eedf74ce218a Pull complete                                                                                                                            0.1s
   ✔ 76dfcc273dd9 Pull complete                                                                                                                            0.1s
2023/10/28 01:24:45 main.go:102: v2.5.1 (a6a2b69 2023-08-15T15:49:28Z, linux/amd64)
2023/10/28 01:24:45 init.go:145: reading config: config.toml
2023/10/28 01:24:45 init.go:273: connecting to db: listmonk_db:5432/listmonk

** first time installation **
** IMPORTANT: This will wipe existing listmonk tables and types in the DB 'listmonk' **
continue (y/N)?  y
2023/10/28 01:25:24 install.go:179: setup complete
2023/10/28 01:25:24 install.go:180: run the program and access the dashboard at 0.0.0.0:9000
>  starting app
[+] Running 2/0
 ✔ Container listmonk_db   Running                                                                                                                         0.0s
 ✔ Container listmonk_app  Created                                                                                                                         0.0s
Error response from daemon: driver failed programming external connectivity on endpoint listmonk_app (a6ead4911fd947589812fe3c1a29ca5517a51233e938ff10b4d92b3a90ffe57c): Error starting userland proxy: listen tcp4 0.0.0.0:9000: bind: address already in use

This is the only other instance I found of this error: https://github.com/knadh/listmonk/issues/880. I ran the commands @knadh suggested:

docker ps
CONTAINER ID   IMAGE         COMMAND                  CREATED          STATUS                    PORTS                                       NAMES
f56824174ef1   postgres:13   "docker-entrypoint.s…"   14 minutes ago   Up 13 minutes (healthy)   0.0.0.0:9432->5432/tcp, :::9432->5432/tcp   listmonk_db

docker compose up -d app db
[+] Running 1/0
 ✔ Container listmonk_db   Running                                                                                                                         0.0s
 ⠋ Container listmonk_app  Starting                                                                                                                        0.0s
Error response from daemon: driver failed programming external connectivity on endpoint listmonk_app (1d9983b94efeda5b204cce280104d24c87699355b2915a752bc08ce58195e7ca): Error starting userland proxy: listen tcp4 0.0.0.0:9000: bind: address already in use

Not sure if this is related:

psql
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
        Is the server running locally and accepting connections on that socket?

[MaximilianKohler]

I also noticed that in both of these sample nginx configs https://github.com/knadh/listmonk/issues/684#issuecomment-1026480289 -- https://github.com/knadh/listmonk/issues/778#issuecomment-1100119710 they use proxy_pass. So I added proxy_pass http://localhost:9000; to my test1.mysite.com.ssl.conf and restarted nginx but it didn't make a difference. I tried with proxy_pass http://127.0.0.1:9000; and proxy_pass http://0.0.0.0:9000; too.

[MaximilianKohler]

Well there was a psql/postgres issue (wasn't installed properly & running). But after I solved that, I get the same error:

sh -c "$(curl -fsSL https://raw.githubusercontent.com/knadh/listmonk/master/install-prod.sh)"

>  checking for an existing docker db volume
x listmonk-data volume already exists. Please use docker compose down -v to remove old volumes for a fresh setup of PostgreSQL.
[03:23][root@centos7test listmonk]# docker compose down -v
[+] Running 4/4
 ✔ Container listmonk_app         Removed                                                                                                                  0.0s
 ✔ Container listmonk_db          Removed                                                                                                                  0.2s
 ✔ Volume listmonk_listmonk-data  Removed                                                                                                                  0.0s
 ✔ Network listmonk_listmonk      Removed                                                                                                                  0.1s
[03:23][root@centos7test listmonk]# sh -c "$(curl -fsSL https://raw.githubusercontent.com/knadh/listmonk/master/install-prod.sh)"

>  checking for an existing docker db volume
>  fetching config.toml from listmonk repo
>  fetching docker-compose.yml from listmonk repo
>  generating a random password
>  modifying config.toml
>  modifying docker-compose.yml
>  running migrations
[+] Running 3/3
 ✔ Network listmonk_listmonk        Created                                                                                                                0.0s
 ✔ Volume "listmonk_listmonk-data"  Created                                                                                                                0.0s
 ✔ Container listmonk_db            Started                                                                                                                0.0s
>  waiting for db container to be up. retrying in 3s
>  waiting for db container to be up. retrying in 3s
>  waiting for db container to be up. retrying in 3s
>  waiting for db container to be up. retrying in 3s
>  waiting for db container to be up. retrying in 3s
[+] Creating 1/0
 ✔ Container listmonk_db  Running                                                                                                                          0.0s
2023/10/29 03:25:40 main.go:102: v2.5.1 (a6a2b69 2023-08-15T15:49:28Z, linux/amd64)
2023/10/29 03:25:40 init.go:145: reading config: config.toml
2023/10/29 03:25:40 init.go:273: connecting to db: listmonk_db:5432/listmonk

** first time installation **
** IMPORTANT: This will wipe existing listmonk tables and types in the DB 'listmonk' **
continue (y/N)?  y
2023/10/29 03:26:05 install.go:179: setup complete
2023/10/29 03:26:05 install.go:180: run the program and access the dashboard at 0.0.0.0:9000
>  starting app
[+] Running 2/2
 ✔ Container listmonk_db   Running                                                                                                                         0.0s
 ✔ Container listmonk_app  Created                                                                                                                         0.0s
Error response from daemon: driver failed programming external connectivity on endpoint listmonk_app (ec1b04789995fe597daab8844ea4ceb1e49e0cbbe4ffa4e452cc43a74154f766): Error starting userland proxy: listen tcp4 0.0.0.0:9000: bind: address already in use

Ah, I found a good hint here https://stackoverflow.com/questions/37896369/error-starting-userland-proxy-listen-tcp-0-0-0-03306-bind-address-already-in

sudo netstat -nlpt |grep 9000
tcp        0      0 127.0.0.1:9000          0.0.0.0:*               LISTEN      62239/php-fpm: mast

I guess I have to either change the port for php-fpm or listmonk. I'm wondering which would be easiest and less problematic?

For listmonk, can I just pick any random number (3876) and replace "9000" in docker-compose.yml and config.toml with "3876"? And then just run docker compose up -d app db again?

[mr-karan]

To change port for listmonk:

• Ensure no other container of listmonk app is running. docker ps | grep listmonk you can check with this.
• Change L11 to 3876:9000. This will expose the port 3876 on your local network to the container's network interface on port 9000.
• For NGINX setup, if you're running NGINX on your local machine, you can proxy_pass to the <MACHINE_IP>:3876. You can also run NGINX as a docker container within the listmonk's container (for that you need to add a service nginx in the docker-compose.yml). If you do that, then proxy_pass will be set to http://app:9000. Docker's network will resolve the DNS for app and directly speak to port 9000 (which the app is exposing within it's own network).

I am not in favour of editing our install-prod.sh or install-demo.sh, as that's an example/quick usage setup which works for majority of setups. If any kind of customisation has to be done, it should be done at the user's end (as demonstrated above).

[MaximilianKohler]

Thank you! That seemed to work. I did:

 docker ps
CONTAINER ID   IMAGE         COMMAND                  CREATED        STATUS                  PORTS                                       NAMES
6cfb36dcf88e   postgres:13   "docker-entrypoint.s…"   45 hours ago   Up 45 hours (healthy)   0.0.0.0:9432->5432/tcp, :::9432->5432/tcp   listmonk_db

[00:36][root@centos7test ~]# docker stop listmonk_db
listmonk_db

[00:36][root@centos7test ~]# docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

Changed L11 to 3870:9000.

[00:55][root@centos7test ~]# cd /home/nginx/domains/test1.mysite.com/public/listmonk
[00:56][root@centos7test listmonk]# docker compose up -d app db
[+] Running 2/2
 ✔ Container listmonk_db   Started                                                                                                                         0.0s
 ✔ Container listmonk_app  Started                                                                                                                         0.0s
[00:56][root@centos7test listmonk]# docker ps
CONTAINER ID   IMAGE                      COMMAND                  CREATED              STATUS                        PORTS                                       NAMES
ce0ed641d1ec   listmonk/listmonk:latest   "./listmonk"             About a minute ago   Up About a minute             0.0.0.0:3870->9000/tcp, :::3870->9000/tcp   listmonk_app
6cfb36dcf88e   postgres:13                "docker-entrypoint.s…"   46 hours ago         Up About a minute (healthy)   0.0.0.0:9432->5432/tcp, :::9432->5432/tcp   listmonk_db

I'm getting a "502 Bad Gateway nginx" when I visit test1.mysite.com though. I'm wondering what I'm missing.

My test1.mysite.com.ssl.conf is:

#x# HTTPS-DEFAULT
 server {
   listen   80;
#x#   listen   [::]:80;
   server_name test1.mysite.com www.test1.mysite.com;
   return 302 https://test1.mysite.com$request_uri;
   root /home/nginx/domains/test1.mysite.com/public;
   include /usr/local/nginx/conf/staticfiles.conf;
 }

#       

server {
  listen 443 ssl http2;

  server_name test1.mysite.com www.test1.mysite.com;

  include /usr/local/nginx/conf/ssl/test1.mysite.com/test1.mysite.com.crt.key.conf;
  include /usr/local/nginx/conf/ssl_include.conf;

  # cloudflare authenticated origin pull cert community.centminmod.com/threads/13847/
  #ssl_client_certificate /usr/local/nginx/conf/ssl/cloudflare/test1.mysite.com/origin.crt;
  #ssl_verify_client on;

  # mozilla recommended
  ssl_ciphers TLS13-AES-128-GCM-SHA256:TLS13-AES-256-GCM-SHA384:TLS13-CHACHA20-POLY1305-SHA256:TLS13-AES-128-CCM-8-SHA256:TLS13-AES-128-CCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
  ssl_prefer_server_ciphers   on;
  #add_header Alternate-Protocol  443:npn-spdy/3;

  # before enabling HSTS line below read centminmod.com/nginx_domain_dns_setup.html#hsts
  #add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
  #add_header X-Frame-Options SAMEORIGIN;
  add_header X-Xss-Protection "1; mode=block" always;
  add_header X-Content-Type-Options "nosniff" always;
  #add_header Referrer-Policy "strict-origin-when-cross-origin";
  #add_header Permissions-Policy "accelerometer=(), camera=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), payment=(), usb=()";
  #spdy_headers_comp 5;
  ssl_buffer_size 1369;
  ssl_session_tickets on;

  # enable ocsp stapling
  #resolver 8.8.8.8 8.8.4.4 1.1.1.1 1.0.0.1 valid=10m;
  #resolver_timeout 10s;
  #ssl_stapling on;
  #ssl_stapling_verify on;

# ngx_pagespeed & ngx_pagespeed handler
#include /usr/local/nginx/conf/pagespeed.conf;
#include /usr/local/nginx/conf/pagespeedhandler.conf;
#include /usr/local/nginx/conf/pagespeedstatslog.conf;

  # limit_conn limit_per_ip 16;
  # ssi  on;

  access_log /home/nginx/domains/test1.mysite.com/log/access.log combined buffer=256k flush=5m;
  #access_log /home/nginx/domains/test1.mysite.com/log/access.json main_json buffer=256k flush=5m;
  error_log /home/nginx/domains/test1.mysite.com/log/error.log;

  include /usr/local/nginx/conf/autoprotect/test1.mysite.com/autoprotect-test1.mysite.com.conf;
  root /home/nginx/domains/test1.mysite.com/public;
  # uncomment cloudflare.conf include if using cloudflare for
  # server and/or vhost site
  include /usr/local/nginx/conf/cloudflare.conf;
  include /usr/local/nginx/conf/503include-main.conf;

  location / {
  include /usr/local/nginx/conf/503include-only.conf;

  proxy_pass http://127.0.0.1:9000;

# block common exploits, sql injections etc
#include /usr/local/nginx/conf/block.conf;

  # Enables directory listings when index file not found
  #autoindex  on;

  # Shows file listing times as local time
  #autoindex_localtime on;

  # Wordpress Permalinks example
  #try_files $uri $uri/ /index.php?q=$uri&$args;

  }

  include /usr/local/nginx/conf/php.conf;

  include /usr/local/nginx/conf/pre-staticfiles-local-test1.mysite.com.conf;
  include /usr/local/nginx/conf/pre-staticfiles-global.conf;
  include /usr/local/nginx/conf/staticfiles.conf;
  include /usr/local/nginx/conf/drop.conf;
  #include /usr/local/nginx/conf/errorpage.conf;
  include /usr/local/nginx/conf/vts_server.conf;
}

Enabling Cloudflare's orange proxy for the subdomain results in:

This page isn’t working right now test1.mysite.com redirected you too many times.
To fix this issue, try clearing your cookies.
ERR_TOO_MANY_REDIRECTS

EDIT: I changed proxy_pass http://127.0.0.1:9000; to proxy_pass http://127.0.0.1:5870, restarted nginx, and I still get the error.

[MaximilianKohler]

I decided to delete the server and retry with a proper SSL cert instead of a self-signed dummy cert. When I got to the docker setup step I noticed an "ip tables" error. I'm not sure if this might be related:

sudo systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: active (running) since Sat 2023-11-04 01:14:19 UTC; 8s ago
     Docs: https://docs.docker.com
 Main PID: 14875 (dockerd)
    Tasks: 8
   Memory: 24.9M
   CGroup: /system.slice/docker.service
           └─14875 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Nov 04 01:14:19 centos7test systemd[1]: Starting Docker Application Container Engine...
Nov 04 01:14:19 centos7test dockerd[14875]: time="2023-11-04T01:14:19.374912801Z" level=info msg="Starting up"
Nov 04 01:14:19 centos7test dockerd[14875]: time="2023-11-04T01:14:19.401961834Z" level=info msg="Loading containers: start."
Nov 04 01:14:19 centos7test dockerd[14875]: time="2023-11-04T01:14:19.496339829Z" level=info msg="Loading containers: done."
Nov 04 01:14:19 centos7test dockerd[14875]: time="2023-11-04T01:14:19.509602894Z" level=warning msg="WARNING: bridge-nf-call-iptables is disabled"
Nov 04 01:14:19 centos7test dockerd[14875]: time="2023-11-04T01:14:19.509624574Z" level=warning msg="WARNING: bridge-nf-call-ip6tables is disabled"
Nov 04 01:14:19 centos7test dockerd[14875]: time="2023-11-04T01:14:19.509648700Z" level=info msg="Docker daemon" commit=311b9ff graphdriver=overlay2...on=24.0.7
Nov 04 01:14:19 centos7test dockerd[14875]: time="2023-11-04T01:14:19.509724623Z" level=info msg="Daemon has completed initialization"
Nov 04 01:14:19 centos7test dockerd[14875]: time="2023-11-04T01:14:19.526123308Z" level=info msg="API listen on /run/docker.sock"
Nov 04 01:14:19 centos7test systemd[1]: Started Docker Application Container Engine.
Hint: Some lines were ellipsized, use -l to show in full.

I found these

• https://stackoverflow.com/questions/63226322/bridge-nf-call-iptables-and-bridge-nf-call-ip6tables-disabled-on-wsl2
• https://stackoverflow.com/questions/28744670/unable-to-set-net-bridge-bridge-nf-call-iptables-within-docker-container/64009376#64009376

And checked mine, and they were 0:

sysctl net.bridge.bridge-nf-call-iptables
net.bridge.bridge-nf-call-iptables = 0

I ignored that for now, and continued through the setup. After completing all steps (except for adding proxy_pass) test1.mysite.com loads with the index.html placeholder still. After adding proxy_pass http://127.0.0.1:5870;, it stays on the index.html until restarting nginx. After that I get a 502 bad gateway.

I changed proxy_pass from http://127.0.0.1:5870; to http://test1.mysite.com:5870; and now it loads, but just basic text and the urls are http://localhost:9000/subscription/form".

I found this https://stackoverflow.com/questions/47091356/docker-nginx-reverse-proxy-gives-502-bad-gateway and changed proxy_pass to http://listmonk:5870; but restarting nginx gives this error: Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xe" for details.

http://app:5870; and http://listmonk_app:5870; give the same error.

I put the IP of the server Eg: http://11.156.121.118:5870; and the result is the same as putting the domain name http://test1.mysite.com:5870.

I ran:

sudo docker compose stop ; sudo docker compose up
[+] Stopping 2/2
 ✔ Container listmonk_app  Stopped                                                                                                                         0.1s
 ✔ Container listmonk_db   Stopped                                                                                                                         0.1s
[+] Running 4/0
 ✔ Container listmonk_demo_db   Created                                                                                                                    0.0s
 ✔ Container listmonk_db        Created                                                                                                                    0.0s
 ✔ Container listmonk_app       Created                                                                                                                    0.0s
 ✔ Container listmonk_demo_app  Created                                                                                                                    0.0s
Attaching to listmonk_app, listmonk_db, listmonk_demo_app, listmonk_demo_db
Error response from daemon: driver failed programming external connectivity on endpoint listmonk_demo_db (d19b6ad3590fb2847cbd628cf3507011786fc5bdddf64825631716878ff3eb4e): Bind for 0.0.0.0:9432 failed: port is already allocated

Then:

sudo netstat -nlpt |grep 9432
tcp        0      0 0.0.0.0:9432            0.0.0.0:*               LISTEN      30810/docker-proxy
tcp6       0      0 :::9432                 :::*                    LISTEN      30815/docker-proxy

And then it was back to "502 bad gateway".

I removed the listmonk_demo lines from docker-compose.yml and ran sudo docker compose stop ; sudo docker compose up again and it seemed to work. But the website still loads with the broken localhost:9000 stuff.

I tried proxy_pass http://localhost:5870; and proxy_pass http://0.0.0.0:5870; and both result in 502 bad gateway.

I checked the error logs and see a bunch of:

2023/11/04 02:44:55 [error] 32415#32415: *18 open() "/home/nginx/domains/test1.mysite.com/public/public/custom.css" failed (2: No such file or directory), client: <my IP>, server: test1.mysite.com, request: "GET /public/custom.css HTTP/2.0", host: "test1.mysite.com", referrer: "https://test1.mysite.com/"

2023/11/04 02:44:55 [error] 32415#32415: *18 open() "/home/nginx/domains/test1.mysite.com/public/public/custom.js" failed (2: No such file or directory), client: <my IP>, server: test1.mysite.com, request: "GET /public/custom.js HTTP/2.0", host: "test1.mysite.com", referrer: "https://test1.mysite.com/"

So I'm guessing that http://test1.mysite.com:5870; is correct and I need to add some directories to the docker-compose.yml?

Not sure if this (http server started on [::]:9000) is useful info:

sudo docker compose stop ; sudo docker compose up
[+] Stopping 2/2
 ✔ Container listmonk_app  Stopped                                                                                                                         0.1s
 ✔ Container listmonk_db   Stopped                                                                                                                         0.1s
[+] Running 2/0
 ✔ Container listmonk_db   Created                                                                                                                         0.0s
 ✔ Container listmonk_app  Recreated                                                                                                                       0.0s
Attaching to listmonk_app, listmonk_db
listmonk_db   |
listmonk_db   | PostgreSQL Database directory appears to contain a database; Skipping initialization
listmonk_db   |
listmonk_db   | 2023-11-04 03:09:39.125 UTC [1] LOG:  starting PostgreSQL 13.12 (Debian 13.12-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
listmonk_db   | 2023-11-04 03:09:39.125 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
listmonk_db   | 2023-11-04 03:09:39.125 UTC [1] LOG:  listening on IPv6 address "::", port 5432
listmonk_db   | 2023-11-04 03:09:39.128 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
listmonk_db   | 2023-11-04 03:09:39.131 UTC [26] LOG:  database system was shut down at 2023-11-04 03:09:38 UTC
listmonk_db   | 2023-11-04 03:09:39.134 UTC [1] LOG:  database system is ready to accept connections
listmonk_app  | 2023/11/04 03:09:39 main.go:102: v2.5.1 (a6a2b69 2023-08-15T15:49:28Z, linux/amd64)
listmonk_app  | 2023/11/04 03:09:39 init.go:145: reading config: config.toml
listmonk_app  | 2023/11/04 03:09:39 init.go:273: connecting to db: listmonk_db:5432/listmonk
listmonk_app  | 2023/11/04 03:09:39 init.go:593: media upload provider: filesystem
listmonk_app  | 2023/11/04 03:09:39 init.go:517: loaded email (SMTP) messenger: username@smtp.yoursite.com
listmonk_app  | ⇨ http server started on [::]:9000

I also tried this:

  location / {
  proxy_pass http://127.0.0.1:5870;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_set_header Host $http_host;
  proxy_set_header X-NginX-Proxy true;
  proxy_redirect off;

The result is 502 bad gateway. And when I substitute in proxy_pass http://test1.mysite.com:5870; it's back to the broken page.

I found this https://stackoverflow.com/questions/63311201/nginx-resolving-localhost-on-proxy-pass and ran:

curl 127.0.0.1:5870
curl: (56) Recv failure: Connection reset by peer

9000 does the same:

curl 127.0.0.1:9000
curl: (56) Recv failure: Connection reset by peer

[MaximilianKohler]

Tried fixing the docker error next:

[04:16][root@centos7test listmonk]# sysctl net.bridge.bridge-nf-call-iptables
net.bridge.bridge-nf-call-iptables = 0
[04:16][root@centos7test listmonk]# sysctl -w net.bridge.bridge-nf-call-iptables="1"
net.bridge.bridge-nf-call-iptables = 1
[04:17][root@centos7test listmonk]# sysctl net.bridge.bridge-nf-call-iptables
net.bridge.bridge-nf-call-iptables = 1
[04:17][root@centos7test listmonk]# sysctl net.bridge.bridge-nf-call-ip6tables
net.bridge.bridge-nf-call-ip6tables = 0
[04:17][root@centos7test listmonk]# sysctl -w net.bridge.bridge-nf-call-ip6tables=1
net.bridge.bridge-nf-call-ip6tables = 1
[04:17][root@centos7test listmonk]# sysctl net.bridge.bridge-nf-call-ip6tables
net.bridge.bridge-nf-call-ip6tables = 1
[04:17][root@centos7test listmonk]# ngxrestart
Redirecting to /bin/systemctl restart nginx.service
[04:18][root@centos7test listmonk]# sysctl net.bridge.bridge-nf-call-iptables
net.bridge.bridge-nf-call-iptables = 1

[04:18][root@centos7test listmonk]# systemctl status docker -l
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2023-11-04 01:14:19 UTC; 3h 4min ago
     Docs: https://docs.docker.com
 Main PID: 14875 (dockerd)
    Tasks: 39
   Memory: 122.4M
   CGroup: /system.slice/docker.service
           ├─14875 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
           ├─45718 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 9432 -container-ip 172.18.0.2 -container-port 5432
           ├─45723 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 9432 -container-ip 172.18.0.2 -container-port 5432
           ├─45808 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 5870 -container-ip 172.18.0.3 -container-port 9000
           └─45814 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 5870 -container-ip 172.18.0.3 -container-port 9000

Nov 04 03:09:38 centos7test dockerd[14875]: time="2023-11-04T03:09:38.614490694Z" level=info msg="ignoring event" container=683b237806b2a99568eb4c4370f997fc7ffc44211ea3ae5fd6ac4a0547802f0c module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 04 03:09:38 centos7test dockerd[14875]: time="2023-11-04T03:09:38.621221817Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=683b237806b2a99568eb4c4370f997fc7ffc44211ea3ae5fd6ac4a0547802f0c daemonShuttingDown=false error="restart canceled" execDuration=26m52.640151909s exitStatus="{2 2023-11-04 03:09:38.603986533 +0000 UTC}" hasBeenManuallyStopped=true restartCount=0
Nov 04 03:09:38 centos7test dockerd[14875]: time="2023-11-04T03:09:38.727534890Z" level=info msg="ignoring event" container=f8d43916e5687750dc83eec936db00f52484a1c74bd0d3ae0fa76347a8df48c7 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 04 03:09:38 centos7test dockerd[14875]: time="2023-11-04T03:09:38.735497038Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=f8d43916e5687750dc83eec936db00f52484a1c74bd0d3ae0fa76347a8df48c7 daemonShuttingDown=false error="restart canceled" execDuration=26m52.921223312s exitStatus="{0 2023-11-04 03:09:38.71605866 +0000 UTC}" hasBeenManuallyStopped=true restartCount=0
Nov 04 03:09:38 centos7test dockerd[14875]: time="2023-11-04T03:09:38.735505304Z" level=error msg="attach failed with error: error attaching stderr stream: write unix /run/docker.sock->@: write: broken pipe"
Nov 04 03:43:52 centos7test dockerd[14875]: time="2023-11-04T03:43:52.864415319Z" level=info msg="ignoring event" container=6738da2529770362947c5b0e084f8c410e762d25f11aa1726e2984132d5f6cf5 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 04 03:43:52 centos7test dockerd[14875]: time="2023-11-04T03:43:52.872488879Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=6738da2529770362947c5b0e084f8c410e762d25f11aa1726e2984132d5f6cf5 daemonShuttingDown=false error="restart canceled" execDuration=34m13.63859669s exitStatus="{2 2023-11-04 03:43:52.852312438 +0000 UTC}" hasBeenManuallyStopped=true restartCount=0
Nov 04 03:43:52 centos7test dockerd[14875]: time="2023-11-04T03:43:52.969658369Z" level=info msg="ignoring event" container=f8d43916e5687750dc83eec936db00f52484a1c74bd0d3ae0fa76347a8df48c7 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 04 03:43:52 centos7test dockerd[14875]: time="2023-11-04T03:43:52.976153599Z" level=error msg="attach failed with error: error attaching stderr stream: write unix /run/docker.sock->@: write: broken pipe"
Nov 04 03:43:52 centos7test dockerd[14875]: time="2023-11-04T03:43:52.976199235Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=f8d43916e5687750dc83eec936db00f52484a1c74bd0d3ae0fa76347a8df48c7 daemonShuttingDown=false error="restart canceled" execDuration=34m13.918769758s exitStatus="{0 2023-11-04 03:43:52.957858708 +0000 UTC}" hasBeenManuallyStopped=true restartCount=0

I changed proxy_pass back: proxy_pass http://127.0.0.1:5870;, then restarted nginx, and it's back to the 502 error.

[MaximilianKohler]

@mr-karan any idea what is causing it?

[meljoann]

Yeah, I'm here from https://github.com/knadh/listmonk/issues/1590 -- just wanted to add I'm on Debian 11.

Subdomain isn't working for me either (nginx reverse proxy, custom port, docker), I'm getting a 502 page.

[MaximilianKohler]

Well, I just found out that the site loads properly on serverIP:5870 and if I change proxy_pass http://127.0.0.1:5870; to proxy_pass http://listmonk.mydomain.com:5870; then it will load on listmonk.mydomain.com:5870. But it gives the 502 error when I visit the site without the port.

If I set proxy_pass http://127.0.0.1:5870; and visit listmonk.mydomain.com:5870 I get:

The connection for this site is not secure
listmonk.mydomain.com sent an invalid response.
[Try running Windows Network Diagnostics](javascript:diagnoseErrors()).
ERR_SSL_PROTOCOL_ERROR

It's gotta be something in the nginx config right?

I tried some of the solutions here https://serverfault.com/questions/351212/nginx-redirects-to-port-8080-when-accessing-url-without-slash

proxy_set_header Host $host:$server_port; didn't seem to change anything.

I also tried

  proxy_pass http://listmonk.example.com:5870/;
  proxy_redirect http://listmonk.example.com:5870/ http://listmonk.example.com/;

Maybe this https://stackoverflow.com/questions/30097334/nodejs-on-nginx-not-working-without-a-port-number-in-the-url is the answer, but I don't understand it.

[meljoann]

Hmmm! Did you try simplifying your nginx to just the essentials, and if it works: adding back your customisations one by one?

[MaximilianKohler]

Yep, I basically copied this simple outline that seems to be working for everyone else https://github.com/knadh/listmonk/issues/1590#issuecomment-1812399067.

I asked on another forum https://lemmy.world/post/8756439 and someone suggested I should try proxy_pass http://listmonk_app:5870; but that gives an error https://lemmy.world/comment/5586470.

[MaximilianKohler]

It was a firewall issue. I disabled my firewall and it works.

[MaximilianKohler]

If we're not using docker we can simply change address = "localhost:9000" in config.toml to any custom port right?