search

crowdsecurity / example-docker-compose

Example integration of crowdsec in docker-compose
114 stars 25 forks source link

NPM not starting when using 'lepresidente/nginxproxymanager:latest' #13

Closed w24rdl closed 8 months ago

w24rdl commented 10 months ago

Trying to get the NPM example up and running from here.

When starting the docker compose stack, the NPM container fails with the following

sed: cannot rename /data/crowdsec/sedg2woQQ: Device or resource busy
s6-rc: warning: unable to start service cs-crowdsec-bouncer: command exited 4
/run/s6/basedir/scripts/rc.init: warning: s6-rc failed to properly bring all the services up! Check your logs (in /run/uncaught-logs/current if you have in-container logging) for more information.

The full log file is:

❯ Configuring npm user ...
❯ Configuring npm group ...
❯ Checking paths ...
❯ Setting ownership ...
❯ Dynamic resolvers ...
❯ IPv6 ...
Disabling IPV6 in hosts in: /etc/nginx/conf.d
- /etc/nginx/conf.d/crowdsec_openresty.conf
- /etc/nginx/conf.d/include/assets.conf
- /etc/nginx/conf.d/include/block-exploits.conf
- /etc/nginx/conf.d/include/ip_ranges.conf
- /etc/nginx/conf.d/include/force-ssl.conf
- /etc/nginx/conf.d/include/letsencrypt-acme-challenge.conf
- /etc/nginx/conf.d/include/ssl-ciphers.conf
- /etc/nginx/conf.d/include/proxy.conf
- /etc/nginx/conf.d/include/resolvers.conf
- /etc/nginx/conf.d/production.conf
- /etc/nginx/conf.d/default.conf
Disabling IPV6 in hosts in: /data/nginx
- /data/nginx/proxy_host/1.conf
- /data/nginx/proxy_host/4.conf
- /data/nginx/proxy_host/2.conf
- /data/nginx/proxy_host/3.conf
❯ Docker secrets ...

-------------------------------------
 _   _ ____  __  __
| \ | |  _ \|  \/  |
|  \| | |_) | |\/| |
| |\  |  __/| |  | |
|_| \_|_|   |_|  |_|
-------------------------------------
User:  npm PUID:0 ID:0 GROUP:0
Group: npm PGID:0 ID:0
-------------------------------------

❯ Starting backend ...
Deploy Crowdsec Openresty Bouncer..
Patch crowdsec-openresty-bouncer.conf ..
[12/30/2023] [4:10:42 PM] [Global   ] › ℹ  info      Using Sqlite: /data/database.sqlite
sed: cannot rename /data/crowdsec/sed68C3rj: Device or resource busy
s6-rc: warning: unable to start service cs-crowdsec-bouncer: command exited 4
/run/s6/basedir/scripts/rc.init: warning: s6-rc failed to properly bring all the services up! Check your logs (in /run/uncaught-logs/current if you have in-container logging) for more information.
[12/30/2023] [4:10:43 PM] [Migrate  ] › ℹ  info      Current database version: none
[12/30/2023] [4:10:49 PM] [Setup    ] › ℹ  info      Added Certbot plugins certbot-dns-cloudflare==$(certbot --version | grep -Eo '[0-9](\.[0-9]+)+') cloudflare
[12/30/2023] [4:10:49 PM] [Setup    ] › ℹ  info      Logrotate Timer initialized
[12/30/2023] [4:10:49 PM] [Setup    ] › ℹ  info      Logrotate completed.
[12/30/2023] [4:10:49 PM] [IP Ranges] › ℹ  info      Fetching IP Ranges from online services...
[12/30/2023] [4:10:49 PM] [IP Ranges] › ℹ  info      Fetching https://ip-ranges.amazonaws.com/ip-ranges.json
[12/30/2023] [4:10:49 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v4
[12/30/2023] [4:10:49 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v6
[12/30/2023] [4:10:50 PM] [SSL      ] › ℹ  info      Let's Encrypt Renewal Timer initialized
[12/30/2023] [4:10:50 PM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...
[12/30/2023] [4:10:50 PM] [IP Ranges] › ℹ  info      IP Ranges Renewal Timer initialized
[12/30/2023] [4:10:50 PM] [Global   ] › ℹ  info      Backend PID 156 listening on port 3000 ...
[12/30/2023] [4:10:52 PM] [Nginx    ] › ℹ  info      Reloading Nginx
[12/30/2023] [4:10:52 PM] [SSL      ] › ✖  error     Error: Command failed: /usr/sbin/nginx -s reload 
nginx: [error] invalid PID number "" in "/run/nginx/nginx.pid"

    at ChildProcess.exithandler (node:child_process:402:12)
    at ChildProcess.emit (node:events:513:28)
    at maybeClose (node:internal/child_process:1100:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:304:5)

the docker-compose.yaml is as follows

version: '3'

services:
  app:
    #image: 'jc21/nginx-proxy-manager:latest'
    image: 'lepresidente/nginxproxymanager:latest'

    restart: unless-stopped
    networks:
      crowdsec:
      nginx:
    ports:
      # These ports are in format :
      - '80:80' # Public HTTP Port
      - '443:443' # Public HTTPS Port
      - '81:81' # Admin Web Port
    environment:
      # Uncomment this if IPv6 is not enabled on your host
      DISABLE_IPV6: 'true'
    volumes:
      - /swarmvol/npm/data:/data
      - /swarmvol/crowdsec/crowdsec-openresty-bouncer.conf:/data/crowdsec/crowdsec-openresty-bouncer.conf
      - /swarmvol/npm/letsencrypt:/etc/letsencrypt
    security_opt:
      - no-new-privileges=true

  crowdsec:
    image: docker.io/crowdsecurity/crowdsec:latest
    container_name: crowdsec
    environment:
      - COLLECTIONS=crowdsecurity/nginx-proxy-manager
    volumes:
      - /swarmvol/crowdsec/crowdsec-db:/var/lib/crowdsec/data/
      - /swarmvol/crowdsec/config:/etc/crowdsec/
      - /swarmvol/npm/data/logs/:/var/log/npm:ro
      - /swarmvol/crowdsec/acquis.yaml:/etc/crowdsec/acquis.yaml
    networks:
      crowdsec:
    restart: unless-stopped
    security_opt:
      - no-new-privileges=true

networks:
    nginx:
      external: true
    crowdsec:
      external: true

Note im using a 3 node docker swarm cluster with /swarmvol being a truenas NFS mount to all hosts.

If it remove the following volume mount, the container starts as expected, but crowdsec is not enabled.

      - /swarmvol/crowdsec/crowdsec-openresty-bouncer.conf:/data/crowdsec/crowdsec-openresty-bouncer.conf

So it seems to me that something is the matter with this config file, I have tied multiple changes within crowdsec-openresty-bouncer.conf, including leaving it just as ENABLED=true, but each time the same error as above occurs.

Any idea what im doing wrong here?

Docker version

Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:08:01 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:01 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.25
  GitCommit:        d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc:
  Version:          1.1.10
  GitCommit:        v1.1.10-0-g18a0cb0
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Ubuntu 20.04.6 LTS

LaurenceJJones commented 10 months ago

sed: cannot rename /data/crowdsec/sed68C3rj: Device or resource busy

I dont believe it an issue with the actual configuration, it an issue within the startup command to sed replace a value. Most likely this line might be best to open an issue directly in the repo than here.

w24rdl commented 10 months ago

I managed to get this working after some trial and error. I had to remove the following line from the docker-compose.yml

- /swarmvol/crowdsec/crowdsec-openresty-bouncer.conf:/data/crowdsec/crowdsec-openresty-bouncer.conf

then add the following environment items:

   environment:
      CROWDSEC_LAPI: "http://crowdsec:8080"
      CROWDSEC_KEY: "<API-KEY-GOES-HERE>"

and then manually edit the crowdsec-openresty-bouncer.conf located within /data/crowdsec/ on the npm container to have ENABLED=true (Leaving the rest of the file as defaults)

I think the way im mounting the crowdsec-openresty-bouncer.conf file to the npm container is causing issues with sed.

What my docker-compose.yml looks like now:

version: '3'

services:
  app:
    #image: 'jc21/nginx-proxy-manager:latest'
    image: 'lepresidente/nginxproxymanager:latest'
    #image: 'lepresidente/nginxproxymanager:debug'

    restart: unless-stopped
    networks:
      crowdsec:
      nginx:
    ports:
      # These ports are in format :
      - '80:80' # Public HTTP Port
      - '443:443' # Public HTTPS Port
      - '81:81' # Admin Web Port
    environment:
      CROWDSEC_LAPI: "http://crowdsec:8080"
      CROWDSEC_KEY: "supersecret"
      # Uncomment this if IPv6 is not enabled on your host
      DISABLE_IPV6: 'true'
    volumes:
      - /swarmvol/npm/data:/data
      #- /swarmvol/crowdsec/crowdsec-openresty-bouncer.conf:/data/crowdsec/crowdsec-openresty-bouncer.conf
      - /swarmvol/npm/letsencrypt:/etc/letsencrypt
    security_opt:
      - no-new-privileges=true

  crowdsec:
    image: docker.io/crowdsecurity/crowdsec:latest
    container_name: crowdsec
    environment:
      - COLLECTIONS=crowdsecurity/nginx-proxy-manager
    volumes:
      - /swarmvol/crowdsec/crowdsec-db:/var/lib/crowdsec/data/
      - /swarmvol/crowdsec/config:/etc/crowdsec/
      - /swarmvol/npm/data/logs/:/var/log/npm:ro
      - /swarmvol/crowdsec/acquis.yaml:/etc/crowdsec/acquis.yaml
    networks:
      crowdsec:
    restart: unless-stopped
    security_opt:
      - no-new-privileges=true

networks:
    nginx:
      external: true
    crowdsec:
      external: true