search

bunkerity / bunkerweb

🛡️ Open-source and next-generation Web Application Firewall (WAF)
https://www.bunkerweb.io
GNU Affero General Public License v3.0
6.29k stars 352 forks source link

[BUG] MODSECURITY_CRS_VERSION 4 with 50+ virtual hosts causes API reload timeout #1525

Open jeremyj opened 1 week ago

jeremyj commented 1 week ago

What happened?

When enabling Modsecurity CRS v4 the bunkerweb container takes time to test the nginx config and the reload API call from the scheduler times out. This is tested on a 32GB 4 vCPU AWS machine.

How to reproduce?

Deploy a Bunkerweb stack on a host that has 50+ virtual hosts attached to the bw_services network and labeled with the following:

bunkerweb.REVERSE_PROXY_URL=/
bunkerweb.REVERSE_PROXY_HOST=http://container:80
bunkerweb.SERVER_NAME=virtualhost
bunkerweb.USE_REVERSE_PROXY=yes

Try changing the Modsecurity CRS from v3 to v4 either with an env variable or via the Web UI

Configuration file(s) (yaml or .env)

services:
  bunkerweb:
    image: bunkerity/bunkerweb:1.5.10
    ports:
      - 80:8080
      - 443:8443
    labels:
      - "bunkerweb.INSTANCE=yes"
    environment:
      - SERVER_NAME=
      - DATABASE_URI=mariadb+pymysql://bunkerweb:xxxxxxxxxxxx@bw-db:3306/db
      - AUTOCONF_MODE=yes
      - MULTISITE=yes
      - API_WHITELIST_IP=127.0.0.0/8 10.20.30.0/24 
      - AUTO_LETS_ENCRYPT=yes
      - EMAIL_LETS_ENCRYPT=xxxxxxxxxxxx
      - USE_LIMIT_REQ=no
      - WHITELIST_IP=
      - USE_CROWDSEC=yes
      - CROWDSEC_API=http://10.10.10.253:8080 # This is the address of the CrowdSec container API in the same network
      - CROWDSEC_API_KEY=xxxxxxxxxxxx # Remember to set a stronger key for the bouncer
      - MODSECURITY_CRS_VERSION=3
      - USE_MODSECURITY=yes
      - USE_MODSECURITY_CRS=yes
      - EXTERNAL_PLUGIN_URLS=https://github.com/bunkerity/bunkerweb-plugins/archive/refs/tags/v1.6.zip      
      - LISTEN_STREAM=no
      - SERVE_FILES=no
      - USE_BROTLI=yes
      - USE_GZIP=yes
      - USE_CLIENT_CACHE=yes
    networks:
      - bw-universe
      - bw-services
      - bw-plugins
    logging:
      driver: syslog
      options:
        syslog-address: "udp://10.10.10.254:514"

  bw-autoconf:
    image: bunkerity/bunkerweb-autoconf:1.5.10
    depends_on:
      - bunkerweb
      - bw-docker
    environment:
      - DATABASE_URI=mariadb+pymysql://bunkerweb:xxxxxxxxxxxx@bw-db:3306/db
      - AUTOCONF_MODE=yes
      - DOCKER_HOST=tcp://bw-docker:2375
    networks:
      - bw-universe
      - bw-docker

  bw-scheduler:
    image: bunkerity/bunkerweb-scheduler:1.5.10
    depends_on:
      - bunkerweb
      - bw-docker
    environment:
      - DATABASE_URI=mariadb+pymysql://bunkerweb:xxxxxxxxxxxx@bw-db:3306/db
      - DOCKER_HOST=tcp://bw-docker:2375
      - AUTOCONF_MODE=yes
      - LOG_LEVEL=info
    networks:
      - bw-universe
      - bw-docker

  bw-docker:
    image: tecnativa/docker-socket-proxy:nightly
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - CONTAINERS=1
      - LOG_LEVEL=warning
    networks:
      - bw-docker

  bw-db:
    image: mariadb:10.10
    environment:
      - MYSQL_RANDOM_ROOT_PASSWORD=yes
      - MYSQL_DATABASE=db
      - MYSQL_USER=bunkerweb
      - MYSQL_PASSWORD=xxxxxxxxxxxx
    volumes:
      - bw-data:/var/lib/mysql
    networks:
      - bw-docker

  bw-ui:
    image: bunkerity/bunkerweb-ui:1.5.10
    networks:
      bw-docker:
      bw-universe:
        aliases:
          - bw-ui
    environment:
      - DATABASE_URI=mariadb+pymysql://bunkerweb:xxxxxxxxxxxx@bw-db:3306/db
      - DOCKER_HOST=tcp://bw-docker:2375
      - AUTOCONF_MODE=yes
      - ADMIN_USERNAME=xxxxxxxxxxxx
      - ADMIN_PASSWORD=xxxxxxxxxxxx
    labels:
      - "bunkerweb.SERVER_NAME=xxxxxxxxxxxx"
      - "bunkerweb.USE_UI=yes"
      - "bunkerweb.USE_REVERSE_PROXY=yes"
      - "bunkerweb.REVERSE_PROXY_URL=/path"
      - "bunkerweb.REVERSE_PROXY_HOST=http://bw-ui:7000"
      - "bunkerweb.MAX_CLIENT_SIZE=50m"

  crowdsec:
    image: crowdsecurity/crowdsec:v1.6.3 # Use the latest version but always pin the version for a better stability/security
    volumes:
      - cs-data:/var/lib/crowdsec/data # To persist the CrowdSec data
      - bw-logs:/var/log:ro # The logs of BunkerWeb for CrowdSec to parse
      - /docker/conf/bw-conf/acquis.yaml:/etc/crowdsec/acquis.yaml # The acquisition file for BunkerWeb logs
    environment:
      BOUNCER_KEY_bunkerweb: "xxxxxxxxxxxx" # Remember to set a stronger key for the bouncer
      COLLECTIONS: "crowdsecurity/nginx crowdsecurity/wordpress crowdsecurity/base-http-scenarios" # If you don't want to use the AppSec Component use this line instead
      LEVEL_INFO: true
      ENROLL_INSTANCE_NAME: "xxxxxxxxxxxx"
    networks:
      bw-plugins:
          ipv4_address: 10.10.10.253

  syslog:
    image: balabit/syslog-ng:4.8.0 # Use the latest version but always pin the version for a better stability/security
    command: --no-caps
    volumes:
      - bw-logs:/var/log # The logs of BunkerWeb for syslog-ng to store
      - /docker/conf/bw-conf/syslog-ng.conf:/etc/syslog-ng/syslog-ng.conf # The syslog-ng configuration file
    networks:
      bw-plugins:
        ipv4_address: 10.10.10.254 # The IP address of the syslog service so BunkerWeb can send logs to it

volumes:
  bw-data:
  bw-logs:
  cs-data:

networks:
  bw-universe:
    name: bw-universe
    ipam:
      driver: default
      config:
        - subnet: 10.20.30.0/24
  bw-plugins:
    ipam:
      driver: default
      config:
        - subnet: 10.10.10.0/24
  bw-services:
    name: bw-services
  bw-docker:
    name: bw-docker

Relevant log output

[2024-09-27 17:18:01 +0000] [SCHEDULER] [130] [ℹ ] - Instances changed, generating ...
[2024-09-27 17:18:01 +0000] [SCHEDULER] [130] [🐛] - Changes: {'custom_configs_changed': False, 'external_plugins_changed': False, 'pro_plugins_changed': False, 'instances_changed': True, 'last_custom_configs_change': datetime.datetime(2024, 9, 27, 17, 12, 31), 'last_external_plugins_change': datetime.datetime(2024, 9, 27, 17, 11, 22), 'last_pro_plugins_change': datetime.datetime(2024, 9, 27, 17, 11, 22), 'last_instances_change': datetime.datetime(2024, 9, 27, 17, 18), 'plugins_config_changed': {}}
[2024-09-27 17:18:01 +0000] [SCHEDULER] [130] [🐛] - Testing write access to the database ...
[2024-09-27 17:18:01 +0000] [SCHEDULER] [130] [ℹ ] - Removing old custom configs files ...
[2024-09-27 17:18:01 +0000] [SCHEDULER] [130] [ℹ ] - Sending /etc/bunkerweb/configs folder ...
[2024-09-27 17:18:01 +0000] [API] [130] [ℹ ] - Successfully sent API request to http://bunkerweb-bunkerweb-1:5000/custom_configs
[2024-09-27 17:18:01 +0000] [SCHEDULER] [130] [ℹ ] - Successfully sent /etc/bunkerweb/configs folder
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Generator started ...
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Settings : /usr/share/bunkerweb/settings.json
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Templates : /usr/share/bunkerweb/confs
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Core : /usr/share/bunkerweb/core
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Plugins : /etc/bunkerweb/plugins
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Pro plugins : /etc/bunkerweb/pro/plugins
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Output : /etc/nginx
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Target : /etc/nginx
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Variables : /var/tmp/bunkerweb/scheduler.env
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Checking arguments ...
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Computing config ...
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Removing old files ...
[2024-09-27 17:18:02 +0000] [GENERATOR] [691] [ℹ ] - Rendering templates ...
[2024-09-27 17:18:09 +0000] [GENERATOR] [691] [ℹ ] - Generator successfully executed !
[2024-09-27 17:18:09 +0000] [SCHEDULER] [130] [ℹ ] - Sending /etc/nginx folder ...
[2024-09-27 17:18:09 +0000] [SCHEDULER] [130] [ℹ ] - Sending /var/cache/bunkerweb folder ...
[2024-09-27 17:18:10 +0000] [API] [130] [ℹ ] - Successfully sent API request to http://bunkerweb-bunkerweb-1:5000/confs
[2024-09-27 17:18:10 +0000] [API] [130] [ℹ ] - Successfully sent API request to http://bunkerweb-bunkerweb-1:5000/cache
[2024-09-27 17:18:10 +0000] [SCHEDULER] [130] [ℹ ] - Successfully sent /var/cache/bunkerweb folder
[2024-09-27 17:18:40 +0000] [API] [130] [❌] - Can't send API request to http://bunkerweb-bunkerweb-1:5000/reload : Request failed: HTTPConnectionPool(host='bunkerweb-bunkerweb-1', port=5000): Read timed out. (read timeout=30)
[2024-09-27 17:18:40 +0000] [SCHEDULER] [130] [❌] - Error while reloading bunkerweb, failing over to last working configuration ...
[2024-09-27 17:18:40 +0000] [SCHEDULER] [130] [ℹ ] - Sending /var/tmp/bunkerweb/failover/config folder ...
[2024-09-27 17:18:40 +0000] [SCHEDULER] [130] [ℹ ] - Sending /var/tmp/bunkerweb/failover/cache folder ...
[2024-09-27 17:18:40 +0000] [SCHEDULER] [130] [ℹ ] - Sending /var/tmp/bunkerweb/failover/custom_configs folder ...
[2024-09-27 17:18:40 +0000] [API] [130] [ℹ ] - Successfully sent API request to http://bunkerweb-bunkerweb-1:5000/custom_configs
[2024-09-27 17:18:40 +0000] [SCHEDULER] [130] [ℹ ] - Successfully sent /var/tmp/bunkerweb/failover/custom_configs folder
[2024-09-27 17:18:41 +0000] [API] [130] [ℹ ] - Successfully sent API request to http://bunkerweb-bunkerweb-1:5000/confs
[2024-09-27 17:18:51 +0000] [API] [130] [❌] - Can't send API request to http://bunkerweb-bunkerweb-1:5000/cache : Request failed: ('Connection aborted.', TimeoutError('timed out'))
[2024-09-27 17:18:51 +0000] [SCHEDULER] [130] [❌] - Error while sending /var/tmp/bunkerweb/failover/cache folder
[2024-09-27 17:19:16 +0000] [API] [130] [ℹ ] - Successfully sent API request to http://bunkerweb-bunkerweb-1:5000/reload
[2024-09-27 17:19:16 +0000] [SCHEDULER] [130] [ℹ ] - Executing job scheduler ...

bwapi 10.20.30.4 - - [27/Sep/2024:17:18:01 +0000] "POST /custom_configs HTTP/1.1" 200 80 "-" "bwapi"
bwapi 10.20.30.4 - - [27/Sep/2024:17:18:09 +0000] "POST /confs HTTP/1.1" 200 67 "-" "bwapi"
bwapi 10.20.30.4 - - [27/Sep/2024:17:18:10 +0000] "POST /cache HTTP/1.1" 200 78 "-" "bwapi"
bwapi 10.20.30.4 - - [27/Sep/2024:17:18:40 +0000] "POST /custom_configs HTTP/1.1" 200 80 "-" "bwapi"
bwapi 10.20.30.4 - - [27/Sep/2024:17:18:41 +0000] "POST /confs HTTP/1.1" 200 67 "-" "bwapi"
bwapi 10.20.30.4 - - [27/Sep/2024:17:18:10 +0000] "POST /reload HTTP/1.1" 200 58 "-" "bwapi"
bwapi 10.20.30.4 - - [27/Sep/2024:17:18:56 +0000] "POST /reload HTTP/1.1" 200 58 "-" "bwapi"
2024/09/27 17:19:16 [warn] 6633#6633: *938 [API] call from 10.20.30.4 on /cache failed : client aborted, client: 10.20.30.4, server: bwapi, request: "POST /cache HTTP/1.1", host: "bwapi"
bwapi 10.20.30.4 - - [27/Sep/2024:17:18:56 +0000] "POST /cache HTTP/1.1" 400 53 "-" "bwapi"

BunkerWeb version

1.5.10

What integration are you using?

Docker

Linux distribution (if applicable)

No response

Removed private data

Code of Conduct

TheophileDiot commented 1 week ago

Hi, thank you for opening this Issue and sorry for the late response. Indeed the problem is present and is caused by ModSecurity. We'll dig into it and let you know about our findings. AFAIK it's because of ModSecurity but we'll have a deeper look!