Closed thelittlefireman closed 4 months ago
Hi @thelittlefireman, could you provide your configuration please ?
@thelittlefireman bump
[2023-10-27 22:59:25] - API - ❌ - Can't send API request to http://nginx:5100/reload : Request failed: HTTPConnectionPool(host='nginx', port=5100): Max retries exceeded with url: /reload (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd160a1b3e0>: Failed to establish a new connection: [Errno 111] Connection refused'))
[2023-10-27 22:59:25] - SCHEDULER - ❌ - Error while reloading nginx
[2023-10-27 22:59:26] - SCHEDULER - ℹ️ - Executing job bunkernet-register from plugin bunkernet ...
[2023-10-27 22:59:27] - BUNKERNET - ℹ️ - BunkerNet is not activated, skipping registration...
[2023-10-27 22:59:27] - SCHEDULER - ℹ️ - Executing job greylist-download from plugin greylist ...
[2023-10-27 22:59:27] - SCHEDULER - ℹ️ - Successfully updated database for the job bunkernet-register from plugin bunkernet
[2023-10-27 22:59:28] - GREYLIST - ℹ️ - Greylist is not activated, skipping downloads...
[2023-10-27 22:59:28] - SCHEDULER - ℹ️ - Successfully updated database for the job greylist-download from plugin greylist
[2023-10-27 22:59:29] - SCHEDULER - ℹ️ - Executing job realip-download from plugin realip ...
[2023-10-27 22:59:30] - REALIP - ℹ️ - RealIP is not activated, skipping download...
[2023-10-27 22:59:30] - SCHEDULER - ℹ️ - Successfully updated database for the job realip-download from plugin realip
[2023-10-27 22:59:31] - SCHEDULER - ℹ️ - Executing job whitelist-download from plugin whitelist ...
[2023-10-27 22:59:33] - WHITELIST - ℹ️ - ✅ Database connection established
[2023-10-27 22:59:33] - WHITELIST - ℹ️ - Whitelist for IP is not cached, processing downloads..
[2023-10-27 22:59:33] - WHITELIST - ℹ️ - Whitelist for RDNS is not cached, processing downloads..
[2023-10-27 22:59:33] - WHITELIST - ℹ️ - Whitelist for ASN is not cached, processing downloads..
[2023-10-27 22:59:33] - WHITELIST - ℹ️ - Whitelist for USER_AGENT is not cached, processing downloads..
[2023-10-27 22:59:33] - WHITELIST - ℹ️ - Whitelist for URI is not cached, processing downloads..
[2023-10-27 22:59:33] - SCHEDULER - ℹ️ - Successfully updated database for the job whitelist-download from plugin whitelist
[2023-10-27 23:57:07] - SCHEDULER - ❌ - An error occurred when checking for changes in the database : Traceback (most recent call last):
File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1969, in _exec_single_context
self.dialect.do_execute(
File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/default.py", line 922, in do_execute
cursor.execute(statement, parameters)
File "/usr/share/bunkerweb/deps/python/pymysql/cursors.py", line 153, in execute
result = self._query(query)
^^^^^^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/pymysql/cursors.py", line 322, in _query
conn.query(q)
File "/usr/share/bunkerweb/deps/python/pymysql/connections.py", line 558, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/pymysql/connections.py", line 822, in _read_query_result
result.read()
File "/usr/share/bunkerweb/deps/python/pymysql/connections.py", line 1200, in read
first_packet = self.connection._read_packet()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/pymysql/connections.py", line 739, in _read_packet
packet_header = self._read_bytes(4)
^^^^^^^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/pymysql/connections.py", line 795, in _read_bytes
raise err.OperationalError(
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/share/bunkerweb/db/Database.py", line 281, in check_changes
.first()
^^^^^^^
File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/query.py", line 2748, in first
return self.limit(1)._iter().first() # type: ignore
^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/query.py", line 2847, in _iter
result: Union[ScalarResult[_T], Result[_T]] = self.session.execute(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/session.py", line 2306, in execute
return self._execute_internal(
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/session.py", line 2188, in _execute_internal
result: Result[Any] = compile_state_cls.orm_execute_statement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/context.py", line 293, in orm_execute_statement
result = conn.execute(
^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1416, in execute
return meth(
^^^^^
File "/usr/share/bunkerweb/deps/python/sqlalchemy/sql/elements.py", line 516, in _execute_on_connection
return connection._execute_clauseelement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1639, in _execute_clauseelement
ret = self._execute_context(
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1848, in _execute_context
return self._exec_single_context(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1988, in _exec_single_context
self._handle_dbapi_exception(
File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 2343, in _handle_dbapi_exception
raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1969, in _exec_single_context
self.dialect.do_execute(
File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/default.py", line 922, in do_execute
cursor.execute(statement, parameters)
File "/usr/share/bunkerweb/deps/python/pymysql/cursors.py", line 153, in execute
result = self._query(query)
^^^^^^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/pymysql/cursors.py", line 322, in _query
conn.query(q)
File "/usr/share/bunkerweb/deps/python/pymysql/connections.py", line 558, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/pymysql/connections.py", line 822, in _read_query_result
result.read()
File "/usr/share/bunkerweb/deps/python/pymysql/connections.py", line 1200, in read
first_packet = self.connection._read_packet()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/pymysql/connections.py", line 739, in _read_packet
packet_header = self._read_bytes(4)
^^^^^^^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/pymysql/connections.py", line 795, in _read_bytes
raise err.OperationalError(
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
[SQL: SELECT bw_metadata.custom_configs_changed AS bw_metadata_custom_configs_changed, bw_metadata.external_plugins_changed AS bw_metadata_external_plugins_changed, bw_metadata.config_changed AS bw_metadata_config_changed, bw_metadata.instances_changed AS bw_metadata_instances_changed
FROM bw_metadata
WHERE bw_metadata.id = %(id_1)s
LIMIT %(param_1)s]
[parameters: {'id_1': 1, 'param_1': 1}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)
[2023-10-27 23:57:07] - ENTRYPOINT - ℹ️ - Scheduler stopped
I'm also losing connection from scheduler and nginx really often. (not stable)
version: '3.5'
networks:
bw-universe:
name: bw-universe
external: true
bw-docker:
name: bw-docker
external: true
backend-net:
external: true
name: backend-net
services:
authelia:
image: authelia/authelia
container_name: authelia
user: "1024:101"
environment:
- PUID=1024
- PGID=101
volumes:
- ./authelia:/config
- "/etc/TZ:/etc/timezone:ro"
- "/etc/localtime:/etc/localtime:ro"
- "./nginx/letsencrypt:/certs:ro"
restart: unless-stopped
ports:
- 127.0.0.1:9091:9091
environment:
- TZ=France/Paris
networks:
- backend-net
bw-syslog-ng:
image: lscr.io/linuxserver/syslog-ng:latest
container_name: bw-syslog-ng
environment:
- PUID=1024
- PGID=101
ports:
- 514:5114/udp
volumes:
- ./bw-syslog-ng/config:/config
- ./bw-syslog-ng/log:/var/log
networks:
- backend-net
bw-docker-proxy:
image: tecnativa/docker-socket-proxy:nightly
container_name: bw-docker-proxy
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
environment:
- CONTAINERS=1
- LOG_LEVEL=warning
networks:
- bw-docker
bw-scheduler:
image: bunkerity/bunkerweb-scheduler:dev
container_name: bw-scheduler
depends_on:
- bw-docker-proxy
- bw-db
- bw-syslog-ng
logging:
driver: syslog
options:
syslog-address: "udp://localhost:514"
volumes:
- ./nginx/letsencrypt:/var/cache/bunkerweb/letsencrypt/etc:mode=0770,uid=101,gid=101
- ./nginx/config:/data
environment:
- DOCKER_HOST=tcp://bw-docker-proxy:2375
- API_HTTP_PORT=5100
- API_SERVER_NAME=nginx
- LOG_LEVEL=notice
- DATABASE_URI=mysql+pymysql://bunkerweb:XXXXXXXXXXX@bw-db:3306/db
networks:
- bw-universe
- bw-docker
# Fix nginx running in host network
extra_hosts:
- "nginx:192.168.1.150"
bw-db:
image: mysql:latest
container_name: bw-db
user: "1024:101"
environment:
- MYSQL_RANDOM_ROOT_PASSWORD=yes
- MYSQL_DATABASE=db
- MYSQL_USER=bunkerweb
- MYSQL_PASSWORD=XXXXXXXXXXX
expose:
- 3306
volumes:
- "./nginx/db:/var/lib/mysql"
networks:
- bw-universe
nginx:
image: bunkerity/bunkerweb:dev
container_name: nginx
restart: always
network_mode: "host"
labels:
- "bunkerweb.INSTANCE" # required for the scheduler to recognize the container
volumes:
# Set timezone
- "/etc/TZ:/etc/timezone:ro"
- "/etc/localtime:/etc/localtime:ro"
# dropping all capabilities
cap_drop:
- ALL
security_opt:
- no-new-privileges
depends_on:
- authelia
- bw-scheduler
- bw-docker-proxy
- bw-db
- bw-syslog-ng
logging:
driver: syslog
options:
syslog-address: "udp://localhost:514"
environment:
- EXTERNAL_PLUGIN_URLS=https://github.com/bunkerity/bunkerweb-plugins/archive/refs/tags/v1.1.zip
- DATABASE_URI=mysql+pymysql://bunkerweb:XXXXXXXXXXX@bw-db:3306/db
- MULTISITE=yes
- LOG_LEVEL=notice
- API_WHITELIST_IP=127.0.0.0/8 172.16.3.0/24 172.16.4.0/24
- API_HTTP_PORT=5100
- API_SERVER_NAME=nginx
- USE_LETS_ENCRYPT_STAGING=no
- AUTO_LETS_ENCRYPT=yes
- EMAIL_LETS_ENCRYPT=admin@test.te
- SERVER_NAME=auth.test.te syno.test.te nextcloud.test.te
- SERVE_FILES=no
- REDIRECT_HTTP_TO_HTTPS=yes
- DISABLE_DEFAULT_SERVER=yes
- DNS_RESOLVERS=1.1.1.1 1.0.0.1
- HTTP_PORT=10080
- HTTPS_PORT=10443
- HTTP2=yes
- SSL_PROTOCOLS=TLSv1.2 TLSv1.3
- USE_GZIP=yes
- GZIP_COMP_LEVEL=5
- GZIP_MIN_LENGTH=500
- USE_BROTLI=yes
- BROTLI_COMP_LEVEL=5
- BROTLI_MIN_LENGTH=500
- STRICT_TRANSPORT_SECURITY=max-age=31536000; includeSubDomains; preload
- USE_BAD_BEHAVIOR=no
- BAD_BEHAVIOR_STATUS_CODES=400 401 403 405 444 429
- USE_DNSBL=yes
- USE_BLACKLIST=yes
- USE_WHITELIST=yes
- WHITELIST_IP=127.0.0.1/8 192.168.0.0/16
- GREYLIST_USER_AGENT=okhttp* WebDAV DAVx5* axios* sindresorhus* *ggpht.com GoogleImageProxy Python* aiohttp* Bitwarden_Mobile*
- USE_REVERSE_PROXY=yes
- REVERSE_PROXY_CONNECT_TIMEOUT=180s
- REVERSE_PROXY_SEND_TIMEOUT=180s
- REVERSE_PROXY_READ_TIMEOUT=180s
- REVERSE_PROXY_URL_999=/authelia
- REVERSE_PROXY_HOST_999=https://127.0.0.1:9091/api/verify
- REVERSE_PROXY_HEADERS_999=X-Original-URL $$scheme://$$http_host$$request_uri;Content-Length ""
- auth.test.te_USE_REVERSE_PROXY=yes
- auth.test.te_REVERSE_PROXY_URL=/
- auth.test.te_REVERSE_PROXY_HOST=https://127.0.0.1:9091
- auth.test.te_REVERSE_PROXY_INTERCEPT_ERRORS=no
- |
auth.test.te_CUSTOM_CONF_SERVER_HTTP_auto-custom-auth=
proxy_busy_buffers_size 256k;
proxy_buffers 8 128k;
proxy_buffer_size 128k;
- nextcloud.test.te_REVERSE_PROXY_URL=/
- nextcloud.test.te_REVERSE_PROXY_HOST=http://127.0.0.1:32680
- nextcloud.test.te_REVERSE_PROXY_KEEPALIVE=yes
- nextcloud.test.te_REVERSE_PROXY_BUFFERING=no
- nextcloud.test.te_ALLOWED_METHODS=GET|POST|HEAD|COPY|DELETE|LOCK|MKCOL|MOVE|PROPFIND|PROPPATCH|PUT|UNLOCK|OPTIONS|REPORT|SEARCH
- nextcloud.test.te_MAX_CLIENT_SIZE=2G
- nextcloud.test.te_CONTENT_SECURITY_POLICY=object-src 'none'; frame-ancestors 'self'; form-action 'self'; block-all-mixed-content; sandbox allow-forms allow-same-origin allow-scripts allow-popups allow-downloads; base-uri 'self';
- nextcloud.test.te_BAD_BEHAVIOR_STATUS_CODES=400 401 403 405 444
- nextcloud.test.te_X_CONTENT_TYPE_OPTIONS=
- nextcloud.test.te_FEATURE_POLICY=accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'none'; battery 'none'; camera 'none'; display-capture 'none'; document-domain 'none'; encrypted-media 'none'; execution-while-not-rendered 'none'; execution-while-out-of-viewport 'none'; fullscreen 'none'; geolocation 'none'; gyroscope 'none'; layout-animation 'none'; legacy-image-formats 'none'; magnetometer 'none'; microphone 'none'; midi 'none'; navigation-override 'none'; payment 'none'; picture-in-picture 'self'; publickey-credentials-get 'none'; speaker-selection 'none'; sync-xhr 'none'; unoptimized-images 'none'; unsized-media 'none'; usb 'none'; screen-wake-lock 'none'; web-share 'none'; xr-spatial-tracking 'none';
- nextcloud.test.te_PERMISSIONS_POLICY=accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), cross-origin-isolated=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), hid=(), idle-detection=(), magnetometer=(), microphone=(), midi=(), navigation-override=(), payment=(), picture-in-picture=(self), publickey-credentials-get=(), screen-wake-lock=(), serial=(), usb=(), web-share=(), xr-spatial-tracking=()
- nextcloud.test.te_REMOVE_HEADERS=x-content-type-options
[...]
- USE_MODSECURITY=yes
- USE_MODSECURITY_CRS=yes
- USE_BUNKERNET=no
- USE_API=no
- USE_LIMIT_REQ=yes
- LIMIT_REQ_RATE=200r/s
- USE_LIMIT_CONN=yes
- LIMIT_CONN_MAX_HTTP1=20
- LIMIT_CONN_MAX_HTTP2=200
- BLACKLIST_COUNTRY=CN RU
- USE_OPEN_FILE_CACHE=yes
- OPEN_FILE_CACHE=max=1000 inactive=60s
- OPEN_FILE_CACHE_ERRORS=yes
- OPEN_FILE_CACHE_MIN_USES=2
- OPEN_FILE_CACHE_VALID=30s
### PLUGINS ###
- USE_CROWDSEC=no
- CROWDSEC_API=http://127.0.0.1:48080
- CROWDSEC_API_KEY=XXXXXXXXXXXXXXXXX
- USE_VIRUSTOTAL=no
- USE_DISCORD=no
- USE_SLACK=no
- USE_CLAMAV=no
- USE_CORAZA=no
I don't know if it' related but i've got watchtower container which update my scheduler, nginx etc containers.
maybe bw-scheduler and idkw it looses connection to nginx container.
Hi @thelittlefireman, thank you for your configuration. By the way you don't need to provide the settings in the scheduler container:
- API_HTTP_PORT=5100
- API_SERVER_NAME=nginx
Do you have issues sending the configuration from the scheduler to BunkerWeb with the setting:
- USE_API=no
?
This can be because of your Docker environment having unstable networks for some reason 🤔
Hello @thelittlefireman,
Can you try to disable watchtower and tell us if does anything ?
I switch to docker version v1.5.3 (vs dev). So watchtower doesn't update it every day and I don't new crash.
But I'm still loosing connection from scheduler to nginx after 1 day, which is quite annoying:
[2023-11-10 13:46:25] - API - ❌ - Can't send API request to http://nginx:5100/cache : Request failed: HTTPConnectionPool(host='nginx', port=5100): Max retries exceeded with url: /cache (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6aa3bfc920>: Failed to establish a new connection: [Errno 111] Connection refused'))
[2023-11-10 13:46:25] - SCHEDULER - ❌ - Error while sending /var/cache/bunkerweb folder
[2023-11-10 13:46:25] - SCHEDULER - ℹ️ - Reloading nginx ...
[2023-11-10 13:46:25] - API - ❌ - Can't send API request to http://nginx:5100/reload : Request failed: HTTPConnectionPool(host='nginx', port=5100): Max retries exceeded with url: /reload (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6aa3bfe150>: Failed to establish a new connection: [Errno 111] Connection refused'))
[2023-11-10 13:46:25] - SCHEDULER - ❌ - Error while reloading nginx
Nginx container seems to lose API_PORT (5100) after a will.
XXXX@XXXX:~$ docker exec -it -u 0 nginx netstat -planet | grep 5100
Any clue ?
Hello @thelittlefireman,
Can you try with new 1.5.5 version ?
@thelittlefireman bump
What happened?
Scheduler crash after some times. Not the first time. occurs since 1.5.2. I'm on dev docker pull on 10-14-2023.
How to reproduce?
start bw-scheduler with letsencrypt and multi site enable
Configuration file(s) (yaml or .env)
No response
Relevant log output
BunkerWeb version
dev
What integration are you using?
Docker
Linux distribution (if applicable)
No response
Removed private data
Code of Conduct