Closed farzadha2 closed 10 months ago
Hi @farzadha2, thank you for opening this feature request. Are you certain that the problem is only caused by the let's encrypt plugin ? Could you provide us with your log if you're not sure ? Thank you.
hi @TheophileDiot thanks for the reply, Sure thing im not sure if its normal but when doing any changes on the WebGUI it takes around 10mins as it seems to reapply everything
as this is the log, as the letsencrypt goes to each domain to see if it needs to renew
flush_context.execute()
File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/unitofwork.py", line 466, in execute
rec.execute(self)
File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/unitofwork.py", line 642, in execute
util.preloaded.orm_persistence.save_obj(
File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/persistence.py", line 85, in save_obj
_emit_update_statements(
File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/persistence.py", line 909, in _emit_update_statements
c = connection.execute(
^^^^^^^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1412, in execute
return meth(
^^^^^
File "/usr/share/bunkerweb/deps/python/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
return connection._execute_clauseelement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1635, in _execute_clauseelement
ret = self._execute_context(
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1844, in _execute_context
return self._exec_single_context(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1984, in _exec_single_context
self._handle_dbapi_exception(
File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 2339, in _handle_dbapi_exception
raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1965, in _exec_single_context
self.dialect.do_execute(
File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/default.py", line 921, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
[SQL: UPDATE bw_jobs SET last_run=? WHERE bw_jobs.name = ?]
[parameters: ('2023-10-16 15:29:08.573519', 'greylist-download')]
(Background on this error at: https://sqlalche.me/e/20/e3q8)
[2023-10-16 15:29:16] - SCHEDULER - ℹ️ - Custom configs changed, generating ...
[2023-10-16 15:29:16] - SCHEDULER - ℹ️ - Config changed, generating ...
[2023-10-16 15:29:16] - SCHEDULER - ℹ️ - Removing old custom configs files ...
[2023-10-16 15:29:16] - SCHEDULER - ℹ️ - Generating new custom configs ...
[2023-10-16 15:29:16] - SCHEDULER - ℹ️ - Sending custom configs to BunkerWeb
[2023-10-16 15:29:16] - API - ℹ️ - Successfully sent API request to http://sites_mybunker_1:5000/custom_configs
[2023-10-16 15:29:20] - SCHEDULER - ℹ️ - Executing job custom-cert from plugin customcert ...
[2023-10-16 15:29:20] - SCHEDULER - ℹ️ - Executing job realip-download from plugin realip ...
[2023-10-16 15:29:20] - SCHEDULER - ℹ️ - Executing job certbot-new from plugin letsencrypt ...
[2023-10-16 15:29:20] - SCHEDULER - ℹ️ - Executing job self-signed from plugin selfsigned ...
[2023-10-16 15:29:21] - SCHEDULER - ℹ️ - Executing job mmdb-country from plugin jobs ...
[2023-10-16 15:29:21] - SCHEDULER - ℹ️ - Successfully updated database for the job custom-cert from plugin customcert
[2023-10-16 15:29:22] - LETS-ENCRYPT.NEW - ℹ️ - Database connection established
[2023-10-16 15:29:22] - REALIP - ℹ️ - RealIP is not activated, skipping download...
[2023-10-16 15:29:22] - SCHEDULER - ℹ️ - Executing job default-server-cert from plugin misc ...
[2023-10-16 15:29:22] - SCHEDULER - ℹ️ - Successfully updated database for the job realip-download from plugin realip
[2023-10-16 15:29:22] - SCHEDULER - ℹ️ - Executing job whitelist-download from plugin whitelist ...
[2023-10-16 15:29:22] - SCHEDULER - ℹ️ - Successfully updated database for the job self-signed from plugin selfsigned
[2023-10-16 15:29:22] - LETS-ENCRYPT.NEW - ℹ️ - Successfully retrieved Let's Encrypt data from db cache
[2023-10-16 15:29:24] - JOBS.MMDB-COUNTRY - ℹ️ - Database connection established
[2023-10-16 15:29:24] - DEFAULT-SERVER-CERT - ℹ️ - Skipping generation of self-signed certificate for default server (already present)
[2023-10-16 15:29:24] - WHITELIST - ℹ️ - Database connection established
[2023-10-16 15:29:24] - JOBS.MMDB-COUNTRY - ℹ️ - country.mmdb is already in cache, skipping download...
[2023-10-16 15:29:24] - SCHEDULER - ℹ️ - Executing job mmdb-asn from plugin jobs ...
[2023-10-16 15:29:24] - WHITELIST - ℹ️ - Whitelist for IP is not cached, processing downloads..
[2023-10-16 15:29:24] - WHITELIST - ℹ️ - Whitelist for RDNS is not cached, processing downloads..
[2023-10-16 15:29:24] - WHITELIST - ℹ️ - Whitelist for ASN is not cached, processing downloads..
[2023-10-16 15:29:24] - WHITELIST - ℹ️ - Whitelist for USER_AGENT is not cached, processing downloads..
[2023-10-16 15:29:24] - WHITELIST - ℹ️ - Whitelist for URI is not cached, processing downloads..
[2023-10-16 15:29:24] - SCHEDULER - ℹ️ - Successfully updated database for the job mmdb-country from plugin jobs
[2023-10-16 15:29:25] - SCHEDULER - ℹ️ - Executing job update-check from plugin misc ...
[2023-10-16 15:29:25] - SCHEDULER - ℹ️ - Successfully updated database for the job default-server-cert from plugin misc
[2023-10-16 15:29:25] - SCHEDULER - ℹ️ - Successfully updated database for the job whitelist-download from plugin whitelist
[2023-10-16 15:29:25] - SCHEDULER - ℹ️ - Executing job greylist-download from plugin greylist ...
[2023-10-16 15:29:27] - GREYLIST - ℹ️ - Greylist is not activated, skipping downloads...
[2023-10-16 15:29:27] - SCHEDULER - ℹ️ - Executing job bunkernet-register from plugin bunkernet ...
[2023-10-16 15:29:28] - UPDATE-CHECK - ℹ️ - Latest version is already installed: v1.5.2
[2023-10-16 15:29:28] - SCHEDULER - ℹ️ - Executing job blacklist-download from plugin blacklist ...
[2023-10-16 15:29:32] - SCHEDULER - ⚠️ - Failed to update database for the job greylist-download from plugin greylist: Traceback (most recent call last):
File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1965, in _exec_single_context
self.dialect.do_execute(
File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/default.py", line 921, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: database is locked
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/share/bunkerweb/db/Database.py", line 1035, in update_job
session.commit()
File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/session.py", line 1923, in commit
trans.commit(_to_root=True)
File "<string>", line 2, in commit
File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/state_changes.py", line 139, in _go
ret_value = fn(self, *arg, **kw)
^^^^^^^^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/session.py", line 1239, in commit
self._prepare_impl()
File "<string>", line 2, in _prepare_impl
File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/state_changes.py", line 139, in _go
ret_value = fn(self, *arg, **kw)
^^^^^^^^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/session.py", line 1214, in _prepare_impl
self.session.flush()
File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/session.py", line 4179, in flush
self._flush(objects)
File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/session.py", line 4314, in _flush
with util.safe_reraise():
File "/usr/share/bunkerweb/deps/python/sqlalchemy/util/langhelpers.py", line 147, in __exit__
raise exc_value.with_traceback(exc_tb)
File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/session.py", line 4275, in _flush
flush_context.execute()
File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/unitofwork.py", line 466, in execute
rec.execute(self)
File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/unitofwork.py", line 642, in execute
util.preloaded.orm_persistence.save_obj(
File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/persistence.py", line 85, in save_obj
_emit_update_statements(
File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/persistence.py", line 909, in _emit_update_statements
c = connection.execute(
^^^^^^^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1412, in execute
return meth(
^^^^^
File "/usr/share/bunkerweb/deps/python/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
return connection._execute_clauseelement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1635, in _execute_clauseelement
ret = self._execute_context(
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1844, in _execute_context
return self._exec_single_context(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1984, in _exec_single_context
self._handle_dbapi_exception(
File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 2339, in _handle_dbapi_exception
raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1965, in _exec_single_context
self.dialect.do_execute(
File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/default.py", line 921, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
[SQL: UPDATE bw_jobs SET last_run=? WHERE bw_jobs.name = ?]
[parameters: ('2023-10-16 15:29:27.153031', 'greylist-download')]
(Background on this error at: https://sqlalche.me/e/20/e3q8)
[2023-10-16 15:29:33] - JOBS.MMDB-ASN - ⚠️ - Can't connect to database, retrying in 5 seconds ...
[2023-10-16 15:29:33] - BUNKERNET - ⚠️ - Can't connect to database, retrying in 5 seconds ...
[2023-10-16 15:29:34] - BLACKLIST - ⚠️ - Can't connect to database, retrying in 5 seconds ...
[2023-10-16 15:29:34] - SCHEDULER - ℹ️ - Successfully updated database for the job update-check from plugin misc
[2023-10-16 15:29:38] - JOBS.MMDB-ASN - ℹ️ - Database connection established
[2023-10-16 15:29:38] - JOBS.MMDB-ASN - ℹ️ - asn.mmdb is already in cache, skipping download...
[2023-10-16 15:29:38] - SCHEDULER - ℹ️ - Executing job download-plugins from plugin jobs ...
[2023-10-16 15:29:38] - SCHEDULER - ℹ️ - Successfully updated database for the job mmdb-asn from plugin jobs
[2023-10-16 15:29:38] - BUNKERNET - ℹ️ - Database connection established
[2023-10-16 15:29:38] - BUNKERNET - ℹ️ - Successfully retrieved BunkerNet ID from db cache
[2023-10-16 15:29:38] - BUNKERNET - ℹ️ - Already registered on BunkerNet API with instance id 1ad38242-fd20-4751-9c3c-2bb25a4b9ca0
[2023-10-16 15:29:39] - BUNKERNET - ℹ️ - Checking connectivity with BunkerNet API ...
[2023-10-16 15:29:39] - BLACKLIST - ℹ️ - Database connection established
[2023-10-16 15:29:39] - BLACKLIST - ℹ️ - Blacklist for IP is already in cache, skipping downloads...
[2023-10-16 15:29:39] - BLACKLIST - ℹ️ - Blacklist for RDNS is not cached, processing downloads..
[2023-10-16 15:29:39] - BLACKLIST - ℹ️ - Blacklist for ASN is not cached, processing downloads..
[2023-10-16 15:29:39] - BLACKLIST - ℹ️ - Blacklist for USER_AGENT is not cached, processing downloads..
[2023-10-16 15:29:39] - BLACKLIST - ℹ️ - Blacklist for URI is not cached, processing downloads..
[2023-10-16 15:29:39] - BLACKLIST - ℹ️ - Blacklist for IGNORE_IP is not cached, processing downloads..
[2023-10-16 15:29:39] - BLACKLIST - ℹ️ - Blacklist for IGNORE_RDNS is not cached, processing downloads..
[2023-10-16 15:29:39] - BLACKLIST - ℹ️ - Blacklist for IGNORE_ASN is not cached, processing downloads..
[2023-10-16 15:29:39] - BLACKLIST - ℹ️ - Blacklist for IGNORE_USER_AGENT is not cached, processing downloads..
[2023-10-16 15:29:39] - BLACKLIST - ℹ️ - Blacklist for IGNORE_URI is not cached, processing downloads..
[2023-10-16 15:29:39] - BLACKLIST - ℹ️ - Downloading blacklist data from https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list ...
[2023-10-16 15:29:39] - JOBS.DOWNLOAD-PLUGINS - ℹ️ - No external plugins to download
[2023-10-16 15:29:39] - SCHEDULER - ℹ️ - Executing job crowdsec-conf from plugin crowdsec ...
[2023-10-16 15:29:39] - SCHEDULER - ℹ️ - Successfully updated database for the job download-plugins from plugin jobs
[2023-10-16 15:29:40] - LETS-ENCRYPT.NEW - ℹ️ - Certificates already exists for domain(s) admin.tirescue.com
[2023-10-16 15:29:40] - BLACKLIST - ℹ️ - Downloaded 647 bad USER_AGENT
[2023-10-16 15:29:40] - BLACKLIST - ℹ️ - New file USER_AGENT.list is identical to cache file, reload is not needed
[2023-10-16 15:29:40] - SCHEDULER - ℹ️ - Successfully updated database for the job blacklist-download from plugin blacklist
[2023-10-16 15:29:40] - BUNKERNET - ⚠️ - BunkerNet API is rate limiting us, trying again later...
[2023-10-16 15:29:40] - BUNKERNET - ❌ - Received invalid data from BunkerNet API while sending db request : rate limited, retrying later...
[2023-10-16 15:29:40] - SCHEDULER - ℹ️ - Executing job bunkernet-data from plugin bunkernet ...
[2023-10-16 15:29:41] - SCHEDULER - ℹ️ - Successfully updated database for the job bunkernet-register from plugin bunkernet
[2023-10-16 15:29:41] - CROWDSEC - ℹ️ - Database connection established
[2023-10-16 15:29:41] - CROWDSEC - ℹ️ - CrowdSec configuration successfully generated
[2023-10-16 15:29:41] - SCHEDULER - ℹ️ - Successfully updated database for the job crowdsec-conf from plugin crowdsec
[2023-10-16 15:29:42] - BUNKERNET - ℹ️ - Database connection established
[2023-10-16 15:29:42] - BUNKERNET - ℹ️ - No BunkerNet ID found in db cache
[2023-10-16 15:29:42] - BUNKERNET - ℹ️ - BunkerNet list is already in cache, skipping download...
[2023-10-16 15:29:42] - SCHEDULER - ℹ️ - Successfully updated database for the job bunkernet-data from plugin bunkernet
[2023-10-16 15:29:57] - LETS-ENCRYPT.NEW - ℹ️ - Certificates already exists for domain(s) xxxx.com
Hi @farzadha2, thank you for your logs, I'll investigate on this and let you know.
@farzadha2, I optimised certbot's renewal job and tested it, it looks promising. This will be available in the next 1.5.x
release.
hi @TheophileDiot Thank you so much for the reply, quick question is the version available in the test environment?
@farzadha2 sorry for the late response, the fix is present in the dev
and testing
tags.
hi @TheophileDiot Quick question if currently running 1.5.2 would i just change it to dev and recreate the container? would there be any issue?
Hi @farzadha2, the database model didn't change since the 1.5.2
so it wouldn't cause any problem. I would recommend using the testing
tag though rather than the dev one as it changes often 😁
Hello @farzadha2,
It should be fixed in the last 1.5.3 release. Don't hesitate to open a new issue if needed.
What's needed and why?
Hi Currently using webGUI and working great, but would be nice everytime i change one site it takes around 10mins to get applied, as i have 31 sites, so checking the logs it shows that its checking every other site of the SSL, not sure if there is another way to be faster?
Implementations ideas (optional)
I was thinking to only check the site that is getting applied
Code of Conduct