search

bunkerity / bunkerweb

🛡️ Open-source and next-generation Web Application Firewall (WAF)
https://www.bunkerweb.io
GNU Affero General Public License v3.0
6.12k stars 339 forks source link

[FEATURE] Slow on applying chances on webGUI #694

Closed farzadha2 closed 10 months ago

farzadha2 commented 10 months ago

What's needed and why?

Hi Currently using webGUI and working great, but would be nice everytime i change one site it takes around 10mins to get applied, as i have 31 sites, so checking the logs it shows that its checking every other site of the SSL, not sure if there is another way to be faster?

Implementations ideas (optional)

I was thinking to only check the site that is getting applied

Code of Conduct

TheophileDiot commented 10 months ago

Hi @farzadha2, thank you for opening this feature request. Are you certain that the problem is only caused by the let's encrypt plugin ? Could you provide us with your log if you're not sure ? Thank you.

farzadha2 commented 10 months ago

hi @TheophileDiot thanks for the reply, Sure thing im not sure if its normal but when doing any changes on the WebGUI it takes around 10mins as it seems to reapply everything

as this is the log, as the letsencrypt goes to each domain to see if it needs to renew 


    flush_context.execute()

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/unitofwork.py", line 466, in execute

    rec.execute(self)

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/unitofwork.py", line 642, in execute

    util.preloaded.orm_persistence.save_obj(

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/persistence.py", line 85, in save_obj

    _emit_update_statements(

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/persistence.py", line 909, in _emit_update_statements

    c = connection.execute(

        ^^^^^^^^^^^^^^^^^^^

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1412, in execute

    return meth(

           ^^^^^

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection

    return connection._execute_clauseelement(

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1635, in _execute_clauseelement

    ret = self._execute_context(

          ^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1844, in _execute_context

    return self._exec_single_context(

           ^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1984, in _exec_single_context

    self._handle_dbapi_exception(

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 2339, in _handle_dbapi_exception

    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1965, in _exec_single_context

    self.dialect.do_execute(

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/default.py", line 921, in do_execute

    cursor.execute(statement, parameters)

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked

[SQL: UPDATE bw_jobs SET last_run=? WHERE bw_jobs.name = ?]

[parameters: ('2023-10-16 15:29:08.573519', 'greylist-download')]

(Background on this error at: https://sqlalche.me/e/20/e3q8)

[2023-10-16 15:29:16] - SCHEDULER - ℹ️  - Custom configs changed, generating ...

[2023-10-16 15:29:16] - SCHEDULER - ℹ️  - Config changed, generating ...

[2023-10-16 15:29:16] - SCHEDULER - ℹ️  - Removing old custom configs files ...

[2023-10-16 15:29:16] - SCHEDULER - ℹ️  - Generating new custom configs ...

[2023-10-16 15:29:16] - SCHEDULER - ℹ️  - Sending custom configs to BunkerWeb

[2023-10-16 15:29:16] - API - ℹ️  - Successfully sent API request to http://sites_mybunker_1:5000/custom_configs

[2023-10-16 15:29:20] - SCHEDULER - ℹ️  - Executing job custom-cert from plugin customcert ...

[2023-10-16 15:29:20] - SCHEDULER - ℹ️  - Executing job realip-download from plugin realip ...

[2023-10-16 15:29:20] - SCHEDULER - ℹ️  - Executing job certbot-new from plugin letsencrypt ...

[2023-10-16 15:29:20] - SCHEDULER - ℹ️  - Executing job self-signed from plugin selfsigned ...

[2023-10-16 15:29:21] - SCHEDULER - ℹ️  - Executing job mmdb-country from plugin jobs ...

[2023-10-16 15:29:21] - SCHEDULER - ℹ️  - Successfully updated database for the job custom-cert from plugin customcert

[2023-10-16 15:29:22] - LETS-ENCRYPT.NEW - ℹ️  - Database connection established

[2023-10-16 15:29:22] - REALIP - ℹ️  - RealIP is not activated, skipping download...

[2023-10-16 15:29:22] - SCHEDULER - ℹ️  - Executing job default-server-cert from plugin misc ...

[2023-10-16 15:29:22] - SCHEDULER - ℹ️  - Successfully updated database for the job realip-download from plugin realip

[2023-10-16 15:29:22] - SCHEDULER - ℹ️  - Executing job whitelist-download from plugin whitelist ...

[2023-10-16 15:29:22] - SCHEDULER - ℹ️  - Successfully updated database for the job self-signed from plugin selfsigned

[2023-10-16 15:29:22] - LETS-ENCRYPT.NEW - ℹ️  - Successfully retrieved Let's Encrypt data from db cache

[2023-10-16 15:29:24] - JOBS.MMDB-COUNTRY - ℹ️  - Database connection established

[2023-10-16 15:29:24] - DEFAULT-SERVER-CERT - ℹ️  - Skipping generation of self-signed certificate for default server (already present)

[2023-10-16 15:29:24] - WHITELIST - ℹ️  - Database connection established

[2023-10-16 15:29:24] - JOBS.MMDB-COUNTRY - ℹ️  - country.mmdb is already in cache, skipping download...

[2023-10-16 15:29:24] - SCHEDULER - ℹ️  - Executing job mmdb-asn from plugin jobs ...

[2023-10-16 15:29:24] - WHITELIST - ℹ️  - Whitelist for IP is not cached, processing downloads..

[2023-10-16 15:29:24] - WHITELIST - ℹ️  - Whitelist for RDNS is not cached, processing downloads..

[2023-10-16 15:29:24] - WHITELIST - ℹ️  - Whitelist for ASN is not cached, processing downloads..

[2023-10-16 15:29:24] - WHITELIST - ℹ️  - Whitelist for USER_AGENT is not cached, processing downloads..

[2023-10-16 15:29:24] - WHITELIST - ℹ️  - Whitelist for URI is not cached, processing downloads..

[2023-10-16 15:29:24] - SCHEDULER - ℹ️  - Successfully updated database for the job mmdb-country from plugin jobs

[2023-10-16 15:29:25] - SCHEDULER - ℹ️  - Executing job update-check from plugin misc ...

[2023-10-16 15:29:25] - SCHEDULER - ℹ️  - Successfully updated database for the job default-server-cert from plugin misc

[2023-10-16 15:29:25] - SCHEDULER - ℹ️  - Successfully updated database for the job whitelist-download from plugin whitelist

[2023-10-16 15:29:25] - SCHEDULER - ℹ️  - Executing job greylist-download from plugin greylist ...

[2023-10-16 15:29:27] - GREYLIST - ℹ️  - Greylist is not activated, skipping downloads...

[2023-10-16 15:29:27] - SCHEDULER - ℹ️  - Executing job bunkernet-register from plugin bunkernet ...

[2023-10-16 15:29:28] - UPDATE-CHECK - ℹ️  - Latest version is already installed: v1.5.2

[2023-10-16 15:29:28] - SCHEDULER - ℹ️  - Executing job blacklist-download from plugin blacklist ...

[2023-10-16 15:29:32] - SCHEDULER - ⚠️  - Failed to update database for the job greylist-download from plugin greylist: Traceback (most recent call last):

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1965, in _exec_single_context

    self.dialect.do_execute(

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/default.py", line 921, in do_execute

    cursor.execute(statement, parameters)

sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  File "/usr/share/bunkerweb/db/Database.py", line 1035, in update_job

    session.commit()

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/session.py", line 1923, in commit

    trans.commit(_to_root=True)

  File "<string>", line 2, in commit

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/state_changes.py", line 139, in _go

    ret_value = fn(self, *arg, **kw)

                ^^^^^^^^^^^^^^^^^^^^

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/session.py", line 1239, in commit

    self._prepare_impl()

  File "<string>", line 2, in _prepare_impl

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/state_changes.py", line 139, in _go

    ret_value = fn(self, *arg, **kw)

                ^^^^^^^^^^^^^^^^^^^^

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/session.py", line 1214, in _prepare_impl

    self.session.flush()

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/session.py", line 4179, in flush

    self._flush(objects)

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/session.py", line 4314, in _flush

    with util.safe_reraise():

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/util/langhelpers.py", line 147, in __exit__

    raise exc_value.with_traceback(exc_tb)

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/session.py", line 4275, in _flush

    flush_context.execute()

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/unitofwork.py", line 466, in execute

    rec.execute(self)

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/unitofwork.py", line 642, in execute

    util.preloaded.orm_persistence.save_obj(

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/persistence.py", line 85, in save_obj

    _emit_update_statements(

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/orm/persistence.py", line 909, in _emit_update_statements

    c = connection.execute(

        ^^^^^^^^^^^^^^^^^^^

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1412, in execute

    return meth(

           ^^^^^

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection

    return connection._execute_clauseelement(

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1635, in _execute_clauseelement

    ret = self._execute_context(

          ^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1844, in _execute_context

    return self._exec_single_context(

           ^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1984, in _exec_single_context

    self._handle_dbapi_exception(

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 2339, in _handle_dbapi_exception

    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/base.py", line 1965, in _exec_single_context

    self.dialect.do_execute(

  File "/usr/share/bunkerweb/deps/python/sqlalchemy/engine/default.py", line 921, in do_execute

    cursor.execute(statement, parameters)

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked

[SQL: UPDATE bw_jobs SET last_run=? WHERE bw_jobs.name = ?]

[parameters: ('2023-10-16 15:29:27.153031', 'greylist-download')]

(Background on this error at: https://sqlalche.me/e/20/e3q8)

[2023-10-16 15:29:33] - JOBS.MMDB-ASN - ⚠️  - Can't connect to database, retrying in 5 seconds ...

[2023-10-16 15:29:33] - BUNKERNET - ⚠️  - Can't connect to database, retrying in 5 seconds ...

[2023-10-16 15:29:34] - BLACKLIST - ⚠️  - Can't connect to database, retrying in 5 seconds ...

[2023-10-16 15:29:34] - SCHEDULER - ℹ️  - Successfully updated database for the job update-check from plugin misc

[2023-10-16 15:29:38] - JOBS.MMDB-ASN - ℹ️  - Database connection established

[2023-10-16 15:29:38] - JOBS.MMDB-ASN - ℹ️  - asn.mmdb is already in cache, skipping download...

[2023-10-16 15:29:38] - SCHEDULER - ℹ️  - Executing job download-plugins from plugin jobs ...

[2023-10-16 15:29:38] - SCHEDULER - ℹ️  - Successfully updated database for the job mmdb-asn from plugin jobs

[2023-10-16 15:29:38] - BUNKERNET - ℹ️  - Database connection established

[2023-10-16 15:29:38] - BUNKERNET - ℹ️  - Successfully retrieved BunkerNet ID from db cache

[2023-10-16 15:29:38] - BUNKERNET - ℹ️  - Already registered on BunkerNet API with instance id 1ad38242-fd20-4751-9c3c-2bb25a4b9ca0

[2023-10-16 15:29:39] - BUNKERNET - ℹ️  - Checking connectivity with BunkerNet API ...

[2023-10-16 15:29:39] - BLACKLIST - ℹ️  - Database connection established

[2023-10-16 15:29:39] - BLACKLIST - ℹ️  - Blacklist for IP is already in cache, skipping downloads...

[2023-10-16 15:29:39] - BLACKLIST - ℹ️  - Blacklist for RDNS is not cached, processing downloads..

[2023-10-16 15:29:39] - BLACKLIST - ℹ️  - Blacklist for ASN is not cached, processing downloads..

[2023-10-16 15:29:39] - BLACKLIST - ℹ️  - Blacklist for USER_AGENT is not cached, processing downloads..

[2023-10-16 15:29:39] - BLACKLIST - ℹ️  - Blacklist for URI is not cached, processing downloads..

[2023-10-16 15:29:39] - BLACKLIST - ℹ️  - Blacklist for IGNORE_IP is not cached, processing downloads..

[2023-10-16 15:29:39] - BLACKLIST - ℹ️  - Blacklist for IGNORE_RDNS is not cached, processing downloads..

[2023-10-16 15:29:39] - BLACKLIST - ℹ️  - Blacklist for IGNORE_ASN is not cached, processing downloads..

[2023-10-16 15:29:39] - BLACKLIST - ℹ️  - Blacklist for IGNORE_USER_AGENT is not cached, processing downloads..

[2023-10-16 15:29:39] - BLACKLIST - ℹ️  - Blacklist for IGNORE_URI is not cached, processing downloads..

[2023-10-16 15:29:39] - BLACKLIST - ℹ️  - Downloading blacklist data from https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents.list ...

[2023-10-16 15:29:39] - JOBS.DOWNLOAD-PLUGINS - ℹ️  - No external plugins to download

[2023-10-16 15:29:39] - SCHEDULER - ℹ️  - Executing job crowdsec-conf from plugin crowdsec ...

[2023-10-16 15:29:39] - SCHEDULER - ℹ️  - Successfully updated database for the job download-plugins from plugin jobs

[2023-10-16 15:29:40] - LETS-ENCRYPT.NEW - ℹ️  - Certificates already exists for domain(s) admin.tirescue.com

[2023-10-16 15:29:40] - BLACKLIST - ℹ️  - Downloaded 647 bad USER_AGENT

[2023-10-16 15:29:40] - BLACKLIST - ℹ️  - New file USER_AGENT.list is identical to cache file, reload is not needed

[2023-10-16 15:29:40] - SCHEDULER - ℹ️  - Successfully updated database for the job blacklist-download from plugin blacklist

[2023-10-16 15:29:40] - BUNKERNET - ⚠️  - BunkerNet API is rate limiting us, trying again later...

[2023-10-16 15:29:40] - BUNKERNET - ❌ - Received invalid data from BunkerNet API while sending db request : rate limited, retrying later...

[2023-10-16 15:29:40] - SCHEDULER - ℹ️  - Executing job bunkernet-data from plugin bunkernet ...

[2023-10-16 15:29:41] - SCHEDULER - ℹ️  - Successfully updated database for the job bunkernet-register from plugin bunkernet

[2023-10-16 15:29:41] - CROWDSEC - ℹ️  - Database connection established

[2023-10-16 15:29:41] - CROWDSEC - ℹ️  - CrowdSec configuration successfully generated

[2023-10-16 15:29:41] - SCHEDULER - ℹ️  - Successfully updated database for the job crowdsec-conf from plugin crowdsec

[2023-10-16 15:29:42] - BUNKERNET - ℹ️  - Database connection established

[2023-10-16 15:29:42] - BUNKERNET - ℹ️  - No BunkerNet ID found in db cache

[2023-10-16 15:29:42] - BUNKERNET - ℹ️  - BunkerNet list is already in cache, skipping download...

[2023-10-16 15:29:42] - SCHEDULER - ℹ️  - Successfully updated database for the job bunkernet-data from plugin bunkernet

[2023-10-16 15:29:57] - LETS-ENCRYPT.NEW - ℹ️  - Certificates already exists for domain(s) xxxx.com
TheophileDiot commented 10 months ago

Hi @farzadha2, thank you for your logs, I'll investigate on this and let you know.

TheophileDiot commented 10 months ago

@farzadha2, I optimised certbot's renewal job and tested it, it looks promising. This will be available in the next 1.5.x release.

farzadha2 commented 10 months ago

hi @TheophileDiot Thank you so much for the reply, quick question is the version available in the test environment?

TheophileDiot commented 10 months ago

@farzadha2 sorry for the late response, the fix is present in the dev and testing tags.

farzadha2 commented 10 months ago

hi @TheophileDiot Quick question if currently running 1.5.2 would i just change it to dev and recreate the container? would there be any issue?

TheophileDiot commented 10 months ago

Hi @farzadha2, the database model didn't change since the 1.5.2 so it wouldn't cause any problem. I would recommend using the testing tag though rather than the dev one as it changes often 😁

fl0ppy-d1sk commented 10 months ago

Hello @farzadha2,

It should be fixed in the last 1.5.3 release. Don't hesitate to open a new issue if needed.