search

bunkerity / bunkerweb

🛡️ Open-source and next-generation Web Application Firewall (WAF)
https://www.bunkerweb.io
GNU Affero General Public License v3.0
6.12k stars 339 forks source link

[BUG] [INIT] not enough memory allocated to load data from /etc/nginx/proxies.list #244

Closed DimitryVo closed 1 year ago

DimitryVo commented 2 years ago

Description docker-compose logs shows error [INIT] not enough memory allocated to load data from /etc/nginx/proxies.list

How to reproduce docker-compose up -d

Logsautoconf_1 | [] Starting autoconf ... autoconf_1 | [2022-05-10 18:52:49] autoconf - INFO - docker mode detected autoconf_1 | [2022-05-10 18:52:49] autoconf - INFO - wait until a bunkerized-nginx instance is started ... autoconf_1 | [2022-05-10 18:52:50] config - INFO - generator output : [] Generation done ! autoconf_1 | autoconf_1 | [2022-05-10 18:53:19] autoconf - INFO - bunkerized-nginx instances started autoconf_1 | [2022-05-10 18:53:19] autoconf - INFO - waiting for events ... autoconf_1 | [2022-05-10 18:55:28] controller - INFO - generating new configuration autoconf_1 | [2022-05-10 18:55:28] config - INFO - generator output : [] Generation done ! autoconf_1 | autoconf_1 | [2022-05-10 18:55:28] controller - INFO - successfully generated new configuration autoconf_1 | [2022-05-10 18:55:28] controller - INFO - successful reload autoconf_1 | [2022-05-10 18:55:28] controller - INFO - generating new configuration autoconf_1 | [2022-05-10 18:55:28] config - INFO - generator output : [] Generation done ! autoconf_1 | autoconf_1 | [2022-05-10 18:55:28] controller - INFO - successfully generated new configuration autoconf_1 | [2022-05-10 18:55:29] controller - INFO - successful reload autoconf_1 | [2022-05-10 18:58:05] controller - INFO - generating new configuration autoconf_1 | [2022-05-10 18:58:05] config - INFO - generator output : [] Generation done ! autoconf_1 | autoconf_1 | [2022-05-10 18:58:05] controller - INFO - successfully generated new configuration autoconf_1 | [2022-05-10 18:58:05] controller - INFO - successful reload whoami_1 | 192.168.64.3 - - [10/May/2022 18:55:38] "GET / HTTP/1.1" 200 - whoami_1 | 192.168.64.3 - - [10/May/2022 18:58:12] "GET / HTTP/1.1" 200 - whoami_1 | 192.168.64.3 - - [10/May/2022 18:58:28] "GET / HTTP/1.1" 200 - nginx_1 | [2022-05-10 18:52:48] entrypoint - INFO - starting bunkerized-nginx ... nginx_1 | [2022-05-10 18:52:48] entrypoint - INFO - skipping configuration process nginx_1 | [2022-05-10 18:52:48] entrypoint - INFO - running nginx ... nginx_1 | 2022/05/10 18:52:48 [notice] 13#13: ModSecurity-nginx v1.0.2 (rules loaded inline/local/remote: 0/908/0) nginx_1 | 2022/05/10 18:52:48 [error] 13#13: [INIT] not enough memory allocated to load data from /etc/nginx/proxies.list nginx_1 | nginx: [error] [INIT] not enough memory allocated to load data from /etc/nginx/proxies.list nginx_1 | 2022/05/10 18:52:48 [error] 13#13: [INIT] NOT AN ERROR loaded 2036840 IPs from /etc/nginx/proxies.list nginx_1 | nginx: [error] [INIT] NOT AN ERROR loaded 2036840 IPs from /etc/nginx/proxies.list nginx_1 | 2022/05/10 18:52:48 [error] 13#13: [INIT] NOT AN ERROR loaded 200794 IPs from /etc/nginx/abusers.list nginx_1 | nginx: [error] [INIT] NOT AN ERROR loaded 200794 IPs from /etc/nginx/abusers.list nginx_1 | 2022/05/10 18:52:48 [error] 13#13: [INIT] NOT AN ERROR loaded 1204 IPs from /etc/nginx/tor-exit-nodes.list nginx_1 | nginx: [error] [INIT] NOT AN ERROR loaded 1204 IPs from /etc/nginx/tor-exit-nodes.list nginx_1 | 2022/05/10 18:52:48 [error] 13#13: [INIT] NOT AN ERROR loaded 2027 entries from /etc/nginx/user-agents.list nginx_1 | nginx: [error] [INIT] NOT AN ERROR loaded 2027 entries from /etc/nginx/user-agents.list nginx_1 | 2022/05/10 18:52:48 [error] 13#13: [INIT] NOT AN ERROR loaded 7095 entries from /etc/nginx/referrers.list nginx_1 | nginx: [error] [INIT] NOT AN ERROR* loaded 7095 entries from /etc/nginx/referrers.list

fl0ppy-d1sk commented 2 years ago

Should be fixed in the new 1.4.0 version. Please note that you can increase the DATASTORE_MEMORY_SIZE setting if that's happening now.

rmrf-sl4sh commented 1 year ago

this happens when I add to env vars:

ignoring DATASTORE_MEMORY_SIZE=2048MB (doesn't exist)

TheophileDiot commented 1 year ago

Hi @rmrf-sl4sh, if you look at the documentation, MB is not an accepted value for nginx. I invite you to have a look at the accepted values for nginx. (with the new version it's gonna be way easier as we implemented regex for a lot of settings, so you would know what you did wrong)

rmrf-sl4sh commented 1 year ago

Hi @rmrf-sl4sh, if you look at the documentation, MB is not an accepted value for nginx. I invite you to have a look at the accepted values for nginx. (with the new version it's gonna be way easier as we implemented regex for a lot of settings, so you would know what you did wrong)

Hi thank you for your reply, I see on the linked page to nginx accepted values, there are the specific ways to denote sizes. Using this I have changed DATASTORE_MEMORY_SIZE=2048MB to DATASTORE_MEMORY_SIZE=2g, but still receive this error:

entrypoint - INFO - generator output : ignoring DATASTORE_MEMORY_SIZE=2g (doesn't exist)

I have tried DATASTORE_MEMORY_SIZE=2048M, DATASTORE_MEMORY_SIZE=2048m, and DATASTORE_MEMORY_SIZE=2G, as well, but still have this problem.

Not sure what's going on here, but further down in the [INIT] I also see this, which is confusing:

bunkerity_1 | 2023/02/21 10:55:43 [error] 120#120: [INIT] not enough memory allocated to load data from /etc/nginx/proxies.list bunkerity_1 | nginx: [error] [INIT] not enough memory allocated to load data from /etc/nginx/proxies.list bunkerity_1 | 2023/02/21 10:55:43 [error] 120#120: [INIT] *NOT AN ERROR* loaded 2036296 IPs from /etc/nginx/proxies.list bunkerity_1 | nginx: [error] [INIT] *NOT AN ERROR* loaded 2036296 IPs from /etc/nginx/proxies.list

Thank you for your help with this

TheophileDiot commented 1 year ago

Hi @rmrf-sl4sh, I will investigate this.

TheophileDiot commented 1 year ago

@rmrf-sl4sh From what I can see we don't yet support the g and G suffixes but the others we do, you just have to allocate a sufficient amount of RAM accordingly.

fl0ppy-d1sk commented 1 year ago

Hello @rmrf-sl4sh,

DATASTORE_MEMORY_SIZE is used by the lua_shared_dict directive : https://github.com/openresty/lua-nginx-module#lua_shared_dict. Quoting their docs : The <size> argument accepts size units such as k and m.

I can see the /etc/nginx/proxies.list path in your logs and the ignoring DATASTORE_MEMORY_SIZE error. Can you tell us which version do you use ? And give us more info about your config (compose, env var, ...) ?