Closed nasirbas1 closed 4 years ago
zimmerle
commented
4 years ago Hi @nasirbas1,
Please make sure that nignx/connector is using the library that you have compiled with lmdb support. It is very likely that your webserver/connector is using the old library.
nasirbas1
commented
4 years ago @zimmerle for the leakage part will requests from blocked IP's stop leaking through with lmdb because with smdb even after hours of blocking some of the request still go through.
zimmerle
commented
4 years ago @nasirbas1 the manner to block the request (timeout, number of attempts, block time, etc...) should consider that the counter increment is not atomic. There may be a difference from multiprocess sync. Regardless, from my experience, the lmdb sync is quite fast. 1 hour without a sync, sounds to me that you are still using the memory collection still.
Regardless, if your intention is to block a given IP address, there are different manners to achieve such functionality in an atomic manner, such us: rbl
nasirbas1
commented
4 years ago @zimmerle Regarding your earlier comment "Please make sure that nignx/connector is using the library that you have compiled with lmdb support. It is very likely that your webserver/connector is using the old library."
I don't thinks this is the case , I removed all the symlinks and folders , basically recompiled everything and its still not working . I think it has taken up the storage mode , because earlier with smdb, atleast it used to set the variable on IP collection but now setvar does happen, but when I try to retrieve the value in next request its blank . There seems to be no place to view exactly whats going wrong ? Is there any way to debug this?
zimmerle
commented
4 years ago you can enable ModSecurity debug logs. You will see the actual value of each variable upon resolution time. you can also set the numbers of workers to 1 to test if the problem is within the workers sync.
nasirbas1
commented
4 years ago @zimmerle So I have already turned the debug logs at level 9. Its correctly setting the variable , but I think its not able to persist it to the lmdb because on the very next request the value previously set in the variable is blank .... The same works perfectly, when I compile it with smdb . There seems to be no entry in the logs related to the persistance of values to lmdb
zimmerle
commented
4 years ago if you don't have multiple workers nor restart the webserver the value should be persistent regardless of the collection backend. Can you share the rules that you are using?
nasirbas1
commented
4 years ago @zimmerle I do have multiple workers Please find below the rules
Initialize IP Collection using the IP address which comes as True-IP from load-balancer
SecRule REQUEST_HEADERS:True-IP "^\b(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\b" "id:660001,phase:1,t:none,pass,log,capture,setvar:tx.client_ip=%{tx.1}" SecAction "phase:1,nolog,pass,initcol:ip=%{tx.client_ip}"
logic to block any ip basically sets IP.blocked = 2
SecRule REQUEST_URI "@contains /ip/blacklist" "id:440001,chain,phase:1,t:none,deny,log,status:200" SecRule REMOTE_ADDR "@ipMatch 127.0.0.1" "chain,t:none" SecRule ARGS:ip "^\b(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\b" "t:none,setvar:ip.allowed=2"
Checks if allowed variable on IP collections for this IP is 2 then drops this request
SecRule IP:ALLOWED "@eq 2" "id:880001,phase:1,t:none,drop,log,logdata:'Dynamic Blacklist'"
zimmerle
commented
4 years ago What is the patch to your lmdb file? is it consistent? can you keep only a single worker for testing?
nasirbas1
commented
4 years ago SecDataDir /etc/nginx/modsec/storage/
Have added all the permissions , so the user has all the permissions to write to this folder
If you meant version its 2.9.1
+ LMDB ....found v2.9.1 -llmdb , -DWITH_LMDB
P.S. Apart from ./configure --with-lmdb & SecDataDir /etc/nginx/modsec/storage/ , do we need to do anything else to turn on lmdb support?
zimmerle
commented
4 years ago please check the content and size of the files....
nasirbas1
commented
4 years ago @zimmerle which files ... the ones which lmdb is supposed to create ? like ip.pag and ip.dir ? If so no such file exists
zimmerle
commented
4 years ago So, the support to lmdb does not seems to be enabled. let's figure out why....
Please paste the results of:
ldd /usr/local/nginx/sbin/nginx | grep mod(change the path to nginx binary to match to your installation)
nasirbas1
commented
4 years ago ldd /usr/local/nginx/sbin/nginx returned this
[root@ip-10-0-1-242 sbin]# ldd /usr/local/nginx/sbin/nginx linux-vdso.so.1 => (0x00007ffd07db5000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f84ff51c000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f84ff300000) libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f84ff0c9000) libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f84fee67000) libz.so.1 => /lib64/libz.so.1 (0x00007f84fec51000) libc.so.6 => /lib64/libc.so.6 (0x00007f84fe883000) /lib64/ld-linux-x86-64.so.2 (0x00007f84ff720000) libfreebl3.so => /lib64/libfreebl3.so (0x00007f84fe680000)
grep mod didnt have anything to return . OS Version : CentOS Linux 7 Nginx Version : 1.16.1 Modsecurity version : 3.03
zimmerle
commented
4 years ago how have you compiled the connector?
nasirbas1
commented
4 years ago @zimmerle I didn't find any instructions to compile the connector anywhere We have used the below steps for compilation
Step 1 -> Installing dependencies apt-get update && apt-get install -y automake bison build-essential g++ gcc libbison-dev libcurl4-openssl-dev libfl-dev libgeoip-dev liblmdb-dev libpcre3-dev libtool libxml2-dev libyajl-dev make pkg-config zlib1g-dev
Step 2 -> Compiling Modsecurity & LMDB
LMDB a) cd /opt/ b) git clone git clone https://github.com/LMDB/lmdb.git c) cd lmdb/libraries/liblmdb d) make e) make install
Modsecurity a) git clone --depth 1 -b v3/master --single-branch https://github.com/SpiderLabs/ModSecurity b) cd ModSecurity c) git submodule init d) git submodule update e) ./build.sh f) ./configure --with-lmdb g) make h) make install
Step 3 -> Building nginx with libmodsecurity a) git clone --depth 1 https://github.com/SpiderLabs/ModSecurity-nginx.git b) wget http://nginx.org/download/nginx-1.16.1.tar.gz c) tar -zxf nginx-1.16.1.tar.gz d) cd nginx-1.16.1/ e) ./configure --add-dynamic-module=../ModSecurity-nginx --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-file-aio --with-ipv6 --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-stream_ssl_preread_module --with-http_addition_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_degradation_module --with-http_slice_module --with-http_stub_status_module --with-http_perl_module=dynamic --with-http_auth_request_module --with-mail=dynamic --with-mail_ssl_module --with-pcre --with-pcre-jit --with-stream=dynamic --with-stream_ssl_module --with-google_perftools_module --with-debug --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --with-ld-opt='-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E' f) make modules
Step 4 -> Copy the modsecurity .so file cp objs/ngx_http_modsecurity_module.so /etc/nginx/modules/ cd /etc/nginx/ load_module modules/ngx_http_modsecurity_module.so;
Is there anything we have missed anywhere , because specifically compiling the connector was not present anywhere?
airween
commented
4 years ago I think that's a wrong way, LMDB was linked to libmodsecurity, not for the Nginx or the Nginx module.
You can check that is linked or not with this command:
ldd /path/to/libmodsecurity.so | grep lmdband you have to see something like this:
liblmdb.so.0 => /usr/lib/x86_64-linux-gnu/liblmdb.so.0 (0x00007f968465e000)If the output of the command is empty, that means lmdb wasn't linked to libmodsecurity.
nasirbas1
commented
4 years ago @airween How is the linking to be done ?
airween
commented
4 years ago You don't need to care with linking, if you pass the --with-lmdb the build system will do that.
What's the output of the command above?
nasirbas1
commented
4 years ago @airween I did run the configure script with lmdb flag and this is the complete output
checking for a BSD-compatible install... /bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking for g++... g++ checking whether the C++ compiler works... yes checking for C++ compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C++ compiler... yes checking whether g++ accepts -g... yes checking for style of include used by make... GNU checking dependency style of g++... gcc3 checking for gcc... gcc checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking dependency style of gcc... gcc3 checking for ar... ar checking the archiver (ar) interface... ar checking whether make sets $(MAKE)... (cached) yes checking for pkg-config... /bin/pkg-config checking pkg-config is at least version 0.9.0... yes configure: Nothing about GeoIP was informed during the configure phase. Trying to detect it on the platform... configure: using YAJL v2.0.4 configure: Nothing about GeoIP was informed during the configure phase. Trying to detect it on the platform... configure: using GeoIP v1.5.0 configure: Nothing about MaxMind was informed during the configure phase. Trying to detect it on the platform... configure: MaxMind library was not found configure: LMDB support was marked as mandatory by the utilization of --with-lmdb=yes configure: Nothing about LMDB was informed during the configure phase. Trying to detect it on the platform... LOOKING AT PATH: yes configure: using LMDB v0.9.22 LOOKING AT PATH: /usr/lib LOOKING AT PATH: /usr/local/lib LOOKING AT PATH: /usr/local/fuzzy LOOKING AT PATH: /usr/local/libfuzzy LOOKING AT PATH: /usr/local LOOKING AT PATH: /opt LOOKING AT PATH: /usr LOOKING AT PATH: /usr/lib64 LOOKING AT PATH: /opt/local configure: SSDEEP library was not found LOOKING AT PATH: /usr/lib LOOKING AT PATH: /usr/local/lib LOOKING AT PATH: /usr/local/lib64 LOOKING AT PATH: /usr/local/lua LOOKING AT PATH: /usr/local/liblua LOOKING AT PATH: /usr/local LOOKING AT PATH: /opt LOOKING AT PATH: /usr LOOKING AT PATH: /usr/lib64 configure: LUA library found at: /usr/lib64//liblua-5.1.so LOOKING AT PATH: /opt/local configure: LUA library found at: /usr/lib64//liblua-5.1.so configure: LUA library was not found checking for libcurl config script... /usr/bin/curl-config configure: curl VERSION: 7.29.0 configure: curl LDADD: checking if libcurl is at least v... yes, 7.29.0 checking if libcurl is linked with gnutls... no configure: using curl v7.29.0 checking for libxml2 config script... /usr/bin/xml2-config configure: xml VERSION: 2.9.1 configure: xml CFLAGS: -I/usr/include/libxml2 -DWITH_LIBXML2 configure: xml LDADD: -lxml2 -lz -lm -ldl checking if libxml2 is at least v2.6.29... yes, 2.9.1 configure: using libxml2 v2.9.1 checking for libpcre config script... /usr/bin/pcre-config configure: pcre VERSION: 8.32 configure: pcre LDADD: -lpcre configure: pcre PCRE_LD_PATH: /-lpcre checking for PCRE JIT... yes configure: using pcre v8.32 checking how to run the C preprocessor... gcc -E checking for grep that handles long lines and -e... /bin/grep checking for egrep... /bin/grep -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking string usability... no checking string presence... no checking for string... no checking iostream usability... no checking iostream presence... no checking for iostream... no checking sys/utsname.h usability... yes checking sys/utsname.h presence... yes checking for sys/utsname.h... yes checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking how to print strings... printf checking for a sed that does not truncate output... /bin/sed checking for fgrep... /bin/grep -F checking for ld used by gcc... /bin/ld checking if the linker (/bin/ld) is GNU ld... yes checking for BSD- or MS-compatible name lister (nm)... /bin/nm -B checking the name lister (/bin/nm -B) interface... BSD nm checking whether ln -s works... yes checking the maximum length of command line arguments... 1572864 checking whether the shell understands some XSI constructs... yes checking whether the shell understands "+="... yes checking how to convert x86_64-unknown-linux-gnu file names to x86_64-unknown-linux-gnu format... func_convert_file_noop checking how to convert x86_64-unknown-linux-gnu file names to toolchain format... func_convert_file_noop checking for /bin/ld option to reload object files... -r checking for objdump... objdump checking how to recognize dependent libraries... pass_all checking for dlltool... no checking how to associate runtime and link libraries... printf %s\n checking for archiver @FILE support... @ checking for strip... strip checking for ranlib... ranlib checking command to parse /bin/nm -B output from gcc object... ok checking for sysroot... no checking for mt... no checking if : is a manifest tool... no checking for dlfcn.h... yes checking for objdir... .libs checking if gcc supports -fno-rtti -fno-exceptions... no checking for gcc option to produce PIC... -fPIC -DPIC checking if gcc PIC flag -fPIC -DPIC works... yes checking if gcc static flag -static works... no checking if gcc supports -c -o file.o... yes checking if gcc supports -c -o file.o... (cached) yes checking whether the gcc linker (/bin/ld -m elf_x86_64) supports shared libraries... yes checking whether -lc should be explicitly linked in... no checking dynamic linker characteristics... GNU/Linux ld.so checking how to hardcode library paths into programs... immediate checking for shl_load... no checking for shl_load in -ldld... no checking for dlopen... no checking for dlopen in -ldl... yes checking whether a program can dlopen itself... yes checking whether a statically linked program can dlopen itself... yes checking whether stripping libraries is possible... yes checking if libtool supports shared libraries... yes checking whether to build shared libraries... yes checking whether to build static libraries... yes checking how to run the C++ preprocessor... g++ -E checking for ld used by g++... /bin/ld -m elf_x86_64 checking if the linker (/bin/ld -m elf_x86_64) is GNU ld... yes checking whether the g++ linker (/bin/ld -m elf_x86_64) supports shared libraries... yes checking for g++ option to produce PIC... -fPIC -DPIC checking if g++ PIC flag -fPIC -DPIC works... yes checking if g++ static flag -static works... no checking if g++ supports -c -o file.o... yes checking if g++ supports -c -o file.o... (cached) yes checking whether the g++ linker (/bin/ld -m elf_x86_64) supports shared libraries... yes checking dynamic linker characteristics... (cached) GNU/Linux ld.so checking how to hardcode library paths into programs... immediate Checking platform... Identified as Linux checking for doxygen... /bin/doxygen checking for perl... /bin/perl checking for valgrind... no checking that generated files are newer than configure... done configure: creating ./config.status config.status: creating modsecurity.pc config.status: creating Makefile config.status: creating doc/Makefile config.status: creating src/Makefile config.status: creating others/Makefile config.status: creating tools/Makefile config.status: creating tools/rules-check/Makefile config.status: creating test/Makefile config.status: creating test/benchmark/Makefile config.status: creating examples/Makefile config.status: creating examples/simple_example_using_c/Makefile config.status: creating examples/multiprocess_c/Makefile config.status: creating examples/reading_logs_with_offset/Makefile config.status: creating examples/reading_logs_via_rule_message/Makefile config.status: creating examples/using_bodies_in_chunks/Makefile config.status: creating src/config.h config.status: src/config.h is unchanged config.status: executing depfiles commands config.status: executing libtool commands
ModSecurity - v3.0.4 for Linux
Mandatory dependencies
SecLang tests ....c8cf2c5
Optional dependencies
LUA ....not found
Other Options
airween
commented
4 years ago Please, run this command, and share with us the result:
ldd /path/to/libmodsecurity.so | grep lmdbI assume your path is /usr/local/lib/libmodsecurity.so or /usr/local/modsecurity/lib/libmodsecurity.so. If you aren't sure, try this: find /usr -name "libmodsecurity.so".
nasirbas1
commented
4 years ago @airween
ldd /usr/local/modsecurity/lib/libmodsecurity.so | grep lmdb
liblmdb.so.0.0.0 => /lib64/liblmdb.so.0.0.0 (0x00007f40147b3000)
airween
commented
4 years ago Right, this means that the lmdb was successfully linked to your library.
Have you seen the modsec-shared-collections and modsec-shared-collections-lock files in your filesystem? Use the find command: find / -name "modsec-shared-collections". What's this output?
nasirbas1
commented
4 years ago @airween The command dumped following output
/etc/nginx/production/modsec-shared-collections /etc/nginx/modsec/modsec-shared-collections /etc/nginx/modules/modsec-shared-collections /etc/nginx/modsec-shared-collections /etc/nginx/modsec-shared-collections /etc/modsec-shared-collections /root/modsec-shared-collections /opt/nginx-1.16.1/modsec-shared-collections /modsec-shared-collections
airween
commented
4 years ago Well, that's a very impressive list :). Which one is the last modified? Could you rename all of them (eg. modsec-shared-collections to modsec-shared-collections.bak), and restart your nginx? Then try to find this file again. IMHO only one file used...
nasirbas1
commented
4 years ago @airween , So I renamed all the modsec-shared-collections and modsec-shared-collections-lock files . After restarting nginx it created a modsec-shared-collections file again in ' / ' (root) directory. How is this information to be used ? Because LMDB is still not working
airween
commented
4 years ago If the modsec-shared-collections file created after the nginx restart, then it means libmodsecurity uses it. You can check it some other way, eg. see the date of file after you sent a request which forces the engine it use the collection (through any rule), eg IP, ...
If it doesn't work that means your rules aren't doing what you expect.
nasirbas1
commented
4 years ago @airween What is the relation between modsec-shared-collections and LMDB storage ? I saw in articles that modsecurity creates ip.pag & ip.dir files for storage (the path to which is defined in SecDataDir)
ALSO Observations after running requests
1️⃣ On every such request which reads values from collection, the modsec-shared-collections-lock files gets updated (date changes).
2️⃣ On requests where we write to the collection (actually where we set IP.BLOCKED=1 ), both the modsec-shared-collections & the modsec-shared-collections-lock file gets updated (date changes)
3️⃣ But Immediately after setting BLOCKED flag on the IP to some value , I am unable to read the variable value from the collection for next requests from the same IP . Its blank .
Also the rules that we are using are as under
Initialize IP Collection using the IP address which comes as True-IP from load-balancer
_SecRule REQUEST_HEADERS:True-IP "^\b(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\b" "id:660001,phase:1,t:none,pass,log,capture,setvar:tx.client_ip=%{tx.1}" SecAction "phase:1,nolog,pass,initcol:ip=%{tx.clientip}"
Logic to block any ip basically sets IP.blocked = 2
_SecRule REQUEST_URI "@contains /ip/blacklist" "id:440001,chain,phase:1,t:none,deny,log,status:200" SecRule REMOTEADDR "@ipMatch 127.0.0.1" "chain,t:none" SecRule ARGS:ip "^\b(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\b" "t:none,setvar:ip.allowed=2"
Checks if allowed variable on IP collections for this IP is 2 then drops this request
SecRule IP:allowed "@eq 2" "id:880001,phase:1,t:none,drop,log,logdata:'Dynamic Blacklist'"
airween
commented
4 years ago hi @nasirbas1
@airween What is the relation between modsec-shared-collections and LMDB storage ?
LMDB stores the datas (key:value pairs) in modsec-shared-collections file. Here is a small tool, which can help you to dump its content:
https://gist.github.com/airween/09a9c8dd033eaf5fa317bc6da07c64a4
To compile see the comment before the included headers. Then you can use:
./lmdbread /path/to/modsec-shared-collections
I saw in articles that modsecurity creates ip.pag & ip.dir files for storage (the path to which is defined in SecDataDir)
and does those files exists?
ALSO Observations after running requests
one On every such request which reads values from collection, the modsec-shared-collections-lock files gets updated (date changes).
correct.
two On requests where we write to the collection (actually where we set IP.BLOCKED=1 ), both the modsec-shared-collections & the modsec-shared-collections-lock file gets updated (date changes)
correct.
three But Immediately after setting BLOCKED flag on the IP to some value , I am unable to read the variable value from the collection for next requests from the same IP . Its blank .
Also the rules that we are using are as under
Initialize IP Collection using the IP address which comes as True-IP from load-balancer
_SecRule REQUEST_HEADERS:True-IP "^\b(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\b" "id:660001,phase:1,t:none,pass,log,capture,setvar:tx.client_ip=%{tx.1}" SecAction "phase:1,nolog,pass,initcol:ip=%{tx.clientip}"
Logic to block any ip basically sets IP.blocked = 2
_SecRule REQUEST_URI "@contains /ip/blacklist" "id:440001,chain,phase:1,t:none,deny,log,status:200" SecRule REMOTEADDR "@ipMatch 127.0.0.1" "chain,t:none" SecRule ARGS:ip "^\b(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\b" "t:none,setvar:ip.allowed=2"
Checks if allowed variable on IP collections for this IP is 2 then drops this request
SecRule IP:ALLOWED "@eq 2" "id:880001,phase:1,t:none,drop,log,logdata:'Dynamic Blacklist'"
now I don't see the reason, I have to check it - I'll try to do that later.
nasirbas1
commented
4 years ago @airween No the ip.pag and ip.dir files don't exist on the system.
UPDATE 1 :
@airween
So I compiled the lmdbread tool and viewed the contents of the modsec-shared-collections
Modsecurity is indeed correctly storing the values to this file, and they are persisting after
restarting the nginx. So the only concern left is why is modsecurity not able to read these variables from this file then?
Output of the command is as under
./pgm/lmdbread modsec-shared-collections key: 0x7f4eda14bfe8 17.182.15.19::::allowed, data: 0x7f4eda14bfff 2
nasirbas1
commented
4 years ago @airween
UPDATE 2 It seems modsecurity and lmdb were working perfectly fine. The data was being stored perfectly as we intended it to. The catch here was that while writing to the IP collections we were writing it as
ip.allowed = 2
But while reading we were reading it as
SecRule IP:ALLOWED "@eq 2" "id:880001,phase:1,t:none,drop,log,logdata:'Dynamic Blacklist'"
So apparently Modsec rules are Case Sensitive 😝
P.S. Apparently I am still wondering how everything is working well, when ldd /usr/local/nginx/sbin/nginx | grep mod returned nothing .
airween
commented
4 years ago hi @nasirbas1,
UPDATE 2 It seems modsecurity and lmdb were working perfectly fine. The data was being stored perfectly as we intended it to. The catch here was that while writing to the IP collections we were writing it as
ip.allowed = 2
But while reading we were reading it as
SecRule IP:ALLOWED "@eq 2" "id:880001,phase:1,t:none,drop,log,logdata:'Dynamic Blacklist'"
So apparently Modsec rules are Case Sensitive stuck_out_tongue_closed_eyes
I think this is a bug, and there is a fix for that.
P.S. Apparently I am still wondering how everything is working well, when ldd /usr/local/nginx/sbin/nginx | grep mod returned nothing .
because you built the modsecurity as module.
nasirbas1
commented
4 years ago @airween Yeah thats pretty much it . We added modsecurity as a dynamic module hence the empty value. Nice thread for anyone in future mingling around modsecurity persistence ✌️
airween
commented
4 years ago Right - feel free to close the issue, if you think you don't have more question (for this topic :))
nasirbas1
commented
4 years ago @airween Already closed 👍
mmelo-yottaa
commented
4 years ago thanks @airween for the lmdb reader gist. I found using it deadlocks after initial run, I had to change the order of cursor/dbi/txn/env close. I am using LMDB 0.9.24: (July 24, 2019).
this works for me without deadlocks -
https://gist.github.com/mmelo-yottaa/8504a5bfc1d97c00c9015dae88159333.
For the curious, this was the deadlock:
NGINX+MODSEC:
Program received signal SIGINT, Interrupt. __pthread_mutex_lock_full (mutex=0x7ffff7ff3040) at ../nptl/pthread_mutex_lock.c:311 311 assume_other_futex_waiters |= FUTEX_WAITERS; (gdb) bt
at collection/backend/lmdb.cc:482ke=...) at ../headers/modsecurity/collection/collection.h:177LMDBREAD:
__pthread_mutex_lock_full (mutex=0x7ffff7ff7040) at ../nptl/pthread_mutex_lock.c:311 311 assume_other_futex_waiters |= FUTEX_WAITERS; (gdb) bt
airween
commented
4 years ago thanks @airween for the lmdb reader gist. I found using it deadlocks after initial run, I had to change the order of cursor/dbi/txn/env close. I am using LMDB 0.9.24: (July 24, 2019).
oh, I just created that small tool to read the filled collection database while I used libmodsecurity's regression test. Thank you for your notification.
Anyway, @defanator also has a fix to avoid the deadlock.
dvershinin
commented
10 months ago @airween isn't this utility which is part of lmdb main packaging already doing the job of dumping database contents?
airween
commented
10 months ago @airween isn't this utility which is part of
lmdbmain packaging already doing the job of dumping database contents?
Perhaps it is. I've never tested yet.
Hi @zimmerle ,
For persistant collections, we recompiled modsecurity with lmdb flag after installing lmdb, and the installation worked perfectly fine and lmdb showed up in the installed packages after running the ./configure --with-lmdb script
However lmdb is not working. Its not showing up in logs, nor are there any errors anywhere. Is there any way to verify the installation .
Also for SMDB installation , given the fact that every worker process has its own memory which they sync up with smdb at intervals, if any ip is blocked (say I set IP.BLOCKED=1 and deny it after checking this BLOCKED flag) , why does it even after hours altogether(considering that all process are in sync by then) let some of the requests from that blocked IP to go through ??
OS Version : CentOS Linux 7 Nginx Version : 1.16.1 Modsecurity version : 3.03
Mandatory dependencies
SecLang tests ....e6b03e4
Optional dependencies