search

owasp-modsecurity / ModSecurity

ModSecurity is an open source, cross platform web application firewall (WAF) engine for Apache, IIS and Nginx. It has a robust event-based programming language which provides protection from a range of attacks against web applications and allows for HTTP traffic monitoring, logging and real-time analysis.
https://www.modsecurity.org
Apache License 2.0
8.21k stars 1.6k forks source link

SIGSEGV writing log on FreeBSD #3255

Closed einsibjarni closed 3 weeks ago

einsibjarni commented 1 month ago

Describe the bug

I'm trying to update the FreeBSD port to 2.9.8 (current version in ports is 2.9.6) and to link with pcre2 instead of pcre.

I can build it and apache runs, but when modsecurity tries to write an audit log, the httpd process dies with SIGSEGV. Setting SecAuditLogType Concurrent stops httpd from dying.

Logs and dumps

Output of:

httpd.zip

Notice: Be carefully to not leak any confidential information.

To Reproduce

Steps to reproduce the behavior:

Anything that causes mod_security to write an audit log.

Expected behavior

A clear and concise description of what you expected to happen.

Server (please complete the following information):

Rule Set (please complete the following information):

Additional context

The stacktrace indicates that the problem is in apr_global_mutex_lock().

The configure command that the ports build generates and uses to build is: ./configure --with-apxs=/usr/local/sbin/apxs --with-pcre2=/usr/local --with-yajl=/usr/local --with-curl=/usr/local --without-ssdeep --without-lua --disable-mlogc --prefix=/usr/local --localstatedir=/var --mandir=/usr/local/share/man --disable-silent-rules --infodir=/usr/local/share/info/ --build=amd64-portbld-freebsd14.1

I'm trying to get the port in FreeBSD ports updated, so even though I can workaround the problem, we need to address this issue if it is to be merged upstream.

airween commented 1 month ago

@einsibjarni thanks for reporting.

Could you share your stacktrace output? Eg. bt full.

@marcstern could you take a look at this? I assume this can be related to PR #3191.

einsibjarni commented 1 month ago

Could you share your stacktrace output? Eg. bt full.

The core dump is attached to the post as httpd.zip, but the output from bt full is:

(lldb) target create "/usr/local/sbin/httpd" --core "/tmp/httpd.core"
Core file '/tmp/httpd.core' (x86_64) was loaded.
(lldb) bt all
warning: This version of LLDB has no plugin for the language "assembler". Inspection of frame variables will be limited.
warning: This version of LLDB has no plugin for the language "assembler". Inspection of frame variables will be limited.
* thread #1, name = 'httpd', stop reason = signal SIGSEGV
  * frame #0: 0x0000000823a07699 libapr-1.so.0`apr_global_mutex_lock + 9
    frame #1: 0x0000000837291a78 mod_security2.so`sec_audit_logger_native + 504
    frame #2: 0x0000000837289e0a mod_security2.so`modsecurity_process_phase + 1594
    frame #3: 0x000000083728866a mod_security2.so`hook_log_transaction + 826
    frame #4: 0x00000000002808b6 httpd`ap_run_log_transaction + 54
    frame #5: 0x0000000000292d22 httpd`___lldb_unnamed_symbol4032 + 50
    frame #6: 0x0000000823a09aae libapr-1.so.0`apr_pool_destroy + 46
    frame #7: 0x0000000000292d73 httpd`___lldb_unnamed_symbol4033 + 35
    frame #8: 0x00000000002937a0 httpd`ap_core_output_filter + 1088
    frame #9: 0x0000000828873665 mod_mpm_event.so`___lldb_unnamed_symbol388 + 1429
    frame #10: 0x0000000828872f96 mod_mpm_event.so`___lldb_unnamed_symbol387 + 854
    frame #11: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e2700) at thr_create.c:289:16
  thread #2, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x0000000825b0256a libc.so.7`__sys_read at _read.S:4
    frame #1: 0x000000082639a773 libthr.so.3`__thr_read(fd=9, buf=0x00000008205d9967, nbytes=1) at thr_syscalls.c:417:8
    frame #2: 0x0000000000275800 httpd`ap_mpm_podx_check + 48
    frame #3: 0x000000082887212b mod_mpm_event.so`___lldb_unnamed_symbol378 + 1467
    frame #4: 0x0000000828871a46 mod_mpm_event.so`___lldb_unnamed_symbol376 + 358
    frame #5: 0x00000008288712ac mod_mpm_event.so`___lldb_unnamed_symbol367 + 4748
    frame #6: 0x0000000000270ded httpd`ap_run_mpm + 77
    frame #7: 0x000000000025e193 httpd`main + 2483
    frame #8: 0x0000000825a4ea6a libc.so.7`__libc_start1(argc=2, argv=0x00000008205d9d18, env=0x00000008205d9d30, cleanup=<unavailable>, mainX=(httpd`main)) at libc_start1.c:157:7
    frame #9: 0x000000000025d760 httpd`_start + 48
  thread #3, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e2000) at thr_create.c:289:16
  thread #4, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e2e00) at thr_create.c:289:16
  thread #5, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e3500) at thr_create.c:289:16
  thread #6, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e3c00) at thr_create.c:289:16
  thread #7, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e4300) at thr_create.c:289:16
  thread #8, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e4a00) at thr_create.c:289:16
  thread #9, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e5100) at thr_create.c:289:16
  thread #10, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e5800) at thr_create.c:289:16
  thread #11, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e5f00) at thr_create.c:289:16
  thread #12, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e6600) at thr_create.c:289:16
  thread #13, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e6d00) at thr_create.c:289:16
  thread #14, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e7400) at thr_create.c:289:16
  thread #15, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e7b00) at thr_create.c:289:16
  thread #16, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e8200) at thr_create.c:289:16
  thread #17, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd21e8900) at thr_create.c:289:16
  thread #18, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd2844000) at thr_create.c:289:16
  thread #19, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd2844700) at thr_create.c:289:16
  thread #20, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd2844e00) at thr_create.c:289:16
  thread #21, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd2845500) at thr_create.c:289:16
  thread #22, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd2845c00) at thr_create.c:289:16
  thread #23, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd2846300) at thr_create.c:289:16
  thread #24, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd2846a00) at thr_create.c:289:16
  thread #25, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd2847100) at thr_create.c:289:16
  thread #26, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x000000082638be2c libthr.so.3`_umtx_op_err at _umtx_op_err.S:38
    frame #1: 0x000000082639bd82 libthr.so.3`_thr_umtx_timedwait_uint(mtx=<unavailable>, id=<unavailable>, clockid=<unavailable>, abstime=<unavailable>, shared=<unavailable>) at thr_umtx.c:233:10
    frame #2: 0x00000008263923f9 libthr.so.3`_thr_sleep(curthread=<unavailable>, clockid=<unavailable>, abstime=<unavailable>) at thr_kern.c:197:9 [artificial]
    frame #3: 0x000000082638d9a7 libthr.so.3`cond_wait_common [inlined] cond_wait_user(cvp=0x000019ddd21841e0, mp=0x000019ddd2600888, abstime=0x0000000000000000, cancel=1) at thr_cond.c:317:11
    frame #4: 0x000000082638d8f5 libthr.so.3`cond_wait_common(cond=<unavailable>, mutex=<unavailable>, abstime=0x0000000000000000, cancel=1) at thr_cond.c:377:11
    frame #5: 0x000000000027b132 httpd`ap_queue_pop_something + 130
    frame #6: 0x0000000828872d8c mod_mpm_event.so`___lldb_unnamed_symbol387 + 332
    frame #7: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd2847800) at thr_create.c:289:16
  thread #27, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x0000000825b0242a libc.so.7`__sys_kevent at _kevent.S:4
    frame #1: 0x000000082639ac24 libthr.so.3`__thr_kevent(kq=16, changelist=0x0000000000000000, nchanges=0, eventlist=0x000019ddd2172028, nevents=114, timeout=0x0000000860ce5ed0) at thr_syscalls.c:397:8
    frame #2: 0x0000000823a10bf0 libapr-1.so.0`___lldb_unnamed_symbol1234 + 128
    frame #3: 0x00000008288749f2 mod_mpm_event.so`___lldb_unnamed_symbol393 + 2178
    frame #4: 0x000000082638eb05 libthr.so.3`thread_start(curthread=0x000019ddd2847f00) at thr_create.c:289:16
(lldb) q
einsibjarni commented 1 month ago

I've tested building 2.9.7 and the error does not occur there, but if I build 2.9.7 with pcre2, ldd shows it links with both pcre and pcre2.

I've tried 2.9.8 with both pcre and pcre2 and both show this error.

marcstern commented 1 month ago

I think it's a side-effect of an already existing problem: when a mutex creation fails (for any reason), we try to use later, although it's null. This PR should fix the problem: https://github.com/owasp-modsecurity/ModSecurity/pull/3257

einsibjarni commented 1 month ago

I applied the patch from https://github.com/owasp-modsecurity/ModSecurity/pull/3257 and I can confirm that it fixes the crashes. It does write "Global mutex was not created" in httpd error log, but the audit log does get written

airween commented 1 month ago

It's good to see that the segfault has gone away - but I still wonder why the mutex cannot be created.

@einsibjarni do you think is there any special settings? I tried to set up the SecAuditLogType Concurrent but I was not able reproduce your behavior.

I'm worrying about that - as you described - this didn't happen in 2.9.7, right? So why happens this?

einsibjarni commented 1 month ago

I tried to set up the SecAuditLogType Concurrent

It only happens with ‘SecAuditLogType Serial’

airween commented 1 month ago

Ah, sorry, my bad...

Then it's more weird: this is the default setting, and we also check this behavior in our CI.

I also use this setting on many servers without any problem - it would be highly recommended if we could find out the cause of the error (I mean the team - with your help).

Could you give some details about your FreeBSD system (architecture, version, special settings, etc...)? I'm going to install a VM and try to reproduce it.

einsibjarni commented 1 month ago

I've tested on FreeBSD 14.1-RELEASE-p4 with apache24 from binary pkg and I get the error.

So just install base FreeBSD 14.1-RELEASE and then pkg install apache24

einsibjarni commented 1 month ago

I'm building mod_security from ports, but with patches to update it to 2.9.8.

Resulting configure cmd is: ./configure --with-apxs=/usr/local/sbin/apxs --with-pcre2=/usr/local --with-yajl=/usr/local --with-curl=/usr/local --without-ssdeep --without-lua --disable-mlogc --prefix=/usr/local --localstatedir=/var --mandir=/usr/local/share/man --disable-silent-rules --infodir=/usr/local/share/info/ --build=amd64-portbld-freebsd14.1

marcstern commented 1 month ago

@einsibjarni : can you check if https://github.com/owasp-modsecurity/ModSecurity/pull/3257 solves the problem?

einsibjarni commented 1 month ago

@marcstern I did try it, and it does fix the segfault.

It does write "Global mutex was not created" in httpd error log, but the audit log does get written

marcstern commented 1 month ago

That's the expected behaviour, indeed. Why is the global mutex not created, that's another issue (concurrency?), but at least we handle that case correctly now.

airween commented 1 month ago

@einsibjarni,

I need some help to bring up ModSecurity2 on FreeBSD. I already installed the VM, Apache 2.4 is running, and compiled the module from source with your ./configure instruction.

After I added the module I got an error message:

httpd: Syntax error on line 184 of /usr/local/etc/apache24/httpd.conf: Syntax error on line 1 of /usr/local/etc/apache24/modules.d/001_security.conf: Can't locate API module structure `mod_security2' in file /usr/local/modsecurity/lib/mod_security2.so: Undefined symbol "mod_security2"

Would you mind to come to Slack and ping me in private? You can join here and look at the #project-modsecurity channel.

EinalButtocks commented 1 month ago

I can't seem to join the slack server unless I have a owasp email address.

I'm building ModSecurity2 through ports, patched to update it to 2.9.8.

https://docs.freebsd.org/en/books/handbook/ports/#ports-using

Attached is my proposed patch to the port to update it to 2.9.8. 0001-Update-www-mod_security-to-2.9.8.patch

airween commented 1 month ago

I can't seem to join the slack server unless I have a owasp email address.

hmm... I registered on OWASP Slack with my gmail.com address. I don't think there is any restriction.

I'm building ModSecurity2 through ports, patched to update it to 2.9.8.

https://docs.freebsd.org/en/books/handbook/ports/#ports-using

Attached is my proposed patch to the port to update it to 2.9.8. 0001-Update-www-mod_security-to-2.9.8.patch

I followed your instructions, and installed Apache from ports, then I built the module. After that I got the result above.

EinalButtocks commented 1 month ago

image

EinalButtocks commented 1 month ago

What's the contents of /usr/local/etc/apache24/modules.d/001_security.conf?

I don't have /usr/local/modsecurity/lib/mod_security2.so, it's in /usr/local/lib/mod_security2.so

airween commented 1 month ago

Ah, I see.

Please share your e-mail address (you can send me in private - see my e-mail address on my GH page), and I can send you an invitation.

airween commented 1 month ago

What's the contents of /usr/local/etc/apache24/modules.d/001_security.conf?

LoadModule mod_security2 /usr/local/modsecurity/lib/mod_security2.so

I don't have /usr/local/modsecurity/lib/mod_security2.so, it's in /usr/local/lib/mod_security2.so

# ls -l /usr/local/modsecurity/lib/mod_security2.so
-rwxr-xr-x  1 root wheel 2572608 Sep 20 14:37 /usr/local/modsecurity/lib/mod_security2.so
airween commented 1 month ago

I could review the issue on FreeBSD, and found an interesting behavior.

mod_security2 uses files for mutexes, see this part:

    rc = apr_file_mktemp(&lock_name, path, 0, mp);
    if (rc != APR_SUCCESS) {
        ap_log_perror(APLOG_MARK, APLOG_ERR, 0, mp, " ModSecurity: Could not create temporary file for global lock");
        return -1;
    }
    // below func always return APR_SUCCESS
    apr_file_name_get(&filename, lock_name);

This function (apr_file_mktemp()) returns with APR_SUCCESS, but file hasn't created (or may be it created but disappeared immediately). This is why the next function returns an error. The filename is something like /tmp/modsec-lock-tmp.yCfr5n - @einsibjarni, do you know is there any special restrictions under FreeBSD regarding to temp files under /tmp?

I tried the locking mechanism without file - simple pass a NULL to the function here:

    rc = apr_global_mutex_create(lock, NULL, APR_LOCK_DEFAULT, mp);

and this solved the problem on FreeBSD. I haven't tested it on other systems, but my question is: do we want to insist the files?

marcstern commented 1 month ago
  1. In acquire_global_lock(), we must initialize fname to NULL. This should avoid the crash in case apr_file_name_get() doesn't succeed (apr_global_mutex_create has a check if fname == NULL). I added it to the PR https://github.com/owasp-modsecurity/ModSecurity/pull/3257
  2. In modsecurity_init(), we stop the processing in case one acquire_global_lock() fails, so the remaining ones are not tried. I think we shouldn't stop as the other locks may succeed.
  3. Using APR_LOCK_DEFAULT may not always work without file. Under Windows, APR_LOCK_DEFAULT (the only supported one) doesn't require a path. If we target only other platforms supporting APR_LOCK_POSIXSEM, we could hard-code it and remove all code related to path: 
    #ifdef WIN32
# define MSC_LOCK_MECH APR_LOCK_DEFAULT
#else
# define MSC_LOCK_MECH APR_LOCK_POSIXSEM
#endif
rc = apr_global_mutex_create(lock, NULL, MSC_LOCK_MECH, mp);
airween commented 1 month ago

This should avoid the crash in case apr_file_name_get() doesn't succeed (apr_global_mutex_create has a check if fname == NULL).

The problem is not this. apr_file_mktemp() returns the filename successfully, without NULL initialization. Then the file exists.

When the apr_global_mutex_create() function is called, then that function removes the lock file and returns with APR_EBADMASK or APR_EINPROGRESS (with avalue 17).

See this summary:

Breakpoint 1, acquire_global_lock (lock=0x8014a1e98, mp=0x801427028) at modsecurity.c:150
150     rc = apr_global_mutex_create(lock, filename, APR_LOCK_DEFAULT, mp);
(gdb) p filename
$1 = 0x801458568 "/tmp/modsec-lock-tmp.fO33bJ"

check the file:

# ls -la /tmp/modsec-lock-tmp.fO33bJ
-rw-------   1 root    wheel  0 Sep 30 13:42 modsec-lock-tmp.fO33bJ

continue the investigation

Breakpoint 2, acquire_global_lock (lock=0x8014a1e98, mp=0x801427028) at modsecurity.c:152
152     if (rc != APR_SUCCESS) {
(gdb) p rc
$2 = 17

check the file again:

# ls -la /tmp/modsec-lock-tmp.fO33bJ
ls: /tmp/modsec-lock-tmp.fO33bJ: No such file or directory

I think this is because of the OS or FS (ZFS vs Ext4) has different behavior. Initialize the filename with NULL does not solve this problem.

In other words: see the truss's output (it's like strace on Linux):

openat(AT_FDCWD,"/tmp/modsec-lock-tmp.fO33bJ",O_RDWR|O_CREAT|O_EXCL,0600) = 10 (0xa)
fcntl(10,F_GETFD,)               = 0 (0x0)
fcntl(10,F_SETFD,FD_CLOEXEC)             = 0 (0x0)
getpid()                     = 43156 (0xa894)
openat(AT_FDCWD,"/tmp/modsec-lock-tmp.fO33bJ",O_WRONLY|O_CREAT|O_EXCL|O_CLOEXEC,0600) ERR#17 'File exists'
unlink("/tmp/modsec-lock-tmp.fO33bJ")        = 0 (0x0)

You can see that the lock file was created with openat(), but later the system calls it again, then it removes. There you can see the error code and message:

ERR#17 File exists

I'm almost sure we use this locking mechanism completely wrong.

marcstern commented 1 month ago

In Apache, they use a wrapper "ap_global_mutex_create" everywhere ... We probably should use either that or option 3 (no filename at all). I tested without filename under Windows and I never egt a warning (I performed some stress tests with concurrent accesses).

airween commented 1 month ago

We probably should use either that or option 3 (no filename at all).

Yes, this works for me (on FreeBSD)

int acquire_global_lock(apr_global_mutex_t **lock, apr_pool_t *mp) {
    apr_status_t rc;
    const char *filename = NULL;

    rc = apr_global_mutex_create(lock, filename, APR_LOCK_DEFAULT, mp);
    if (rc != APR_SUCCESS) {
        ap_log_perror(APLOG_MARK, APLOG_ERR, 0, mp, " ModSecurity: Could not create global mutex");
        return -1;
    }

#if !defined(MSC_TEST)
#ifdef __SET_MUTEX_PERMS
#if AP_SERVER_MAJORVERSION_NUMBER > 1 && AP_SERVER_MINORVERSION_NUMBER > 2
    rc = ap_unixd_set_global_mutex_perms(*lock);
#else
    rc = unixd_set_global_mutex_perms(*lock);
#endif
    if (rc != APR_SUCCESS) {
        ap_log_perror(APLOG_MARK, APLOG_ERR, 0, mp, " ModSecurity: Could not set permissions on global mutex"); 
        return -1;
    }
#endif /* SET_MUTEX_PERMS */
#endif /* MSC_TEST */
    return APR_SUCCESS;
}

I tested without filename under Windows and I never egt a warning (I performed some stress tests with concurrent accesses).

I'm not sure stress tests are relevant here, because - if I'm right - the lock is created only once, when the engine starts. But it's good to know that on Windows it works as well too. I'll check that on Linux too, and will notice you here. If it works, then we should modify that wrapper function (acquire_global_lock()).

marcstern commented 1 month ago

After looking at the code in APR, APR_LOCK_DEFAULT seems the right choice, no need to hard-code one. Limitation is that it may not be portable to some exotic platforms, but does it really work on something else than Unix/Windows? +1 for me

airween commented 1 month ago

I asked this on Apache's developer mailing list - see the short thread here.

marcstern commented 1 month ago

PR with the fix: https://github.com/owasp-modsecurity/ModSecurity/pull/3269

airween commented 3 weeks ago

Closing as solved with #3269.