Closed mbunkus closed 2 months ago
Could you please do the same with the asan build?
Unfortunately the ASAN build crashes when starting the daemon. It does not generate a coredump. systemd just tries to restart it until it reaches the startup limits.
The ASAN log files created contain the following:
=================================================================
==3425981==ERROR: AddressSanitizer: odr-violation (0x7f60c2fc99e0):
[1] size=1 'piecewise_construct' /usr/include/c++/11/bits/stl_pair.h:83:53
[2] size=1 'piecewise_construct' /usr/include/c++/11/bits/stl_pair.h:83:53
These globals were registered at these points:
[1]:
#0 0x559da1ef2138 (/usr/bin/rspamd+0x75138)
#1 0x7f60c26a336c (/usr/lib/rspamd/librspamd-server.so+0x3e936c)
#2 0x7f60c342847d in call_init elf/dl-init.c:70
[2]:
#0 0x559da1ef2138 (/usr/bin/rspamd+0x75138)
#1 0x7f60c07d853c (/usr/lib/rspamd/librspamd-replxx.so+0xa653c)
#2 0x7f60c342847d in call_init elf/dl-init.c:70
==3425981==HINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_odr_violation=0
SUMMARY: AddressSanitizer: odr-violation: global 'piecewise_construct' at /usr/include/c++/11/bits/stl_pair.h:83:53
==3425981==ABORTING
Set ASAN_OPTIONS=detect_odr_violation=0
environment variable as suggested. It is a known issue with some of the libstdc++ versions.
I'll do that later today. Just wanted to state that this is not mentioned in the ASAN section of the FAQ.
Got it running (and segfaulting later) with ASAN_OPTIONS=detect_odr_violation=0
. Thanks.
The generated ASAN reports are kind of hard to correlate to the coredumps, as the PIDs in the ASAN file names don't match the PIDs from coredumpctl list
. However, I've looked through all the ASAN reports, and all they contain is leak information — no instances of invalid memory access, use after free & related issues.
It's also quite possible that the ASAN routines don't have time to write a report anymore due to the segfaults terminating the processes?
Here's the journal from one such segfault with the ASAN binaries:
Jan 23 14:30:24 firesword rspamd[3543398]: <fc4d84>; proxy; proxy_accept_socket: accepted milter connection from 2a01:4f8:c0:fb01::6 port 39068
Jan 23 14:30:24 firesword rspamd[3543398]: <fc4d84>; milter; rspamd_milter_process_command: got connection from [2a01:4f8:c0:fb01::a]:41996
Jan 23 14:30:24 firesword rspamd[3543400]: <3433ad>; task; rspamd_worker_body_handler: accepted connection from 127.0.0.1 port 46892, task ptr: 00006210007CB198
Jan 23 14:30:24 firesword rspamd[3543400]: <3433ad>; task; rspamd_message_parse: loaded message; id: <undef>; queue-id: <C3673261713>; size: 246; checksum: <4db9eb8d6f5c61ff0b80e9829a80d6e7>
Jan 23 14:30:24 firesword rspamd[3543400]: <3433ad>; task; rspamd_url_text_extract: got empty text part
Jan 23 14:30:24 firesword rspamd[3543400]: <3433ad>; lua; settings.lua:391: <undef> apply static settings authenticated (id = 1937017268); authenticated matched; priority high
Jan 23 14:30:24 firesword rspamd[3543400]: <3433ad>; lua; greylist.lua:219: skip greylisting for local networks and/or authorized users
Jan 23 14:30:24 firesword rspamd[3543400]: <3433ad>; task; dkim_symbol_callback: skip DKIM checks for local networks and authorized users
Jan 23 14:30:24 firesword rspamd[3543400]: <3433ad>; lua; dmarc.lua:353: skip DMARC checks as either SPF or DKIM were not checked
Jan 23 14:30:24 firesword rspamd[3543400]: <3433ad>; lua; once_received.lua:102: Skipping once_received for authenticated user or local network
Jan 23 14:30:24 firesword rspamd[3543400]: <>; ; rspamd_crash_sig_handler: caught fatal signal 11(Segmentation fault), pid: 3543400, trace:
Jan 23 14:30:26 firesword rspamd[3543398]: <c8bdbf>; milter; rspamd_milter_process_command: got connection from 141.98.11.68:51040
Jan 23 14:30:26 firesword rspamd[3543398]: <c8bdbf>; proxy; proxy_milter_finish_handler: finished milter connection
Jan 23 14:30:27 firesword rspamd[3543400]: <>; ; log_backtrace: #0: [0x7efced1ede01]: ./debian/build/flavor-asan/src/./contrib/backward-cpp/backward.hpp:874 in backward::StackTraceImpl<backward::system_tag::linux_tag>::load_here(unsigned long, void*, void*)
Jan 23 14:30:28 firesword rspamd[3543400]: <>; ; log_backtrace: #1: [0x7efced20fc2e]: ./debian/build/flavor-asan/src/./src/libserver/backtrace.cxx:31 in rspamd::log_backtrace()
Jan 23 14:30:28 firesword rspamd[3543400]: <>; ; log_backtrace: #2: [0x7efced213d71]: ./debian/build/flavor-asan/src/./src/libserver/backtrace.cxx:59 in rspamd_print_crash
Jan 23 14:30:28 firesword rspamd[3543400]: <>; ; log_backtrace: #3: [0x7efced120ed4]: ./debian/build/flavor-asan/src/./src/libserver/worker_util.c:1637 in rspamd_crash_sig_handler
Jan 23 14:30:28 firesword rspamd[3543400]: <>; ; log_backtrace: #4: [0x7efcec57451f]: /lib/x86_64-linux-gnu/libc.so.6 in __GI___sigaction
Jan 23 14:30:29 firesword rspamd[3543400]: <>; ; log_backtrace: #5: [0x7efced35ea0c]: ./debian/build/flavor-asan/src/./src/libstat/backends/redis_backend.cxx:152 in redis_stat_runtime<float, true>::process_tokens(_GPtrArray*) const
Jan 23 14:30:29 firesword rspamd[3543400]: <>; ; log_backtrace: #6: [0x7efced7c62a5]: languages.cc:0 in lj_BC_FUNCC
Jan 23 14:30:29 firesword rspamd[3543400]: <>; ; log_backtrace: #7: [0x7efced7b2618]: /usr/lib/rspamd/librspamd-server.so in lua_pcall
Jan 23 14:30:29 firesword rspamd[3543400]: <>; ; log_backtrace: #8: [0x7efced448f78]: ./debian/build/flavor-asan/src/./src/lua/lua_redis.c:407 in lua_redis_push_data
Jan 23 14:30:29 firesword rspamd[3543400]: <>; ; log_backtrace: #9: [0x7efced449836]: ./debian/build/flavor-asan/src/./src/lua/lua_redis.c:479 in lua_redis_callback
Jan 23 14:30:29 firesword rspamd[3543400]: <>; ; log_backtrace: #10: [0x7efced6d1adb]: ./debian/build/flavor-asan/src/../contrib/hiredis/./contrib/hiredis/async.c:269 in __redisRunCallback
Jan 23 14:30:29 firesword rspamd[3543400]: <>; ; log_backtrace: #11: [0x7efced6e2f03]: ./debian/build/flavor-asan/src/../contrib/hiredis/./contrib/hiredis/async.c:470 in redisProcessCallbacks
Jan 23 14:30:29 firesword rspamd[3543400]: <>; ; log_backtrace: #12: [0x7efced6e329e]: ./debian/build/flavor-asan/src/../contrib/hiredis/./contrib/hiredis/async.c:539 in redisAsyncHandleRead
Jan 23 14:30:29 firesword rspamd[3543400]: <>; ; log_backtrace: #13: [0x7efced039c1f]: ./debian/build/flavor-asan/src/./contrib/hiredis/adapters/libev.h:53 in redisLibevReadEvent(ev_loop*, ev_io*, int)
Jan 23 14:30:29 firesword rspamd[3543400]: <>; ; log_backtrace: #14: [0x7efcecb4f8d3]: ./debian/build/flavor-asan/contrib/libev/./contrib/libev/ev.c:3809 in ev_invoke_pending
Jan 23 14:30:29 firesword rspamd[3543400]: <>; ; log_backtrace: #15: [0x7efcecb5edcb]: ./debian/build/flavor-asan/contrib/libev/./contrib/libev/ev.c:4231 in ev_run
Jan 23 14:30:29 firesword rspamd[3543400]: <>; ; log_backtrace: #16: [0x55f37aff491d]: ./debian/build/flavor-asan/src/./contrib/libev/ev.h:830 in ev_loop
Jan 23 14:30:29 firesword rspamd[3543400]: <>; ; log_backtrace: #17: [0x7efced11fcc1]: ./debian/build/flavor-asan/src/./src/libserver/worker_util.c:1179 in rspamd_handle_child_fork
Jan 23 14:30:29 firesword rspamd[3543400]: <>; ; log_backtrace: #18: [0x7efced120a1d]: ./debian/build/flavor-asan/src/./src/libserver/worker_util.c:1296 in rspamd_fork_worker
Jan 23 14:30:29 firesword rspamd[3543400]: <>; ; log_backtrace: #19: [0x55f37afedb41]: ./debian/build/flavor-asan/src/./src/rspamd.c:631 in spawn_worker_type
Jan 23 14:30:29 firesword rspamd[3543400]: <>; ; log_backtrace: #20: [0x55f37aff00d3]: ./debian/build/flavor-asan/src/./src/rspamd.c:717 in spawn_workers
Jan 23 14:30:29 firesword rspamd[3543400]: <>; ; log_backtrace: #21: [0x55f37b0029cc]: ./debian/build/flavor-asan/src/./src/rspamd.c:1678 in main
Jan 23 14:30:29 firesword rspamd[3543400]: <>; ; log_backtrace: #22: [0x7efcec55bd8f]: ./csu/../sysdeps/nptl/libc_start_call_main.h:58 in __libc_start_call_main
Jan 23 14:30:29 firesword rspamd[3543400]: <>; ; log_backtrace: #23: [0x7efcec55be3f]: ./csu/../csu/libc-start.c:392 in __libc_start_main_impl
Jan 23 14:30:29 firesword rspamd[3543400]: <>; ; log_backtrace: #24: [0x55f37afc76d4]: /usr/bin/rspamd in _start
Jan 23 14:30:29 firesword rspamd[3543400]: <>; ; log_backtrace: #25: [0xffffffffffffffff]: in
Jan 23 14:30:30 firesword rspamd[3543400]: <>; ; rspamd_crash_sig_handler: please see Rspamd FAQ to learn how to dump core files and how to fill a bug report
Jan 23 14:30:30 firesword systemd[1]: Started Process Core Dump (PID 3544010/UID 0).
Jan 23 14:30:30 firesword rspamd[3543398]: <0e9468>; proxy; proxy_accept_socket: accepted milter connection from 2a01:4f8:c0:fb01::6 port 57752
Jan 23 14:30:30 firesword rspamd[3543398]: <0e9468>; milter; rspamd_milter_process_command: got connection from 90.187.127.129:56745
Jan 23 14:30:33 firesword rspamd[3543398]: <b2f9c4>; proxy; proxy_accept_socket: accepted milter connection from 2a01:4f8:c0:fb01::6 port 57758
Jan 23 14:30:33 firesword rspamd[3543398]: <b2f9c4>; milter; rspamd_milter_process_command: got connection from 90.187.127.129:56746
Jan 23 14:30:33 firesword systemd-coredump[3544011]: Process 3543400 (rspamd) of user 116 dumped core.
Found module linux-vdso.so.1 with build-id: ea99e5d980dd1a4d23af20aa35a7d823b5c92f97
Found module libicudata.so.70 with build-id: b1c2496dd0543023c7a19c961bb7f3abc818f465
Found module libpcre.so.3 with build-id: 3982f316c887e3ad9598015fa5bae8557320476a
Found module ld-linux-x86-64.so.2 with build-id: 15921ea631d9f36502d20459c43e5c85b7d6ab76
Found module libstdc++.so.6 with build-id: e37fe1a879783838de78cbc8c80621fa685d58a2
Found module libbfd-2.38-system.so with build-id: 210401535bbcb3b2ad45cfab8186e99ec77f90fc
Found module librspamd-kann.so with build-id: e0a43be4c263c23a5d6776fa65db090f0536c9d3
Found module librspamd-actrie.so with build-id: 2548ff91814ee9db0e5caf3cd3f4a56083b8cb49
Found module librspamd-replxx.so with build-id: 16f97abf00085b37e6439360a95ef43d58e4ddce
Found module libz.so.1 with build-id: 30840b79ac329ecbf1dec0bb60180eed256d319f
Found module libcrypto.so.3 with build-id: 156e054fb88f59a4100ca7edc74a79e3908027a8
Found module libssl.so.3 with build-id: ce838f6c51f037b73ade040b4abd647d7ae7d62d
Found module libicui18n.so.70 with build-id: ff2dbcdd92cbe5a63d20291e295d8fed9f87d35b
Found module libicuuc.so.70 with build-id: bef3ff1d70aadd68aab07d858a759360c8b919ae
Found module libpcre2-8.so.0 with build-id: 184a841c55fb7fe5e3873fcda8368c71016cd54c
Found module libhs.so.5 with build-id: 91ceba793c9a4ff3995ae36e33bd1f22ec8a394f
Found module libc.so.6 with build-id: c289da5071a3399de893d2af81d6a30c62646e1e
Found module libgcc_s.so.1 with build-id: e3a44e0da9c6e835d293ed8fd2882b4c4a87130c
Found module libm.so.6 with build-id: a88ef0199bd5e742ebd0c359edf5cb2be0ec08b5
Found module libsodium.so.23 with build-id: b85aa4f896b2fd434a2a77009679314a209cd271
Found module libsqlite3.so.0 with build-id: 87d00c07f9d8a822a4251d4f053484e5c0e4ead8
Found module libglib-2.0.so.0 with build-id: c74e800dfd5f72649d673b44292f4a817e45150b
Found module librspamd-ev.so with build-id: 23d05e11a1570ad6c70d68a67a382b9386c5b5a2
Found module librspamd-server.so with build-id: d053883860ec70bc6db75e8b3bafb2d9c26a47c9
Found module rspamd with build-id: bdd632bff3d4f219185aa53ea6f3b86b68e4ca5b
Stack trace of thread 3543400:
#0 0x00007efced35ea0c _ZL23rspamd_redis_classifiedP9lua_State (librspamd-server.so + 0x7f0a0c)
#1 0x00007efced7c62a6 lj_BC_FUNCC (librspamd-server.so + 0xc582a6)
#2 0x00007efced7b2619 lua_pcall (librspamd-server.so + 0xc44619)
#3 0x00007efced448f79 lua_redis_push_data (librspamd-server.so + 0x8daf79)
#4 0x00007efced449837 lua_redis_callback (librspamd-server.so + 0x8db837)
#5 0x00007efced6d1adc __redisRunCallback (librspamd-server.so + 0xb63adc)
#6 0x00007efced6e2f04 redisProcessCallbacks (librspamd-server.so + 0xb74f04)
#7 0x00007efced6e329f redisAsyncHandleRead (librspamd-server.so + 0xb7529f)
#8 0x00007efced039c20 redisLibevReadEvent (librspamd-server.so + 0x4cbc20)
#9 0x00007efcecb4f8d4 ev_invoke_pending (librspamd-ev.so + 0x58d4)
#10 0x00007efcecb5edcc ev_run (librspamd-ev.so + 0x14dcc)
#11 0x000055f37aff491e start_worker (rspamd + 0x5391e)
#12 0x00007efced11fcc2 rspamd_handle_child_fork (librspamd-server.so + 0x5b1cc2)
#13 0x00007efced120a1e rspamd_fork_worker (librspamd-server.so + 0x5b2a1e)
#14 0x000055f37afedb42 spawn_worker_type (rspamd + 0x4cb42)
#15 0x000055f37aff00d4 spawn_workers (rspamd + 0x4f0d4)
#16 0x000055f37b0029cd main (rspamd + 0x619cd)
#17 0x00007efcec55bd90 __libc_start_call_main (libc.so.6 + 0x29d90)
#18 0x00007efcec55be40 __libc_start_main_impl (libc.so.6 + 0x29e40)
#19 0x000055f37afc76d5 _start (rspamd + 0x266d5)
Jan 23 14:30:34 firesword systemd[1]: systemd-coredump@11-3544010-0.service: Deactivated successfully.
Jan 23 14:30:34 firesword systemd[1]: systemd-coredump@11-3544010-0.service: Consumed 3.489s CPU time.
Jan 23 14:30:34 firesword rspamd[3543394]: <b36a9b>; main; rspamd_srv_handler: cannot read from worker's srv pipe connection closed; command = heartbeat
Jan 23 14:30:34 firesword rspamd[3543398]: <fc4d84>; proxy; proxy_backend_master_error_handler: abnormally closing connection from backend: 127.0.0.1:11333, error: IO read error: unexpected EOF, retries left: 4
Jan 23 14:30:34 firesword rspamd[3543394]: <b36a9b>; main; rspamd_check_termination_clause: normal process 3543400 terminated abnormally by signal: Segmentation fault and created core file; please see Rspamd FAQ to learn how to extract data from core file and fill a bug report
Here's the corresponding coredumpctl debug 3543400
with bt full
:
[0 root@firesword ~] coredumpctl debug 3543400
PID: 3543400 (rspamd)
UID: 116 (_rspamd)
GID: 122 (_rspamd)
Signal: 11 (SEGV)
Timestamp: Tue 2024-01-23 14:30:30 CET (13min ago)
Command Line: $'rspamd: normal process (localhost:11333)' "" "" "" "" ""
Executable: /usr/bin/rspamd
Control Group: /system.slice/rspamd.service
Unit: rspamd.service
Slice: system.slice
Boot ID: 43d0a12039ee41cfaf7ff4b5da8409f8
Machine ID: c7ee5341b29b497f9731e62369e13fe0
Hostname: firesword
Storage: /var/lib/systemd/coredump/core.rspamd.116.43d0a12039ee41cfaf7ff4b5da8409f8.3543400.1706016630000000.zst (present)
Disk Size: 43.2M
Message: Process 3543400 (rspamd) of user 116 dumped core.
Found module linux-vdso.so.1 with build-id: ea99e5d980dd1a4d23af20aa35a7d823b5c92f97
Found module libicudata.so.70 with build-id: b1c2496dd0543023c7a19c961bb7f3abc818f465
Found module libpcre.so.3 with build-id: 3982f316c887e3ad9598015fa5bae8557320476a
Found module ld-linux-x86-64.so.2 with build-id: 15921ea631d9f36502d20459c43e5c85b7d6ab76
Found module libstdc++.so.6 with build-id: e37fe1a879783838de78cbc8c80621fa685d58a2
Found module libbfd-2.38-system.so with build-id: 210401535bbcb3b2ad45cfab8186e99ec77f90fc
Found module librspamd-kann.so with build-id: e0a43be4c263c23a5d6776fa65db090f0536c9d3
Found module librspamd-actrie.so with build-id: 2548ff91814ee9db0e5caf3cd3f4a56083b8cb49
Found module librspamd-replxx.so with build-id: 16f97abf00085b37e6439360a95ef43d58e4ddce
Found module libz.so.1 with build-id: 30840b79ac329ecbf1dec0bb60180eed256d319f
Found module libcrypto.so.3 with build-id: 156e054fb88f59a4100ca7edc74a79e3908027a8
Found module libssl.so.3 with build-id: ce838f6c51f037b73ade040b4abd647d7ae7d62d
Found module libicui18n.so.70 with build-id: ff2dbcdd92cbe5a63d20291e295d8fed9f87d35b
Found module libicuuc.so.70 with build-id: bef3ff1d70aadd68aab07d858a759360c8b919ae
Found module libpcre2-8.so.0 with build-id: 184a841c55fb7fe5e3873fcda8368c71016cd54c
Found module libhs.so.5 with build-id: 91ceba793c9a4ff3995ae36e33bd1f22ec8a394f
Found module libc.so.6 with build-id: c289da5071a3399de893d2af81d6a30c62646e1e
Found module libgcc_s.so.1 with build-id: e3a44e0da9c6e835d293ed8fd2882b4c4a87130c
Found module libm.so.6 with build-id: a88ef0199bd5e742ebd0c359edf5cb2be0ec08b5
Found module libsodium.so.23 with build-id: b85aa4f896b2fd434a2a77009679314a209cd271
Found module libsqlite3.so.0 with build-id: 87d00c07f9d8a822a4251d4f053484e5c0e4ead8
Found module libglib-2.0.so.0 with build-id: c74e800dfd5f72649d673b44292f4a817e45150b
Found module librspamd-ev.so with build-id: 23d05e11a1570ad6c70d68a67a382b9386c5b5a2
Found module librspamd-server.so with build-id: d053883860ec70bc6db75e8b3bafb2d9c26a47c9
Found module rspamd with build-id: bdd632bff3d4f219185aa53ea6f3b86b68e4ca5b
Stack trace of thread 3543400:
#0 0x00007efced35ea0c _ZL23rspamd_redis_classifiedP9lua_State (librspamd-server.so + 0x7f0a0c)
#1 0x00007efced7c62a6 lj_BC_FUNCC (librspamd-server.so + 0xc582a6)
#2 0x00007efced7b2619 lua_pcall (librspamd-server.so + 0xc44619)
#3 0x00007efced448f79 lua_redis_push_data (librspamd-server.so + 0x8daf79)
#4 0x00007efced449837 lua_redis_callback (librspamd-server.so + 0x8db837)
#5 0x00007efced6d1adc __redisRunCallback (librspamd-server.so + 0xb63adc)
#6 0x00007efced6e2f04 redisProcessCallbacks (librspamd-server.so + 0xb74f04)
#7 0x00007efced6e329f redisAsyncHandleRead (librspamd-server.so + 0xb7529f)
#8 0x00007efced039c20 redisLibevReadEvent (librspamd-server.so + 0x4cbc20)
#9 0x00007efcecb4f8d4 ev_invoke_pending (librspamd-ev.so + 0x58d4)
#10 0x00007efcecb5edcc ev_run (librspamd-ev.so + 0x14dcc)
#11 0x000055f37aff491e start_worker (rspamd + 0x5391e)
#12 0x00007efced11fcc2 rspamd_handle_child_fork (librspamd-server.so + 0x5b1cc2)
#13 0x00007efced120a1e rspamd_fork_worker (librspamd-server.so + 0x5b2a1e)
#14 0x000055f37afedb42 spawn_worker_type (rspamd + 0x4cb42)
#15 0x000055f37aff00d4 spawn_workers (rspamd + 0x4f0d4)
#16 0x000055f37b0029cd main (rspamd + 0x619cd)
#17 0x00007efcec55bd90 __libc_start_call_main (libc.so.6 + 0x29d90)
#18 0x00007efcec55be40 __libc_start_main_impl (libc.so.6 + 0x29e40)
#19 0x000055f37afc76d5 _start (rspamd + 0x266d5)
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/rspamd...
Reading symbols from /usr/lib/debug/.build-id/bd/d632bff3d4f219185aa53ea6f3b86b68e4ca5b.debug...
warning: Can't open file /var/lib/rspamd/4612c5b2850bcf5f37b1a61e18706264bfe489b0f3ca3c0e85ba51bc2b39b4a4.hsmc.unser during file-backed mapping note processing
warning: Can't open file /var/lib/rspamd/011552be555c528552cd58b14955920ababef1827c602313f65c82d4732c76c4.hsmp.unser during file-backed mapping note processing
warning: Can't open file /var/lib/rspamd/328b056a0453cb5c88782ba27e267d82eeb40ca5fafe741735f95516d385ec18.hsmp.unser during file-backed mapping note processing
warning: Can't open file /var/lib/rspamd/bd94534b246729b579d5632c7cd575a4e13334e7987982d329dee0675ce21101.hsmp.unser during file-backed mapping note processing
warning: Can't open file /var/lib/rspamd/f81e6bc222efc45a539cbdf1c6080e4fddc6cbf7a2085196a1e11a15786b7c23.hsmp.unser during file-backed mapping note processing
warning: Can't open file /dev/zero (deleted) during file-backed mapping note processing
[New LWP 3543400]
bt fu[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `rspamd: normal process (localhost:11333) '.
ll
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007efced35ea0c in redis_stat_runtime<float, true>::process_tokens (tokens=0x619000395ac0, this=0x607000074230) at ./src/libstat/backends/redis_backend.cxx:152
152 ./src/libstat/backends/redis_backend.cxx: No such file or directory.
(gdb) bt full
#0 0x00007efced35ea0c in redis_stat_runtime<float, true>::process_tokens (tokens=0x619000395ac0, this=0x607000074230) at ./src/libstat/backends/redis_backend.cxx:152
idx = @0x7ffd0b41c830: 4
val = @0x7ffd0b41c834: 15
__for_range = <optimized out>
__for_begin = <optimized out>
__for_end = <optimized out>
tok = 0x6210007c5530
tok = <optimized out>
idx = <optimized out>
val = <optimized out>
__for_range = <optimized out>
__for_begin = <optimized out>
__for_end = <optimized out>
#1 rspamd_redis_classified (L=0x7efce6332f60) at ./src/libstat/backends/redis_backend.cxx:863
filler_func = <optimized out>
opposite_rt_maybe = <optimized out>
cookie = <optimized out>
task = 0x6210007cb198
rt = 0x6070000741c0
result = <optimized out>
__func__ = <optimized out>
__PRETTY_FUNCTION__ = <optimized out>
#2 0x00007efced7c62a6 in lj_BC_FUNCC () from /usr/lib/rspamd/librspamd-server.so
No symbol table info available.
c#3 0x00007efced7b2619 in lua_pcall () from /usr/lib/rspamd/librspamd-server.so
No symbol table info available.
#4 0x00007efced448f79 in lua_redis_push_data (r=r@entry=0x6040000ca5d0, ctx=ctx@entry=0x60e000085660, sp_ud=sp_ud@entry=0x60b0000bb880) at ./src/lua/lua_redis.c:407
--Type <RET> for more, q to quit, c to continue without paging--c
err_idx = 1
ret = <optimized out>
ud = 0x60e000085668
cbs = <optimized out>
L = 0x7efce6332f60
__func__ = "lua_redis_push_data"
#5 0x00007efced449837 in lua_redis_callback (c=0x61300008e340, r=0x6040000ca5d0, priv=0x60b0000bb880) at ./src/lua/lua_redis.c:479
reply = 0x6040000ca5d0
sp_ud = 0x60b0000bb880
ctx = 0x60e000085660
ud = 0x60e000085668
ac = <optimized out>
__func__ = "lua_redis_callback"
#6 0x00007efced6d1adc in __redisRunCallback (ac=ac@entry=0x61300008e340, cb=cb@entry=0x7ffd0b41cb90, reply=<optimized out>) at ../contrib/hiredis/./contrib/hiredis/async.c:269
c = 0x61300008e340
#7 0x00007efced6e2f04 in redisProcessCallbacks (ac=ac@entry=0x61300008e340) at ../contrib/hiredis/./contrib/hiredis/async.c:470
c = 0x61300008e340
cb = <optimized out>
reply = <optimized out>
status = 0
__PRETTY_FUNCTION__ = "redisProcessCallbacks"
#8 0x00007efced6e329f in redisAsyncHandleRead (ac=0x61300008e340) at ../contrib/hiredis/./contrib/hiredis/async.c:539
c = 0x61300008e340
#9 0x00007efced039c20 in redisLibevReadEvent (loop=<optimized out>, watcher=<optimized out>, revents=<optimized out>) at ./contrib/hiredis/adapters/libev.h:53
e = <optimized out>
#10 0x00007efcecb4f8d4 in ev_invoke_pending (loop=0x61a00000cc80) at ./contrib/libev/ev.c:3809
p = <optimized out>
#11 0x00007efcecb5edcc in ev_run (loop=0x61a00000cc80, flags=flags@entry=0) at ./contrib/libev/ev.c:4231
__PRETTY_FUNCTION__ = "ev_run"
#12 0x000055f37aff491e in ev_loop (flags=0, loop=<optimized out>) at ./contrib/libev/ev.h:830
No locals.
#13 start_worker (worker=0x61200001a440) at ./src/worker.c:524
ctx = 0x7efce46da2d0
is_controller = 0
__func__ = "start_worker"
#14 0x00007efced11fcc2 in rspamd_handle_child_fork (wrk=wrk@entry=0x61200001a440, rspamd_main=rspamd_main@entry=0x6120000004c0, cf=cf@entry=0x60c00001b700, listen_sockets=listen_sockets@entry=0x61d000001980) at ./src/libserver/worker_util.c:1179
rc = <optimized out>
rlim = <optimized out>
it = <optimized out>
k = <optimized out>
v = <optimized out>
cur = <optimized out>
__func__ = "rspamd_handle_child_fork"
#15 0x00007efced120a1e in rspamd_fork_worker (rspamd_main=rspamd_main@entry=0x6120000004c0, cf=cf@entry=0x60c00001b700, index=index@entry=0, ev_base=ev_base@entry=0x7efcecb6ca40 <default_loop_struct>, term_handler=term_handler@entry=0x55f37afedd44 <rspamd_cld_handler>, listen_sockets=0x61d000001980) at ./src/libserver/worker_util.c:1296
wrk = 0x61200001a440
__func__ = "rspamd_fork_worker"
#16 0x000055f37afedb42 in spawn_worker_type (rspamd_main=rspamd_main@entry=0x6120000004c0, event_loop=event_loop@entry=0x7efcecb6ca40 <default_loop_struct>, cf=cf@entry=0x60c00001b700) at ./src/rspamd.c:631
i = 0
__func__ = "spawn_worker_type"
#17 0x000055f37aff00d4 in spawn_workers (rspamd_main=rspamd_main@entry=0x6120000004c0, ev_base=ev_base@entry=0x7efcecb6ca40 <default_loop_struct>) at ./src/rspamd.c:717
cur = 0x619000022360
ls = <optimized out>
cf = 0x60c00001b700
p = <optimized out>
key = <optimized out>
bcf = 0x0
listen_ok = 1
seen_mandatory_workers = 0x619000384ca0
cw = <optimized out>
wrk = <optimized out>
i = <optimized out>
__func__ = "spawn_workers"
#18 0x000055f37b0029cd in main (argc=<optimized out>, argv=<optimized out>, env=<optimized out>) at ./src/rspamd.c:1678
i = <optimized out>
res = 0
signals = <optimized out>
sigpipe_act = <optimized out>
pworker = <optimized out>
type = <optimized out>
control_addr = <optimized out>
event_loop = 0x7efcecb6ca40 <default_loop_struct>
rspamd_main = 0x6120000004c0
skip_pid = 1
control_signals = <optimized out>
__func__ = "main"
stat_update_time = <optimized out>
(gdb)
quit
I don't have an ASAN log for that PID, only for others. But like I said, they only contain leak information.
Here's an example ASAN log from a different PID:
=================================================================
==3543401==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 40 byte(s) in 1 object(s) allocated from:
#0 0x55f37b093247 in calloc (/usr/bin/rspamd+0xf2247)
#1 0x55f37afe969f in kh_init_fuzzy_key_ids_set src/fuzzy_storage.c:110
#2 0x55f37afe969f in init_fuzzy src/fuzzy_storage.c:2703
#3 0x7efcecf712a7 in rspamd_rcl_worker_handler src/libserver/cfg_rcl.cxx:792
#4 0x7efcecf7f94b in rspamd_rcl_process_section src/libserver/cfg_rcl.cxx:2563
#5 0x7efcecf8b9e5 in rspamd_rcl_parse src/libserver/cfg_rcl.cxx:2675
#6 0x7efcecf8e9a9 in rspamd_config_read src/libserver/cfg_rcl.cxx:3784
#7 0x55f37aff0541 in load_rspamd_config src/rspamd.c:956
#8 0x55f37b001d2d in main src/rspamd.c:1518
#9 0x7efcec55bd8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
Direct leak of 40 byte(s) in 1 object(s) allocated from:
#0 0x55f37b093247 in calloc (/usr/bin/rspamd+0xf2247)
#1 0x55f37afe966e in kh_init_fuzzy_key_ids_set src/fuzzy_storage.c:110
#2 0x55f37afe966e in init_fuzzy src/fuzzy_storage.c:2702
#3 0x7efcecf712a7 in rspamd_rcl_worker_handler src/libserver/cfg_rcl.cxx:792
#4 0x7efcecf7f94b in rspamd_rcl_process_section src/libserver/cfg_rcl.cxx:2563
#5 0x7efcecf8b9e5 in rspamd_rcl_parse src/libserver/cfg_rcl.cxx:2675
#6 0x7efcecf8e9a9 in rspamd_config_read src/libserver/cfg_rcl.cxx:3784
#7 0x55f37aff0541 in load_rspamd_config src/rspamd.c:956
#8 0x55f37b001d2d in main src/rspamd.c:1518
#9 0x7efcec55bd8f in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
SUMMARY: AddressSanitizer: 80 byte(s) leaked in 2 allocation(s).
In the core file inspection, could you please type p *tokens
and show the result?
Sure. Unfortunately the core dump I used above has already been removed (auto-updates installed the 3.8.0 last night, meaning there were TONS of new segfaults & core dumps over during the night…).
Here's a new full backtrace and the requested peek into tokens from a newly observed crash from a couple of minutes ago:
[0 root@firesword ~] coredumpctl debug 125743
PID: 125743 (rspamd)
UID: 116 (_rspamd)
GID: 122 (_rspamd)
Signal: 11 (SEGV)
Timestamp: Wed 2024-01-24 12:10:06 CET (33s ago)
Command Line: $'rspamd: normal process (localhost:11333)' "" "" "" "" ""
Executable: /usr/bin/rspamd
Control Group: /system.slice/rspamd.service
Unit: rspamd.service
Slice: system.slice
Boot ID: 43d0a12039ee41cfaf7ff4b5da8409f8
Machine ID: c7ee5341b29b497f9731e62369e13fe0
Hostname: firesword
Storage: /var/lib/systemd/coredump/core.rspamd.116.43d0a12039ee41cfaf7ff4b5da8409f8.125743.1706094606000000.zst (present)
Disk Size: 91.3M
Message: Process 125743 (rspamd) of user 116 dumped core.
Found module linux-vdso.so.1 with build-id: ea99e5d980dd1a4d23af20aa35a7d823b5c92f97
Found module libicudata.so.70 with build-id: b1c2496dd0543023c7a19c961bb7f3abc818f465
Found module libpcre.so.3 with build-id: 3982f316c887e3ad9598015fa5bae8557320476a
Found module ld-linux-x86-64.so.2 with build-id: 15921ea631d9f36502d20459c43e5c85b7d6ab76
Found module libstdc++.so.6 with build-id: e37fe1a879783838de78cbc8c80621fa685d58a2
Found module libbfd-2.38-system.so with build-id: 210401535bbcb3b2ad45cfab8186e99ec77f90fc
Found module librspamd-kann.so with build-id: e0a43be4c263c23a5d6776fa65db090f0536c9d3
Found module librspamd-actrie.so with build-id: 2548ff91814ee9db0e5caf3cd3f4a56083b8cb49
Found module librspamd-replxx.so with build-id: 16f97abf00085b37e6439360a95ef43d58e4ddce
Found module libz.so.1 with build-id: 30840b79ac329ecbf1dec0bb60180eed256d319f
Found module libcrypto.so.3 with build-id: 156e054fb88f59a4100ca7edc74a79e3908027a8
Found module libssl.so.3 with build-id: ce838f6c51f037b73ade040b4abd647d7ae7d62d
Found module libicui18n.so.70 with build-id: ff2dbcdd92cbe5a63d20291e295d8fed9f87d35b
Found module libicuuc.so.70 with build-id: bef3ff1d70aadd68aab07d858a759360c8b919ae
Found module libpcre2-8.so.0 with build-id: 184a841c55fb7fe5e3873fcda8368c71016cd54c
Found module libhs.so.5 with build-id: 91ceba793c9a4ff3995ae36e33bd1f22ec8a394f
Found module libc.so.6 with build-id: c289da5071a3399de893d2af81d6a30c62646e1e
Found module libgcc_s.so.1 with build-id: e3a44e0da9c6e835d293ed8fd2882b4c4a87130c
Found module libm.so.6 with build-id: a88ef0199bd5e742ebd0c359edf5cb2be0ec08b5
Found module libsodium.so.23 with build-id: b85aa4f896b2fd434a2a77009679314a209cd271
Found module libsqlite3.so.0 with build-id: 87d00c07f9d8a822a4251d4f053484e5c0e4ead8
Found module libglib-2.0.so.0 with build-id: c74e800dfd5f72649d673b44292f4a817e45150b
Found module librspamd-ev.so with build-id: 23d05e11a1570ad6c70d68a67a382b9386c5b5a2
Found module librspamd-server.so with build-id: d053883860ec70bc6db75e8b3bafb2d9c26a47c9
Found module rspamd with build-id: bdd632bff3d4f219185aa53ea6f3b86b68e4ca5b
Stack trace of thread 125743:
#0 0x00007f3707f44a0c _ZL23rspamd_redis_classifiedP9lua_State (librspamd-server.so + 0x7f0a0c)
#1 0x00007f37083ac2a6 lj_BC_FUNCC (librspamd-server.so + 0xc582a6)
#2 0x00007f3708398619 lua_pcall (librspamd-server.so + 0xc44619)
#3 0x00007f370802ef79 lua_redis_push_data (librspamd-server.so + 0x8daf79)
#4 0x00007f370802f837 lua_redis_callback (librspamd-server.so + 0x8db837)
#5 0x00007f37082b7adc __redisRunCallback (librspamd-server.so + 0xb63adc)
#6 0x00007f37082c8f04 redisProcessCallbacks (librspamd-server.so + 0xb74f04)
#7 0x00007f37082c929f redisAsyncHandleRead (librspamd-server.so + 0xb7529f)
#8 0x00007f3707c1fc20 redisLibevReadEvent (librspamd-server.so + 0x4cbc20)
#9 0x00007f37077358d4 ev_invoke_pending (librspamd-ev.so + 0x58d4)
#10 0x00007f3707744dcc ev_run (librspamd-ev.so + 0x14dcc)
#11 0x0000555a474aa91e start_worker (rspamd + 0x5391e)
#12 0x00007f3707d05cc2 rspamd_handle_child_fork (librspamd-server.so + 0x5b1cc2)
#13 0x00007f3707d06a1e rspamd_fork_worker (librspamd-server.so + 0x5b2a1e)
#14 0x0000555a474a3b42 spawn_worker_type (rspamd + 0x4cb42)
#15 0x0000555a474a60d4 spawn_workers (rspamd + 0x4f0d4)
#16 0x0000555a474b89cd main (rspamd + 0x619cd)
#17 0x00007f3707141d90 __libc_start_call_main (libc.so.6 + 0x29d90)
#18 0x00007f3707141e40 __libc_start_main_impl (libc.so.6 + 0x29e40)
#19 0x0000555a4747d6d5 _start (rspamd + 0x266d5)
bt full
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/rspamd...
Reading symbols from /usr/lib/debug/.build-id/bd/d632bff3d4f219185aa53ea6f3b86b68e4ca5b.debug...
warning: Can't open file /dev/zero (deleted) during file-backed mapping note processing
[New LWP 125743]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `rspamd: normal process (localhost:11333) '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f3707f44a0c in redis_stat_runtime<float, true>::process_tokens (tokens=0x6190003807a0, this=0x6070056edb10) at ./src/libstat/backends/redis_backend.cxx:152
152 ./src/libstat/backends/redis_backend.cxx: No such file or directory.
(gdb) bt full
#0 0x00007f3707f44a0c in redis_stat_runtime<float, true>::process_tokens (tokens=0x6190003807a0, this=0x6070056edb10) at ./src/libstat/backends/redis_backend.cxx:152
idx = @0x7ffd5f4134f0: 2
val = @0x7ffd5f4134f4: 59
__for_range = <optimized out>
__for_begin = <optimized out>
__for_end = <optimized out>
tok = 0x62100083bc40
tok = <optimized out>
idx = <optimized out>
val = <optimized out>
__for_range = <optimized out>
__for_begin = <optimized out>
__for_end = <optimized out>
#1 rspamd_redis_classified (L=0x7f3700f18f60) at ./src/libstat/backends/redis_backend.cxx:863
filler_func = <optimized out>
opposite_rt_maybe = <optimized out>
cookie = <optimized out>
task = 0x62100101f998
rt = 0x6070056f03c0
result = <optimized out>
__func__ = <optimized out>
__PRETTY_FUNCTION__ = <optimized out>
#2 0x00007f37083ac2a6 in lj_BC_FUNCC () from /usr/lib/rspamd/librspamd-server.so
No symbol table info available.
#3 0x00007f3708398619 in lua_pcall () from /usr/lib/rspamd/librspamd-server.so
No symbol table info available.
#4 0x00007f370802ef79 in lua_redis_push_data (r=r@entry=0x6040011a03d0, ctx=ctx@entry=0x60e000720640, sp_ud=sp_ud@entry=0x60b003781c80) at ./src/lua/lua_redis.c:407
--Type <RET> for more, q to quit, c to continue without paging--c
err_idx = 1
ret = <optimized out>
ud = 0x60e000720648
cbs = <optimized out>
L = 0x7f3700f18f60
__func__ = "lua_redis_push_data"
#5 0x00007f370802f837 in lua_redis_callback (c=0x613000c58a00, r=0x6040011a03d0, priv=0x60b003781c80) at ./src/lua/lua_redis.c:479
reply = 0x6040011a03d0
sp_ud = 0x60b003781c80
ctx = 0x60e000720640
ud = 0x60e000720648
ac = <optimized out>
__func__ = "lua_redis_callback"
#6 0x00007f37082b7adc in __redisRunCallback (ac=ac@entry=0x613000c58a00, cb=cb@entry=0x7ffd5f413850, reply=<optimized out>) at ../contrib/hiredis/./contrib/hiredis/async.c:269
c = 0x613000c58a00
#7 0x00007f37082c8f04 in redisProcessCallbacks (ac=ac@entry=0x613000c58a00) at ../contrib/hiredis/./contrib/hiredis/async.c:470
c = 0x613000c58a00
cb = <optimized out>
reply = <optimized out>
status = 0
__PRETTY_FUNCTION__ = "redisProcessCallbacks"
#8 0x00007f37082c929f in redisAsyncHandleRead (ac=0x613000c58a00) at ../contrib/hiredis/./contrib/hiredis/async.c:539
c = 0x613000c58a00
#9 0x00007f3707c1fc20 in redisLibevReadEvent (loop=<optimized out>, watcher=<optimized out>, revents=<optimized out>) at ./contrib/hiredis/adapters/libev.h:53
e = <optimized out>
#10 0x00007f37077358d4 in ev_invoke_pending (loop=0x61a0014c0480) at ./contrib/libev/ev.c:3809
p = <optimized out>
#11 0x00007f3707744dcc in ev_run (loop=0x61a0014c0480, flags=flags@entry=0) at ./contrib/libev/ev.c:4231
__PRETTY_FUNCTION__ = "ev_run"
#12 0x0000555a474aa91e in ev_loop (flags=0, loop=<optimized out>) at ./contrib/libev/ev.h:830
No locals.
#13 start_worker (worker=0x61200161e3c0) at ./src/worker.c:524
ctx = 0x7f36ff2da2d0
is_controller = 0
__func__ = "start_worker"
#14 0x00007f3707d05cc2 in rspamd_handle_child_fork (wrk=wrk@entry=0x61200161e3c0, rspamd_main=rspamd_main@entry=0x6120000004c0, cf=cf@entry=0x60c00001b700, listen_sockets=listen_sockets@entry=0x61d000001980) at ./src/libserver/worker_util.c:1179
rc = <optimized out>
rlim = <optimized out>
it = <optimized out>
k = <optimized out>
v = <optimized out>
cur = <optimized out>
__func__ = "rspamd_handle_child_fork"
#15 0x00007f3707d06a1e in rspamd_fork_worker (rspamd_main=rspamd_main@entry=0x6120000004c0, cf=cf@entry=0x60c00001b700, index=index@entry=0, ev_base=ev_base@entry=0x7f3707752a40 <default_loop_struct>, term_handler=term_handler@entry=0x555a474a3d44 <rspamd_cld_handler>, listen_sockets=0x61d000001980) at ./src/libserver/worker_util.c:1296
wrk = 0x61200161e3c0
__func__ = "rspamd_fork_worker"
#16 0x0000555a474a3b42 in spawn_worker_type (rspamd_main=rspamd_main@entry=0x6120000004c0, event_loop=event_loop@entry=0x7f3707752a40 <default_loop_struct>, cf=cf@entry=0x60c00001b700) at ./src/rspamd.c:631
i = 0
__func__ = "spawn_worker_type"
#17 0x0000555a474a60d4 in spawn_workers (rspamd_main=rspamd_main@entry=0x6120000004c0, ev_base=ev_base@entry=0x7f3707752a40 <default_loop_struct>) at ./src/rspamd.c:717
cur = 0x619000022360
ls = <optimized out>
cf = 0x60c00001b700
p = <optimized out>
key = <optimized out>
bcf = 0x0
listen_ok = 1
seen_mandatory_workers = 0x619000385000
cw = <optimized out>
wrk = <optimized out>
i = <optimized out>
__func__ = "spawn_workers"
#18 0x0000555a474b89cd in main (argc=<optimized out>, argv=<optimized out>, env=<optimized out>) at ./src/rspamd.c:1678
i = <optimized out>
res = 0
signals = <optimized out>
sigpipe_act = <optimized out>
pworker = <optimized out>
type = <optimized out>
control_addr = <optimized out>
event_loop = 0x7f3707752a40 <default_loop_struct>
rspamd_main = 0x6120000004c0
skip_pid = 1
control_signals = <optimized out>
__func__ = "main"
stat_update_time = <optimized out>
(gdb) p tokens
$1 = (GPtrArray *) 0x6190003807a0
(gdb) p *tokens
$2 = {pdata = 0x621001216100, len = 367}
(gdb) p *tokens->pdata
$3 = (gpointer) 0x62100083bc10
(gdb)
Still crashing in the same function; still from the ASAN build.
Unfortunately, I have not a single idea now. It is not reproduced anywhere in our CI/test environments.
Thanks for all your feedback so far. I'll try to build a recreatable test case by tcpdump-ing the mail server traffic. Won't get around to it until next week, though, as I'll be out of town for business for a couple of days.
I think I have one idea now, hold on.
Do you have any per user script for the statistics?
I… don't think so. Don't even know what those are or how to check if I do, to be honest.
We're seeing the same segfaults on a Debian 12 bookworm system since version 3.8.0 (also rolled back to 3.7.5 as a stopgap). If you need more things to troubleshoot with, ask away.
I'm currently testing again with rspamd 3.8.4, let's see
Today I tried updating my server to Ubuntu 24.04. As the rspamd-provided APT repo doesn't exist for 24.04 yet, I tried updating to the rspamd version that Ubuntu 24.04 includes, 3.8.1. This version segfaulted within the first two minutes of running. Therefore I reverted to a snapshot created before the update.
Just as a heads-up for anyone else running into this & thinking about updating to 24.04.
Waiting for an rspamd-provided APT repo for Ubuntu 24.04 now.
I decided to give 3.8.4 a try on my Ubuntu 22.04 today, too. Unfortunately it still crashes, though the backtrace is slightly different:
Jun 29 17:31:13 firesword rspamd[17713]: <>; ; rspamd_crash_sig_handler: caught fatal signal 11(Segmentation fault), pid: 17713, trace:
Jun 29 17:31:16 firesword rspamd[17713]: <>; ; log_backtrace: #0: [0x7f39550ce349]: ./debian/build/flavor-release/src/./src/libserver/backtrace.cxx:31 in rspamd::log_backtrace()
Jun 29 17:31:16 firesword rspamd[16365]: <d0979b>; proxy; proxy_accept_socket: accepted milter connection from 2a01:4f8:c0:fb01::6 port 55124
Jun 29 17:31:17 firesword rspamd[17713]: <>; ; log_backtrace: #1: [0x7f395509c95f]: ./debian/build/flavor-release/src/./src/libserver/worker_util.c:1637 in rspamd_crash_sig_handler
Jun 29 17:31:17 firesword rspamd[17713]: <>; ; log_backtrace: #2: [0x7f395496351f]: /lib/x86_64-linux-gnu/libc.so.6 in __GI___sigaction
Jun 29 17:31:17 firesword rspamd[17713]: <>; ; log_backtrace: #3: [0x7f3955107f00]: ./debian/build/flavor-release/src/./src/libstat/backends/redis_backend.cxx:155 in redis_stat_runtime<float, true>::process_tokens(_GPtrArray*) const
Jun 29 17:31:17 firesword rspamd[17713]: <>; ; log_backtrace: #4: [0x7f39552b2be5]: languages.cc:0 in lj_BC_FUNCC
Jun 29 17:31:17 firesword rspamd[17713]: <>; ; log_backtrace: #5: [0x7f395529ef58]: /usr/lib/rspamd/librspamd-server.so in lua_pcall
Jun 29 17:31:17 firesword rspamd[17713]: <>; ; log_backtrace: #6: [0x7f395514d716]: ./debian/build/flavor-release/src/./src/lua/lua_redis.c:407 in lua_redis_push_data
Jun 29 17:31:17 firesword rspamd[17713]: <>; ; log_backtrace: #7: [0x7f3955213c8c]: ./debian/build/flavor-release/src/../contrib/hiredis/./contrib/hiredis/async.c:269 in __redisRunCallback
Jun 29 17:31:17 firesword rspamd[17713]: <>; ; log_backtrace: #8: [0x7f3954f1d515]: ./debian/build/flavor-release/contrib/libev/./contrib/libev/ev.c:3809 in ev_invoke_pending
Jun 29 17:31:17 firesword rspamd[17713]: <>; ; log_backtrace: #9: [0x7f3954f20dee]: ./debian/build/flavor-release/contrib/libev/./contrib/libev/ev.c:4231 in ev_run
Jun 29 17:31:17 firesword rspamd[17713]: <>; ; log_backtrace: #10: [0x5566f8d72506]: ./debian/build/flavor-release/src/./contrib/libev/ev.h:830 in ev_loop
Jun 29 17:31:17 firesword rspamd[17713]: <>; ; log_backtrace: #11: [0x7f395509bfff]: ./debian/build/flavor-release/src/./src/libserver/worker_util.c:1179 in rspamd_handle_child_fork
Jun 29 17:31:17 firesword rspamd[17713]: <>; ; log_backtrace: #12: [0x5566f8d701d3]: ./debian/build/flavor-release/src/./src/rspamd.c:374 in rspamd_fork_delayed_cb
Jun 29 17:31:17 firesword rspamd[17713]: <>; ; log_backtrace: #13: [0x7f3954f1d515]: ./debian/build/flavor-release/contrib/libev/./contrib/libev/ev.c:3809 in ev_invoke_pending
Jun 29 17:31:17 firesword rspamd[17713]: <>; ; log_backtrace: #14: [0x7f3954f20dee]: ./debian/build/flavor-release/contrib/libev/./contrib/libev/ev.c:4231 in ev_run
Jun 29 17:31:17 firesword rspamd[17713]: <>; ; log_backtrace: #15: [0x5566f8d614f0]: ./debian/build/flavor-release/src/./contrib/libev/ev.h:830 in ev_loop
Jun 29 17:31:17 firesword rspamd[17713]: <>; ; log_backtrace: #16: [0x7f395494ad8f]: ./csu/../sysdeps/nptl/libc_start_call_main.h:58 in __libc_start_call_main
Jun 29 17:31:17 firesword rspamd[17713]: <>; ; log_backtrace: #17: [0x7f395494ae3f]: ./csu/../csu/libc-start.c:392 in __libc_start_main_impl
Jun 29 17:31:17 firesword rspamd[17713]: <>; ; log_backtrace: #18: [0x5566f8d61a04]: /usr/bin/rspamd in _start
Jun 29 17:31:17 firesword rspamd[17713]: <>; ; log_backtrace: #19: [0xffffffffffffffff]: in
Jun 29 17:31:17 firesword rspamd[17713]: <>; ; rspamd_crash_sig_handler: please see Rspamd FAQ to learn how to dump core files and how to fill a bug report
As this is a production system I cannot leave it as is to provide further information, unfortunately. I've rolled back to the snapshot with 3.7.x.
It all look like a fixed issue when you disable BAYES_SPAM
or BAYES_HAM
via settings.
And https://github.com/rspamd/rspamd/pull/4932 is only in the master
Ah. Dang. I had read the other reports about disabling BAYES_SPAM
also requiring disabling BAYES_HAM
, but that didn't click for me as I was pretty sure that I never disabled BAYES_SPAM
. However, your replies made me actually look instead of relying on my faulty human brain, and I actually do have BAYES_SPAM
disabled, but not BAYES_HAM
.
I'll try updating again to 24.04 with Ubuntu's own 3.8.1 package & disabling BAYES_HAM
tomorrow. Let's see if it works for me, too.
No crash since updating to Ubuntu 24.04 with rspamd 3.8.1 Saturday evening. Looks like adding BAYES_HAM
to the disabled symbols fixed the crash for me, too.
Prerequisites
Describe the bug
We've been running rpsamd 3.7.x on Ubuntu 22.04 successfully for quite a while. We're using packages from rspamd.com/apt-stable/
On Friday we updated to the latest available package,
3.8.0-1~28391190c~jammy
. Shortly afterwards rspamd starts to segfault quite a bit. Not for each and every email, but for enough of them that keeping it running is unfeasible; this is a production system for our whole company.We've circumvented the problem by downgrading to
3.7.5-2~8c86c1676~jammy
again. No other changes were made.All other packages on this system are up to date.
The journal contains the following information:
Steps to Reproduce
Versions
See above, and:
Ubuntu 22.04 on amd64. Packages from rspamd.com/apt-stable/
Additional Information
I have not been able to run the ASAN builds for 3.8.0. Those crash right when starting the daemon, unfortunately.
I have installed the non-ASAN debugging symbols & run
coredumpctl debug <PID>
withbt full
for one of the crashes. The output was: