rspamd / rspamd

Rapid spam filtering system.
Other
2.01k stars 376 forks source link

[BUG] rspamadm dmarc_report does not work with redis sentinel #5055

Open rekup opened 1 month ago

rekup commented 1 month ago

Prerequisites

Describe the bug When using redis-sentinel the rspamadm dmarc_report command will not send any dmarc reports unless the first server defined in the redis configs servers option is the current primary (and thus writable) node.

Steps to Reproduce

  1. Create a redis-sentinel cluster
  2. define all redis nodes in the servers option of the rspamd redis config while making sure the first defined server is not the current primary server
  3. run the rspamadm dmarc_report -v command

Expected behavior I would expect that the reports will be sent successfully.

Versions

root@mx03.example.org:~$ egrep "NAME|VERSION" /etc/os-release 
NAME="Rocky Linux"
VERSION="9.4 (Blue Onyx)"

root@mx03.example.org:~$ rpm -qa|grep rspamd
rspamd-3.8.4-1.el9.x86_64

root@mx03.example.org:~$ rspamd --version
Rspamd daemon version 3.8.4

CPU architecture x86_64; features: avx2, avx, sse2, sse3, ssse3, sse4.1, sse4.2, rdrand
Hyperscan enabled: TRUE
Jemalloc enabled: TRUE
LuaJIT enabled: TRUE (LuaJIT version: LuaJIT 2.1.1707061634)
ASAN enabled: FALSE
BLAS enabled: TRUE
Fasttext enabled: TRUE

Additional Information

If we run the command, we get the following output:

bash-5.1$ rspamadm dmarc_report -v
previous last report date is 1720735200
Process date 20240714
send data for 0 domains (from 1 to 0)
bash-5.1$ echo $?
0

It seems like there is no error, but there are also no reports sent. We then used strace to troubleshoot further:

1421377 read(45, "+OK\r\n-READONLY You can't write a"..., 16384) = 61
1421377 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 46
1421377 fcntl(46, F_GETFL)              = 0x2 (flags O_RDWR)
1421377 fcntl(46, F_SETFL, O_RDWR|O_NONBLOCK) = 0
1421377 setsockopt(46, SOL_TCP, TCP_NODELAY, [1], 4) = 0

As soon as rspamd tries to rename a key in redis probably here: https://github.com/rspamd/rspamd/blob/27b858776562a743a6036d5f6e1ee022d2006f79/lualib/rspamadm/dmarc_report.lua#L485 we get an error which indicates that rspamd tried to write to a read only replica of redis. If we run the command with the --no-opt flag (which does not perform any write operatios to redis) we can see that there are multiple reports available.

I suspect that https://github.com/rspamd/rspamd/blob/27b858776562a743a6036d5f6e1ee022d2006f79/lualib/lua_redis.lua#L78 is not being called during rspamadm dmarc_report. As a result, rspamd will use the first redis server defined in the redis config:

bash-5.1$ cat /etc/rspamd/local.d/redis.conf 
# Warning: File is managed by Ansible
servers = "mailstore02.example.org,mx03.example.org,mx04.example.org";
password = REDACTED;
sentinels = mailstore02.example.org:26379,mx03.example.org:26379,mx04.example.org:26379;
sentinel_watch_time = 1min;

If the first server in the list is a replica, it will be read-only and the rspamadm dmarc_report command will not send any reports and fail silently.

vstakhov commented 1 month ago

Rspamadm does not execute on_load or periodic scripts, hence it cannot deal with Sentinels resolution. It can probably be fixed but it'll require quite a lot of manual efforts for little benefit so I'd set it as a very low priority task (probably never to be done).