search

spamhaus / rspamd-dqs

Spamhaus code for RSPAMD Plugin. See https://docs.spamhaustech.com/40-real-world-usage/Rspamd/000-intro.html for instructions
Apache License 2.0
60 stars 11 forks source link

Memory usage with url_to_hash for SPAMHAUS_HBL_URL #26

Open japc opened 5 months ago

japc commented 5 months ago

Hi,

Have had an increase of memory consumption with SPAMHAUS_HBL_URL enabled. Usually the rspamd normal processes take 100MB to 400MB but with SPAMHAUS_HBL_URL on there are some consuming about 3GB.

Tried in a couple of servers:

with SPAMHAUS_HBL_URL enabled on both, both have the huge mem processes with SPAMHAUS_HBL_URL only on one of them, only that will have the huge mem processes

Straced the rspamd workers since the rspamd start and I can see an increasing memory allocation:

6158  14:09:02.075729 mmap(0x7f1439d73000, 2101248, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1439b52000
6158  14:09:02.117232 mremap(0x7f1439751000, 4198400, 8392704, MREMAP_MAYMOVE) = 0x7f14363b2000
....
6158  14:09:29.035418 mmap(0x7f141f7ae000, 805310464, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f13873aa000
6158  14:09:31.988004 munmap(0x7f14077ad000, 402657280) = 0
6158  14:09:38.889077 mmap(0x7f13b73ab000, 1610616832, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f13273a9000
6158  14:09:44.383377 munmap(0x7f13873aa000, 805310464) = 0
6158  14:09:59.413881 writev(6, [{iov_base="2024-03-21 14:09:59 #6158(normal) ", iov_len=34}, {iov_base="<25f1da>; task; lua_metric_symbol_callback: ", iov_len=44}, {iov_base="call to (RBL_SPAMHAUS_HBL_URL) failed (2): table overflow; trace: [1]:{[C]:-1 - search [C]}; [2]:{/etc/rspamd/rspamd.local.lua:209 - url_to_hash [Lua]}; [3]:{/etc/rspamd/rspamd.local.lua:248 - <unknown> [Lua]};", iov_len=210}, {iov_base="\n", iov_len=1}], 4) = 289

And, although from the trace the memory appears to be unmapped, it is still allocated by the process:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6158 _rspamd   20   0 3713760   3.3g  30708 S   2.3   5.3   3:26.04 rspamd
7f13b73ab000-7f13f73ac000 rw-p 00000000 00:00 0
Size:            1048580 kB
Rss:             1048580 kB
Pss:             1048580 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:   1048580 kB
Referenced:      1048552 kB
Anonymous:       1048580 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB

The mem not being allocated may be a different problem, but there seems to be something weird going on on the url algo table processing.

pmcintyre commented 5 months ago

At spamhaus we noticed the memory loss too, and traced it to the rspamd_regexp plugin. We are busy making a workaround without the regexp plugin that still preserves practically all functionality. Hope to have a solution in a few days. Thanks for the detailed bug report, and for raising the rspamd issue!

pmcintyre commented 5 months ago

The regexp issue got a workaround, and we've rewritten the leaky cryptowallet match code. Should be fixed now, our own testing no longer shows memory loss issues.

japc commented 5 months ago

Thank you.

There was also a rspamd problem with the regex code that was promptly fixed (issue linked).

pmcintyre commented 5 months ago

There was also a rspamd problem with the regex code that was promptly fixed (issue linked).

Yes, I've seen it. In fact the current release mentions that issue because it still uses PCRE but the regexes should be safe to use even for rspamd versions that still contain the bug.

Thank YOU very much for your excellent debugging work, finding an example URL that triggered the issue and the point in the code where it went wrong!