Memory usage goes up until OOM-killer kills doveadm

nielspeen commented 3 years ago

The server has 6GB RAM of which approx 0.5GB is used by other processes.

fts-xapian is version 1.4.13.

Mailbox size is 2.2G.

Config:

plugin {

  fts = xapian
  fts_xapian = partial=2 full=20 verbose=2

  fts_autoindex = yes
  fts_enforced = yes

  fts_autoindex_exclude = \Trash

  # Index attachements
  #fts_decoder = decode2text

}

service indexer-worker {
  # Increase vsz_limit to 2GB or above.
  # Or 0 if you have rather large memory usable on your server, which is preferred for performance)
  vsz_limit = 2G
}

When running /usr/bin/doveadm -v fts optimize -A memory goes up quickly during the expunging process. Eventually output stops. (Always at the same UID=10165.)

While the output is stopped, dovadm is still running and memory usage continues to grow. Eventually the OOM-killer kicks in and kills doveadm.

doveadm(niels.peen@xxx.com): Info: FTS Xapian: Expunging UID=10163 '/var/vmail/xxx.com/niels.peen//Maildir/xapian-indexes/db_3f052c1163e45d61a1190000f0f6226a'
doveadm(niels.peen@xxx.com): Info: FTS Xapian: Expunging UID=10164 '/var/vmail/xxx.com/niels.peen//Maildir/xapian-indexes/db_3f052c1163e45d61a1190000f0f6226a'
doveadm(niels.peen@xxx.com): Info: FTS Xapian: Expunging UID=10165 '/var/vmail/xxx.com/niels.peen//Maildir/xapian-indexes/db_3f052c1163e45d61a1190000f0f6226a'
Killed

What can I do to reduce the memory usage? I've tried lowering the vsz_limit for the indexer-worker, but it seems to be ignored.

grosjo commented 3 years ago

@nielspeen Can you share the log before the few lines before getting killed ?

grosjo commented 3 years ago

(using the latest git for tracing the bug)

nielspeen commented 3 years ago

Please find the entire log attached. log.txt

grosjo commented 3 years ago

THank you Can you try again with latest git ?

nielspeen commented 3 years ago

It manages to continue a little longer now (3420 log lines vs 3323 in the previous one), but it still gets killed in the same way. Should I upload the new log?

grosjo commented 3 years ago

Can you kindly regenerate the log with the very latest git ?

nielspeen commented 3 years ago

New log attached. log.txt

nielspeen commented 3 years ago

The new debug info led me to try and raise XAPIAN_MIN_RAM.

When I set XAPIAN_MIN_RAM to 250L, it manages to keep the free mem around 100MB and the process completes normally:

doveadm(x.x@xxx.com): Warning: FTS Xapian: Free memory 119 MB vs 250 MB minimum
doveadm(x.x@xxx.com): Warning: FTS Xapian: Expunging with low memory (119 MB)-> Commiting on disk
doveadm(x.x@xxx.com): Info: FTS Xapian: Expunging (3a) UID=5363 (Free mem= 119 MB) '/var/vmail/xxx.com/x.x//Maildir/xapian-indexes/db_200d370a3add5d618e180000f0f6226a'
doveadm(x.x@xxx.com): Info: FTS Xapian: Expunging (3b) UID=5363 (Free mem= 119 MB) done
doveadm(x.x@xxx.com): Info: FTS Xapian: Expunging (5) '/var/vmail/xxx.com/x.x//Maildir/xapian-indexes/db_200d370a3add5d618e180000f0f6226a' done in 23.45 secs
doveadm(x.x@xxx.com): Info: FTS Xapian: Deinit /var/vmail/wanwire.com/x.x//Maildir/xapian-indexes)
doveadm(xx@xxx.com): Info: fts_backend_xapian_init : xapian
doveadm(xx@xxx.com): Info: FTS Xapian: Index path = /var/vmail/xxx.com/x//Maildir/xapian-indexes
doveadm(xx@xxx.com): Info: FTS Xapian: Starting with partial=2 full=20 verbose=2
doveadm(xx@xxx.com): Info: FTS Xapian: fts_backend_xapian_optimize '/var/vmail/xxx.com/x//Maildir/xapian-indexes'
doveadm(xx@xxx.com): Info: FTS Xapian: Deinit /var/vmail/xxx.com/x//Maildir/xapian-indexes)

Perhaps XAPIAN_MIN_RAM can be made configurable?