search

grosjo / fts-xapian

Dovecot FTS plugin based on Xapian
GNU Lesser General Public License v2.1
91 stars 19 forks source link

Xapian Memory Usage #118

Closed DerLinkman closed 2 years ago

DerLinkman commented 2 years ago

Hey,

can you explain me how Xapian handles the memory usage? I´m a bit confused tbh.

I´ve assigned 2048M for the index workers.

But Xapian says (when dovecot is freshly started) about a Gigabyte more than i´ve set (lowmem vs freemem)

After i start a reindex and some searches the freemem drops and won´t recover.

(Currently 450MB instead of 2048) what is the reason for that?

Is there a way to clean the Xapian "Cache" which flush the used Xapian Ram?

Thx for your help and your awesome plugin!

Kind regards Niklas

grosjo commented 2 years ago

The memory shall recover after indexing or searching, so there seems to be a leaking somewhere

Can you tell which version of the plugin are you using ? Can you test with latest git ?

DerLinkman commented 2 years ago

Hi yes sure.

We build the xapian plugin by ourselfs from the latest git. I rebuild it just recently, so it is commit: https://github.com/grosjo/fts-xapian/commit/c6e4ab0c3925d8fecb4d6205dccdfcfb4981461f

grosjo commented 2 years ago

Hi, any update on this ?

DerLinkman commented 2 years ago

Hi,

i´ll try it with the latest Git Commit https://github.com/grosjo/fts-xapian/commit/1a7361e1d1169f10546b6854cb335ed73d529bdd

I´ll keep you up to date!

DerLinkman commented 2 years ago

Sorry, i´ll test this tomorrow :(

DerLinkman commented 2 years ago

Hi i think it got even worse with the latest patch.

Now the plugin is always in low memory mode even if i allocated 2048 MB the configuration of the plugin is untouched.

Please let me know if you need something else from me.

Build: https://github.com/grosjo/fts-xapian/commit/13f08da2e1ed13412e600348ce72982f2163572f

DerLinkman commented 2 years ago

Now the plugin is always in low memory mode even if i allocated 2048 MB the configuration of the plugin is untouched.

That was a false positive it is not running in lowmemory mode everytime, sorry for the confusion.

But the issue still seems to appear. If i reindexed the mails with doveadm fts optimize -A i get that in the logs: Info: FTS Xapian: Starting with partial=3 full=20 verbose=1 lowmemory=250 MB vs freemem=262 MB .

I saw something that might have something to do with it: In htop the RAM of the dovecot/index-workers is only cleaned up after a few minutes, once this happens the RAM count of the fts-xapian log increases again.

DerLinkman commented 2 years ago

Is this also the reason why the reindexing process goes into LowMemory mode after some time?

It will probably cache the files in RAM before writing them to disk or not? Then the RAM should be empty again after a write process to the disk theoretically (if it is so at all, unfortunately I'm not that deep in the matter :( ).