search

grosjo / fts-xapian

Dovecot FTS plugin based on Xapian
GNU Lesser General Public License v2.1
91 stars 19 forks source link

Memory error when trying to index large inbox #83

Closed ghost closed 3 years ago

ghost commented 3 years ago

While everything seems to work fine, I did encounter one problem with a large inbox (42GB), when generating indexes, I get:

: Error: FTS Xapian: Memory error 'std::bad_alloc'
: Error: Mailbox INBOX: Mail search failed: Internal error occurred. Refer to server log for more information.
: Error: Mailbox INBOX: Transaction commit failed: FTS transaction commit failed: transaction context (attempted to index 888 messages (UIDs 1..890))
grosjo commented 3 years ago

Can you clarify how many emails shall your search return ? How much RAM do you have on your server ?

ghost commented 3 years ago

I'm not doing a search, I'm using the doveadm command to index some old imap accounts. I used the command doveadm -v index -F emails.txt -q '*'.

One account has 61000 emails (25GB total size), another is of similar size, the total storage is 42GB. The server has 4GB of ram with 1.7GB of free ram. I am the only client connecting to the server.

This error appeared 11 times while indexing.

grosjo commented 3 years ago

PLease kindly set "verbose=1" in the conf file and restart the indexing and resend me the logs before and after the crash THank you

grosjo commented 3 years ago

@Bill-ColeUS How it goes on this ?

ghost commented 3 years ago

Unfortunately, I can't test this any more, because the system has gone into production. I will try to find a copy of an email account and test in a VM and report back.