Closed brong closed 13 years ago
From: 32044
We currently have 1.200.000+ mailboxes split in 3 backend servers using 1 frontend / master server (both services running on the same server) for a grand total of 3.400.000+ mailboxes and subfolders.
the servers are used for imap access only. no pop3, no sieve scripts, no nntp, nothing else besides imap.
Now, the problem in question.
after some days running smoothly I start getting errors on the master/frontend server. It has happened a few times now..
So far I've seen it happen only on the master/frontend server.. not on the backend servers.
the errors:
cyrus/mupdate[1433]: DBERROR: skiplist recovery /var/lib/imap/mailboxes.db: 141BAD94 should be ADD or DELETE cyrus/mupdate[1433]: DBERROR: error updating database user.1215407: cyrusdb error
then after some time I start getting these:
cyrus/lmtp[1276]: authentication to remote mupdate server failed: EOF from server cyrus/lmtp[1276]: couldn't connect to 10.1.5.101: no authentication to server
after a little more time
cyrus/imap[30969]: kick_mupdate: can't connect to target: Connection refused cyrus/lmtp[2240]: mupdate-client: connection to server closed: end of file reached cyrus/lmtp[2240]: couldn't connect to 10.1.5.101: no connection to server
then this is what I get when trying to restart cyrus frontend/master
cyrus/ctl_cyrusdb[30607]: recovering cyrus databases cyrus/ctl_cyrusdb[30607]: DBERROR: skiplist recovery /var/lib/imap/mailboxes.db: 141BAD94 should be ADD or DELETE cyrus/ctl_cyrusdb[30607]: DBERROR: opening /var/lib/imap/mailboxes.db: cyrusdb error
what I need to do next is completely delete the dbs from the master/frontend and reimport all the mailboxes.
Thank you very much,
João Assad
Below is my frontend/master configuration
START {
recover cmd="ctl_cyrusdb -r" }
SERVICES {
mupdate cmd="mupdate -m" listen=3905 prefork=1 lmtp cmd="lmtpproxyd" listen="lmtp" prefork=0 imap cmd="proxyd" listen="imap" prefork=0 }
EVENTS {
checkpoint cmd="ctl_cyrusdb -c" period=240 }
configdirectory: /var/lib/imap partition-default: /tmp admins: cyrus sievedir: /var/lib/imap/sieve sendmail: /usr/sbin/sendmail hashimapspool: true sasl_pwcheck_method: saslauthd sasl_mech_list: PLAIN tls_cert_file: /usr/share/ssl/certs/crt.crt tls_key_file: /usr/share/ssl/certs/key.key tls_ca_file: /usr/share/ssl/certs/ca.ca
allowusermoves: 1
cyrus-be1_password: cyrus-be2_password: cyrus-be3_password: *** proxy_authname: cyrus
mupdate_server:10.1.5.101 mupdate_authname:cyrus mupdate_password: ***
maxmessagesize: 2097152 syslog_prefix:cyrus lmtp_over_quota_perm_failure:1 quotawarn: 110 imapidlepoll: 0 fulldirhash: 1 munge8bit: 0 timeout: 10
tls_session_timeout: 0 mupdate_connections_max: 1024 berkeley_cachesize:102400 berkeley_txns_max:500
Fedora bugzilla bug id 152548 https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=152548
From: 32044
New info
It seems before the corruption , I always get the following error
cyrus/mupdate[12614]: IOERROR: mapping /var/lib/imap/mailboxes.db file: Cannot allocate memory cyrus/mupdate[12614]: failed to mmap /var/lib/imap/mailboxes.db file cyrus/master[12580]: service mupdate pid 12614 in READY state: terminated abnormally
This error is given by the map_refresh method ... there are 2 declarations of the method. one in lib/map_stupidshared.c and another in lib/map_shared.c .
I dont know which one cyrus use when compiled under fedora.
Regards
From: 32044
2 gdb backtraces
s=0x8d52f070 "Internal error: assertion failed: mupdate.c: 586: 0", code=75) at mupdate.c:586
expr=0x8082ce1 "0") at assert.c:61
s=0x8d52f4c0 "failed to mmap /var/lib/imap/mailboxes.db file", code=75) at mupdate.c:586
len=0x8ad56e4, newlen=366993408, name=0x8ad5710 "/var/lib/imap/mailboxes.db", mboxname=0x0) at map_shared.c:105
at cyrusdb_skiplist.c:572
at cyrusdb_skiplist.c:1345
keylen=12, data=0x8c5a248 "1 cyrus-be2.gazzag.com!default 1440034\tlrswipcda\t", datalen=49, tid=0x0, overwrite=1) at cyrusdb_skiplist.c:1225
keylen=12, data=0x8c5a248 "1 cyrus-be2.gazzag.com!default 1440034\tlrswipcda\t", datalen=49, tid=0x0) at cyrusdb_skiplist.c:1244
mbtype=0, host=0x8b144f0 "cyrus-be2.gazzag.com!default", acl=0x8c4dab8 "1440034\tlrswipcda\t", tid=0x0) at mboxlist.c:801
mailbox=0x8bb6ec8 "user.1440034", server=0x8bb54b8 "cyrus-be2.gazzag.com!default", acl=0x8c36e28 "1440034\tlrswipcda\t", t=SET_ACTIVE) at mupdate.c:1527
s=0x9fcd6070 "Internal error: assertion failed: mupdate.c: 586: 0", code=75) at mupdate.c:586
expr=0x8082ce1 "0") at assert.c:61
s=0x9fcd64c0 "failed to mmap /var/lib/imap/mailboxes.db file", code=75) at mupdate.c:586
len=0x85856e4, newlen=366583808, name=0x8585710 "/var/lib/imap/mailboxes.db", mboxname=0x0) at map_shared.c:105
at cyrusdb_skiplist.c:572
at cyrusdb_skiplist.c:1345
key=0x88046a78 "user.950836._TRASH", keylen=18, data=0x88056cc0 "1 cyrus-be3.gazzag.com!default 950836\tlrswipcda\t", datalen=48, tid=0x0, overwrite=1) at cyrusdb_skiplist.c:1225
keylen=18, data=0x88056cc0 "1 cyrus-be3.gazzag.com!default 950836\tlrswipcda\t", datalen=48, tid=0x0) at cyrusdb_skiplist.c:1244
name=0x88046a78 "user.950836._TRASH", mbtype=0, host=0x88019a70 "cyrus-be3.gazzag.com!default", acl=0x8804b7c0 "950836\tlrswipcda\t", tid=0x0) at mboxlist.c:801
mailbox=0x9ddb1140 "user.950836._TRASH", server=0x9ddb5360 "cyrus-be3.gazzag.com!default", acl=0x88016f70 "950836\tlrswipcda\t", t=SET_ACTIVE) at mupdate.c:1527
Attachment-Id: 348 From: 32044 Type: text/plain File: map_shared-mremap.patch
patch to replace munmap followed by mmap for a mremap
From: Sergio Bruder
any additional work on this particular bug? the attached patch really solves it? any plan to include it in the next release?
From: Ken Murchison
Both the problem and solution appear to be GNU and/or FC specific. At the very least, the use of mremap() has to be a compile-time decision, with all of the necessary configure foo.
From: Sergio Bruder
you mean linux (the kernel) specific. Ive saw this very same problem in CentOS (ok, not far from a FC) and SuSE.
From: 32044 Bugzilla-Id: 2640 Version: 2.2.x Owner: Ken Murchison