search

cyrusimap / cyrus-imapd

Cyrus IMAP is an email, contacts and calendar server
http://cyrusimap.org
Other
531 stars 146 forks source link

IOERROR: locking index ... Unknown code ____ 255 #1582

Open brong opened 12 years ago

brong commented 12 years ago

From: Mark Nipper Bugzilla-Id: 3720 Version: 2.4.16 Owner: Bron Gondwana

brong commented 12 years ago

From: Mark Nipper

Per my conversation with Bron via email, I'm opening this report concerning the error message above. Here are the pertinent parts of our conversation:

On Mon, Jul 16, 2012, at 04:38 PM, Mark Nipper wrote: > On 16 Jul 2012, Bron Gondwana wrote: > > On Mon, Jul 16, 2012, at 01:29 PM, Mark Nipper wrote: > > > Jul 16 04:03:14 king cyrus/lmtpunix[17822]: Delivered: <201207160902.q6G925Fu018045@topcat.laits.utexas.edu> to mailbox: user.nipsy@bitgnome^net.utexas.laitsadmin.lw > > > Jul 16 04:04:37 king cyrus/lmtpunix[17868]: Delivered: <201207160902.q6G922EO025278@vm-cake.la.utexas.edu> to mailbox: user.nipsy@bitgnome^net.utexas.laitsadmin.lw > > > > Thanks Mark - can you tell us what's in your mailboxes.db for nipsy@bitgnome.net? > > You may very well have a bug - even those successful deliveries look wrong - I > > would expect to be seeing user/nipsy/utexas/laitsadmin/lw@bitgnome.net or > > bitgnome.net!user.nipsy.utexas.laitsadmin.lw > > > > It's more likely that this configuration is buggy, since we don't run unixhs at > > FastMail, so I haven't used it as much. > > So, running (let me know if this isn't right): > --- > cyr_dbtool /var/lib/cyrus/mailboxes.db skiplist show | grep nipsy@bitgnome.net > > shows me things like: > --- > user.nipsy@bitgnome^net 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda > user.nipsy@bitgnome^net.Deleted Items 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda > user.nipsy@bitgnome^net.Drafts 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda > user.nipsy@bitgnome^net.Junk 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda > user.nipsy@bitgnome^net.Sent 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda > user.nipsy@bitgnome^net.Sent Items 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda > user.nipsy@bitgnome^net.Trash 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda > user.nipsy@bitgnome^net.bitgnome 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda > user.nipsy@bitgnome^net.cyrus.info 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda > user.nipsy@bitgnome^net.nagios 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda > user.nipsy@bitgnome^net.utexas 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda > user.nipsy@bitgnome^net.utexas.laitsadmin 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda > user.nipsy@bitgnome^net.utexas.laitsadmin.lw 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda > user.nipsy@bitgnome^net.utexas.laitsadmin.monit 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda > > That's not nearly all of them, but a decent subset anyway. > > Let me know if you need me to check anything else. > Should I be worried about potential mail loss? It looks like the > default behavior is to dump messages into the root INBOX at > least, so it seems safe enough for now.

Oooh. Ok, so you're running "virtdomains: off".

I'm really not happy about the way virtdomains: off interacts with things. There's all sorts of mess around mailbox names. But of course, your username really is actually "nipsy@bitgnome.net" rather than just "nipsy".

Yes, I think this is OK. But I'm not surprised there's a bug, because it's a weird configuration.

brong commented 11 years ago

From: Bron Gondwana

Weird or not, this is a blocker for 2.5 - gotta work right.

brong commented 11 years ago

From: Mark Nipper

As an update to this, I migrated to using "virtdomains: yes", and I'm still seeing problems with Sieve and delivery to the proper mailboxes. Here's my current cyr_dbtool:

cyr_dbtool /var/lib/cyrus/mailboxes.db skiplist show | grep nipsy@bitgnome.net bitgnome.net!user.nipsy 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda
bitgnome.net!user.nipsy.Deleted Items 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda
bitgnome.net!user.nipsy.Drafts 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda
bitgnome.net!user.nipsy.Junk 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda
bitgnome.net!user.nipsy.Sent 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda
bitgnome.net!user.nipsy.Sent Items 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda
bitgnome.net!user.nipsy.Trash 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda
bitgnome.net!user.nipsy.bitgnome 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda
bitgnome.net!user.nipsy.cyrus.info 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda
bitgnome.net!user.nipsy.nagios 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda
bitgnome.net!user.nipsy.utexas 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda
bitgnome.net!user.nipsy.utexas.laitsadmin 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda
bitgnome.net!user.nipsy.utexas.laitsadmin.lw 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda
bitgnome.net!user.nipsy.utexas.laitsadmin.monit 0 default nipsy@bitgnome.net lrswipcda admin lrswipcda

It's also worth mentioning that I have the following defined:

bypass BDB hell

annotation_db: skiplist duplicate_db: skiplist mboxlist_db: skiplist ptscache_db: skiplist quota_db: skiplist seenstate_db: skiplist subscription_db: skiplist tlscache_db: skiplist

The way it's behaving makes me think "race condition" because typically what happens is I get a ton of email all at the same time for things like Nagios, cron, TSM, or logwatch and some of the messages will end up in the proper subfolder, others will end up a little higher up (because of a more generic Sieve rule), and then a few will make it straight into my INBOX as the default when everything else fails to match.

Anyway, let me know if I should not be using skiplist for something specifically. I can switch back to using BDB if need be for certain things. I just hated the problems that came from using it previously when switching machines or upgrading across different versions of BDB. But I'd also rather things work 100% of the time too. :)

brong commented 11 years ago

From: Bron Gondwana

This is really odd. If you can give me some more detail - exact imapd.conf, the sieve script, and syslog from the next time you see it, it will help!

Thanks,

Bron.

brong commented 11 years ago

Attachment-Id: 1506 From: Mark Nipper Type: application/octet-stream File: sieve-brokeness.tar.xz

Relevant files to weird sieve behavior.

brong commented 11 years ago

From: Mark Nipper

What's weird is that I'm not even getting errors directly related to misfiled messages at this point. Things are just getting misfiled. I'm including an example message and all the relevant files (and snippets from various logs, which are mostly the same information over and over). The weird thing about this one example is that while the first rule that should take affect fails to happen, one further down the rule chain actually does end up happening. But nothing is consistent from one day to the next. Sometimes stuff gets sorted correctly. Sometimes it partially fails. Sometimes, it fails entirely and hits the catch all at the end which is just my INBOX.

brong commented 11 years ago

From: Paweł Kwiatkowski

Hi,

I have the same problem with locking folders. A small example:

Jan 11 11:13:24 k3 cyrus/imaps[9545]: IOERROR: locking index for user.awiman.serwery.cron: Unknown code ____ 255 Jan 11 11:13:24 k3 cyrus/imaps[9545]: IOERROR: locking index user.awiman.serwery.cron: System I/O error

The process do noting:

strace -p 9545

Process 9545 attached - interrupt to quit select(1, [0], NULL, NULL, {1694, 752062}^C <unfinished ...> Process 9545 detached

And lsof: -cut----------------------------------------------------------- root@k3|11:17:32|/data/home/pkwiatko# lsof +p 9545 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME imapd 9545 cyrus cwd DIR 252,2 4096 2 / imapd 9545 cyrus rtd DIR 252,2 4096 2 / imapd 9545 cyrus txt REG 252,2 1630504 427499 /usr/lib/cyrus/bin/imapd imapd 9545 cyrus mem REG 252,1 1849280 39092459 /data/cyrus/spool/a/user/awiman/serwery/cron/cyrus.index imapd 9545 cyrus mem REG 252,1 848712 38551965 /data/cyrus/spool/a/user/awiman/cyrus.cache imapd 9545 cyrus mem REG 252,1 70016 38551963 /data/cyrus/spool/a/user/awiman/cyrus.index imapd 9545 cyrus mem REG 252,3 888960 1050711 /var/cache/nscd/group imapd 9545 cyrus mem REG 252,2 31104 135753 /lib/x86_64-linux-gnu/libnss_dns-2.15.so imapd 9545 cyrus mem REG 252,2 52120 135742 /lib/x86_64-linux-gnu/libnss_files-2.15.so imapd 9545 cyrus mem REG 252,1 3258132 27902185 /data/cyrus/imap-conf/mailboxes.db imapd 9545 cyrus mem REG 252,2 18512 395754 /usr/lib/x86_64-linux-gnu/sasl2/libplain.so.2.0.25 imapd 9545 cyrus mem REG 252,2 34896 395753 /usr/lib/x86_64-linux-gnu/sasl2/libntlm.so.2.0.25 imapd 9545 cyrus mem REG 252,2 18512 395749 /usr/lib/x86_64-linux-gnu/sasl2/liblogin.so.2.0.25 imapd 9545 cyrus mem REG 252,2 18512 395756 /usr/lib/x86_64-linux-gnu/sasl2/libanonymous.so.2.0.25 imapd 9545 cyrus mem REG 252,2 43288 135741 /lib/x86_64-linux-gnu/libcrypt-2.15.so imapd 9545 cyrus mem REG 252,2 664504 391853 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6 imapd 9545 cyrus mem REG 252,2 300704 407186 /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0 imapd 9545 cyrus mem REG 252,2 61264 407180 /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0 imapd 9545 cyrus mem REG 252,2 166096 407183 /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0 imapd 9545 cyrus mem REG 252,2 14352 136203 /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0 imapd 9545 cyrus mem REG 252,2 72752 407155 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0 imapd 9545 cyrus mem REG 252,2 67864 391664 /usr/lib/x86_64-linux-gnu/libtasn1.so.3.1.12 imapd 9545 cyrus mem REG 252,2 86024 407149 /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0 imapd 9545 cyrus mem REG 252,2 209184 407177 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0 imapd 9545 cyrus mem REG 252,2 655152 407152 /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0 imapd 9545 cyrus mem REG 252,2 547488 407189 /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0 imapd 9545 cyrus mem REG 252,2 27624 407192 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0 imapd 9545 cyrus mem REG 252,2 516056 130606 /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0 imapd 9545 cyrus mem REG 252,2 766856 391676 /usr/lib/x86_64-linux-gnu/libgnutls.so.26.21.8 imapd 9545 cyrus mem REG 252,2 252312 407195 /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0 imapd 9545 cyrus mem REG 252,2 105288 135737 /lib/x86_64-linux-gnu/libresolv-2.15.so imapd 9545 cyrus mem REG 252,2 55528 399764 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.1 imapd 9545 cyrus mem REG 252,2 314936 395761 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.1 imapd 9545 cyrus mem REG 252,2 22520 421322 /usr/lib/x86_64-linux-gnu/sasl2/libldapdb.so.2.0.25 imapd 9545 cyrus mem REG 252,2 22432 395764 /usr/lib/x86_64-linux-gnu/sasl2/libsasldb.so.2.0.25 imapd 9545 cyrus mem REG 252,2 55760 395755 /usr/lib/x86_64-linux-gnu/sasl2/libdigestmd5.so.2.0.25 imapd 9545 cyrus mem REG 252,2 22608 395757 /usr/lib/x86_64-linux-gnu/sasl2/libcrammd5.so.2.0.25 imapd 9545 cyrus mem REG 252,1 42360832 27902117 /data/cyrus/imap-conf/db/db.012 imapd 9545 cyrus mem REG 252,1 3145728 27902055 /data/cyrus/imap-conf/db/db.011 imapd 9545 cyrus mem REG 252,1 1318912 27902052 /data/cyrus/imap-conf/db/db.010 imapd 9545 cyrus mem REG 252,1 1318912 27902051 /data/cyrus/imap-conf/db/db.009 imapd 9545 cyrus mem REG 252,1 1318912 27902049 /data/cyrus/imap-conf/db/db.008 imapd 9545 cyrus mem REG 252,1 1318912 27902046 /data/cyrus/imap-conf/db/db.007 imapd 9545 cyrus mem REG 252,1 1318912 27902044 /data/cyrus/imap-conf/db/db.006 imapd 9545 cyrus mem REG 252,1 1318912 27902043 /data/cyrus/imap-conf/db/db.005 imapd 9545 cyrus mem REG 252,1 1318912 27902038 /data/cyrus/imap-conf/db/db.004 imapd 9545 cyrus mem REG 252,1 7380992 27901968 /data/cyrus/imap-conf/db/db.002 imapd 9545 cyrus mem REG 252,2 97248 135755 /lib/x86_64-linux-gnu/libnsl-2.15.so imapd 9545 cyrus mem REG 252,2 135366 135734 /lib/x86_64-linux-gnu/libpthread-2.15.so imapd 9545 cyrus mem REG 252,2 14768 135728 /lib/x86_64-linux-gnu/libdl-2.15.so imapd 9545 cyrus mem REG 252,2 1811128 135730 /lib/x86_64-linux-gnu/libc-2.15.so imapd 9545 cyrus mem REG 252,2 36432 136242 /lib/x86_64-linux-gnu/libwrap.so.0.7.6 imapd 9545 cyrus mem REG 252,2 14696 130335 /lib/x86_64-linux-gnu/libcom_err.so.2.1 imapd 9545 cyrus mem REG 252,2 92720 130619 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 imapd 9545 cyrus mem REG 252,2 1852792 130349 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 imapd 9545 cyrus mem REG 252,2 374608 130347 /lib/x86_64-linux-gnu/libssl.so.1.0.0 imapd 9545 cyrus mem REG 252,2 1518928 391142 /usr/lib/x86_64-linux-gnu/libdb-5.1.so imapd 9545 cyrus mem REG 252,2 109288 395768 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25 imapd 9545 cyrus mem REG 252,2 149280 135748 /lib/x86_64-linux-gnu/ld-2.15.so imapd 9545 cyrus mem REG 252,3 438990 1050696 /var/cache/nscd/passwd imapd 9545 cyrus mem REG 252,1 134876 27902223 /data/cyrus/imap-conf/tls_sessions.db imapd 9545 cyrus mem REG 252,1 49152 27902118 /data/cyrus/imap-conf/db/db.013 imapd 9545 cyrus mem REG 252,1 1318912 27901970 /data/cyrus/imap-conf/db/db.003 imapd 9545 cyrus mem REG 252,1 144 27902178 /data/cyrus/imap-conf/annotations.db imapd 9545 cyrus mem REG 252,1 24576 27901964 /data/cyrus/imap-conf/db/__db.001 imapd 9545 cyrus 0u IPv4 87832232 0t0 TCP k3.oelne.pl:imaps->awiman.cen.oelne.pl:6707 (ESTABLISHED) imapd 9545 cyrus 1u IPv4 87832232 0t0 TCP k3.oelne.pl:imaps->awiman.cen.oelne.pl:6707 (ESTABLISHED) imapd 9545 cyrus 2u IPv4 87832232 0t0 TCP k3.oelne.pl:imaps->awiman.cen.oelne.pl:6707 (ESTABLISHED) imapd 9545 cyrus 3w FIFO 0,8 0t0 67525457 pipe imapd 9545 cyrus 4u IPv4 67525456 0t0 TCP *:imaps (LISTEN) imapd 9545 cyrus 5u unix 0xffff88009e5e6180 0t0 87061823 socket imapd 9545 cyrus 6u REG 252,1 3258132 27902185 /data/cyrus/imap-conf/mailboxes.db imapd 9545 cyrus 7r FIFO 0,8 0t0 67540398 pipe imapd 9545 cyrus 8w FIFO 0,8 0t0 67540398 pipe imapd 9545 cyrus 9r FIFO 0,8 0t0 67540399 pipe imapd 9545 cyrus 10w FIFO 0,8 0t0 67540399 pipe imapd 9545 cyrus 11u REG 252,1 0 27903389 /data/cyrus/imap-conf/user_deny.db imapd 9545 cyrus 12u unix 0xffff88009e5e57c0 0t0 87061824 socket imapd 9545 cyrus 13u REG 252,1 144 27902178 /data/cyrus/imap-conf/annotations.db imapd 9545 cyrus 14u REG 252,1 0 27902226 /data/cyrus/imap-conf/socket/imaps-0.lock imapd 9545 cyrus 15u REG 0,15 56 87888863 /run/cyrus/proc/9545 imapd 9545 cyrus 16u REG 252,1 134876 27902223 /data/cyrus/imap-conf/tls_sessions.db imapd 9545 cyrus 17uR REG 0,15 0 115958 /run/cyrus/lock/a/user/awiman.lock imapd 9545 cyrus 18u REG 252,1 70016 38551963 /data/cyrus/spool/a/user/awiman/cyrus.index imapd 9545 cyrus 19r REG 252,1 210 38551928 /data/cyrus/spool/a/user/awiman/cyrus.header imapd 9545 cyrus 20u REG 252,1 848712 38551965 /data/cyrus/spool/a/user/awiman/cyrus.cache imapd 9545 cyrus 21uR REG 0,15 0 210089 /run/cyrus/lock/a/user/awiman/serwery/cron.lock imapd 9545 cyrus 22u REG 252,1 1849280 39092459 /data/cyrus/spool/a/user/awiman/serwery/cron/cyrus.index -cut-----------------------------------------------------------

To fix it I have to kill locked processes and reconstruct folder.

Details:

/etc/cyrus.conf -cut----------------------------------------------------------- START {

Do not delete this entry!

recover cmd="ctl_cyrusdb -r"

This is only necessary if using idled for IMAP IDLE.

idled cmd="idled" }

UNIX sockets start with a slash and are put into /var/imap/socket.

SERVICES {

Add or remove based on preferences.

imap cmd="imapd" listen="127.0.0.1:imap2" prefork=0 pop3 cmd="pop3d" listen="127.0.0.1:pop-3" prefork=0 imap2 cmd="imapd" listen="10.2.111.2:imap2" prefork=0 imap3 cmd="imapd" listen="10.2.112.2:imap2" prefork=0

Don't forget to generate the needed keys for SSL or TLS

(see doc/html/install-configure.html).

imaps cmd="imapd -s" listen="imaps" prefork=0 pop3s cmd="pop3d -s" listen="pop3s" prefork=0

sieve cmd="timsieved" listen="sieve" prefork=0

at least one LMTP is required for delivery

lmtp cmd="lmtpd" listen="lmtp" prefork=0

lmtpunix cmd="lmtpd -T 300" listen="/var/run/cyrus/socket/lmtp" prefork=3

this is only necessary if using notifications

notify cmd="notifyd" listen="/var/run/cyrus/socket/notify" proto="udp" prefork=1

}

EVENTS {

This is required.

checkpoint cmd="ctl_cyrusdb -c" period=30

This is only necessary if using duplicate delivery suppression.

delprune cmd="cyr_expire -E 3" at=0400

This is only necessary if caching TLS sessions.

tlsprune cmd="tls_prune" at=0400 } -cut-----------------------------------------------------------

/etc/imapd.conf -cut-----------------------------------------------------------

when you change these directories (read the docs).

configdirectory: /data/cyrus/imap-conf defaultpartition: default

Polish active users will go to this partition

partition-default: /data/cyrus/spool

inactive users will go here

partition-backup: /backup/cyrus/backup

remote (UK, IE, AE) users will go to this partition

partition-sata: /backup/cyrus/sata sievedir: /data/cyrus/imap-conf/sieve

tls_ca_path: /etc/ssl/certs

tls_ca_file: /etc/imapd/ssl/rapidssl-ca-intermediate.cert.pem tls_cert_file: /etc/imapd/ssl/wildcard.cert.pem tls_key_file: /etc/imapd/ssl/wildcard.key.pem

Don't use an everyday user as admin.

admins: cyrus root

hashimapspool: yes allowanonymouslogin: no allowplaintext: yes

Allow renaming of top-level mailboxes.

allowusermoves: no

Use this if sieve-scripts could be in ~user/.sieve.

sieveusehomedir: yes

Use saslauthd if you want to use pam for imap.

But be warned: login with DIGEST-MD5 or CRAM-MD5

is not possible using pam.

sasl_pwcheck_method: saslauthd

####################################################

This is a recommended authentication method if you

emerge cyrus-sasl with 'postgres' or 'mysql'

To use with mysql database uncomment those lines below.

sasl_pwcheck_method: auxprop

sasl_auxprop_plugin: sql

possible values for sasl_auxprop_plugin 'mysql', 'pgsql', 'sqlite'.

sasl_sql_engine: mysql

all possible values.

sasl_mech_list: CRAM-MD5 DIGEST LOGIN PLAIN

sasl_mech_list: LOGIN PLAIN

or limit to CRAM-MD5 only

sasl_mech_list: CRAM-MD5

change below to suit your setup.

sasl_sql_user: mailsqluser

sasl_sql_passwd: password

sasl_sql_database: mailsqldb

sasl_sql_hostnames: localhost

sasl_sql_select: SELECT clear FROM users WHERE email = '%u@%r'

KEEP THESE IN SYNC WITH cyrus.conf

Unix domain socket that lmtpd listens on.

lmtpsocket: /var/run/cyrus/socket/lmtp

Unix domain socket that idled listens on.

idlesocket: /var/run/cyrus/socket/idle

Unix domain socket that the new mail notification daemon listens on.

notifysocket: /var/run/cyrus/socket/notify

XXX: admin added

syslog_prefix: cyrus proc_path: /run/cyrus/proc mboxname_lockpath: /run/cyrus/lock duplicate_db_path: /run/cyrus/duplicate.db statuscache_db_path: /run/cyrus/statuscache.db -cut-----------------------------------------------------------

brong commented 11 years ago

From: Paweł Kwiatkowski

Ofcourse there is not always need to do a reconstruct. Killing processes helps in most cases.

brong commented 9 years ago

From: Bron Gondwana

Batch moving bugs that won't be in 2.5