search

cyrusimap / cyrus-imapd

Cyrus IMAP is an email, contacts and calendar server
http://cyrusimap.org
Other
543 stars 149 forks source link

LIST command hides mailboxes in certain cases, children attribute is wrong #3793

Open aminski316 opened 2 years ago

aminski316 commented 2 years ago

I've done some tests using Cyrus IMAP 3.2.6-Debian-3.2.6-2~bpo10+1 with enabled virt- and crossdomain support, altnamespace, unixhierarchysep and improved_mboxlist_sort.

I've done the tests running two identical installations, the first one without defaultdomain configured, the second one with defaultdomain default.com configured.

Two things are weird. LIST command hides mailboxes in certain cases and folders with domain part have always no children.

I've added line numbers:

1  . create user/user.one
2  . OK [MAILBOXID (ih9cx8siiy19flip9fhk7j6f)] Completed
3  . create user/user.one/one
4  . OK [MAILBOXID (rc97nsp4h2ipa08o1x4mg9bb)] Completed
5  . create user/user.one@domain.com
6  . OK [MAILBOXID (wya2ura60ha87uppcvfc33tg)] Completed
7  . list "" "user/*"
8  * LIST (\HasNoChildren) "/" user/user.one@domain.com
9  * LIST (\HasChildren) "/" user/user.one
10 * LIST (\HasNoChildren) "/" user/user.one/one
11 . OK Completed (0.000 secs 3 calls)

That looks pretty good, created three folders, three folders listed, one has a child (line 9).

12 . create user/user.one/one@domain.com
13 . OK [MAILBOXID (7o4j2umwlc319t9wnel09gcs)] Completed
14 . list "" "user/*"
15 * LIST (\HasNoChildren) "/" user/user.one@domain.com
16 * LIST (\HasNoChildren) "/" user/user.one/one@domain.com
17 * LIST (\HasNoChildren) "/" user/user.one/one
18 . OK Completed (0.000 secs 4 calls)

Created one child folder, four folders created but three listed. One listed folder has a child but is listed with no children attribute (line 15).

19 . create user/user.two@domain.com
20 . OK [MAILBOXID (m9nilk5p4p8yb4z3ngcgp94k)] Completed
21 . list "" "user/*"
22 * LIST (\HasNoChildren) "/" user/user.one@domain.com
23 * LIST (\HasNoChildren) "/" user/user.one/one@domain.com
24 * LIST (\HasNoChildren) "/" user/user.two@domain.com
25 * LIST (\HasChildren) "/" user/user.one
26 * LIST (\HasNoChildren) "/" user/user.one/one
27 . OK Completed (0.000 secs 5 calls)

Created a second parent folder for domain.com, five folders created, five folders listed, missing folder is back and has a child (line 25). The folder for user.one@domain.com is listed with no children attribute (line 22), which is wrong.

Both test servers behave equal.

I've changed the test, using full qualified folder names, one is the defaultdomain default.com.

This is the output from the second server, which has defaultdomain configured:

1  . create user/user.one@default.com
2  . OK [MAILBOXID (cy26z9y9k07llbyy14zkwgsa)] Completed
3  . create user/user.one/one@default.com
4  . OK [MAILBOXID (a4af5crjeej4csosf9e6os8s)] Completed
5  . create user/user.one@domain.com
6  . OK [MAILBOXID (0ehqz764zxntr6x3f1cur61v)] Completed
7  . list "" "user/*"
8  * LIST (\HasNoChildren) "/" user/user.one@domain.com
9  * LIST (\HasChildren) "/" user/user.one
10 * LIST (\HasNoChildren) "/" user/user.one/one
11 . OK Completed (0.000 secs 3 calls)

That looks ok.

12 . create user/user.one/one@domain.com
13 . OK [MAILBOXID (pn5xr9dl63kt9osncleyzydg)] Completed
14 . list "" "user/*"
15 * LIST (\HasNoChildren) "/" user/user.one@domain.com
16 * LIST (\HasNoChildren) "/" user/user.one/one@domain.com
17 * LIST (\HasNoChildren) "/" user/user.one/one
18 . OK Completed (0.000 secs 4 calls)

One folder is missing.

19 . create user/user.two@domain.com
20 . OK [MAILBOXID (o7g52uu5frc4kyanyjdnbgfd)] Completed
21 . list "" "user/*"
22 * LIST (\HasNoChildren) "/" user/user.one@domain.com
23 * LIST (\HasNoChildren) "/" user/user.one/one@domain.com
24 * LIST (\HasNoChildren) "/" user/user.two@domain.com
25 * LIST (\HasChildren) "/" user/user.one
26 * LIST (\HasNoChildren) "/" user/user.one/one
27 . OK Completed (0.000 secs 5 calls)

Missing folder is back, children attribute is wrong for domain.com (line 22).

The output from the first server, without defaultdomain configured, looks ok except children attribute.

14 . list "" "user/*"
15 * LIST (\HasNoChildren) "/" user/user.one@default.com
16 * LIST (\HasNoChildren) "/" user/user.one/one@default.com
17 * LIST (\HasNoChildren) "/" user/user.one@domain.com
18 * LIST (\HasNoChildren) "/" user/user.one/one@domain.com
19 . OK Completed (0.000 secs 4 calls)

Third test using full qualified folder names and domains other than default.com:

14 . list "" "user/*"
15 * LIST (\HasNoChildren) "/" user/user.one@domain.com
16 * LIST (\HasNoChildren) "/" user/user.one/one@domain.com
17 * LIST (\HasNoChildren) "/" user/user.one@otherdomain.com
18 * LIST (\HasNoChildren) "/" user/user.one/one@otherdomain.com
19 . OK Completed (0.000 secs 4 calls)

Both test servers behave equal. All folders are listed, children attributes are wrong (lines 15,17).

I've done the tests using Cyrus IMAP 3.4.2, same results.

configdirectory: /volumes/imap-data/var/lib/cyrus
proc_path: /dev/shm/cyrus-imapd/proc
mboxname_lockpath: /dev/shm/cyrus-imapd/lock
duplicate_db_path: /dev/shm/cyrus-imapd/deliver.db
statuscache_db_path: /dev/shm/cyrus-imapd/statuscache.db
tls_sessions_db_path: /dev/shm/cyrus-imapd/tls_sessions.db
defaultpartition: default
partition-default: /volumes/imap-data/var/spool/cyrus/mail
partition-news: /volumes/imap-data/var/spool/cyrus/news
newsspool: /volumes/imap-data/var/spool/news
altnamespace: yes
unixhierarchysep: yes
reject8bit: yes
munge8bit: yes
lmtp_downcase_rcpt: yes
admins: cyrus
allowanonymouslogin: no
autocreate_quota: 0
autocreate_post: 0
autocreate_inbox_folders: Drafts|Sent|Junk|Trash|Archives|Templates
autocreate_subscribe_folders: Drafts|Sent|Junk|Trash|Archives|Templates
umask: 077
sendmail: /usr/sbin/sendmail
sieveusehomedir: false
sievedir: /volumes/imap-data/var/spool/sieve
hashimapspool: true
allowplaintext: yes
sasl_mech_list: PLAIN LOGIN
virtdomains: userid

# server imap-test1
# defaultdomain: default.com
# server imap-test2
defaultdomain: default.com

sasl_pwcheck_method: auxprop saslauthd
sasl_auxprop_plugin: sasldb
sasl_auto_transition: no
tls_server_cert: /etc/ssl/cyrus-imapd/imap.xx.xx.pem
tls_server_key: /etc/ssl/cyrus-imapd/imap.xx.xx.key
tls_client_ca_file: /etc/ssl/cyrus-imapd/imap.xx.xx.ca.pem
tls_client_ca_dir: /etc/ssl/certs
tls_session_timeout: 1440
tls_require_cert: false
lmtpsocket: /run/cyrus/socket/lmtp
idlesocket: /run/cyrus/socket/idle
notifysocket: /run/cyrus/socket/notify
syslog_prefix: cyrus
allowusermoves: 0
allowdeleted: 0
defaultacl: cyrus lrswipkxtecdan anonymous p anyone p
duplicatesuppression: 1
autocreate_sieve_script: /volumes/imap-data/var/lib/sieve/scripts/default/egr-filter.script
disconnect_on_vanished_mailbox: 1
anyoneuseracl: 0
improved_mboxlist_sort: 1
crossdomains: 1
crossdomains_onlyother: 0
disable_shared_namespace: 0
elliefm commented 2 years ago

It looks like you've done these tests in a single session, as an admin user, correct?

What happens if you CREATE these mailboxes as the admin user (and SETACL them properly), but then run the LIST commands as the users who own the mailboxes?

I ask, because inside LIST's implementation there is some dependency on whether or not we're in the admin namespace or a user's namespace. So whether the problem presents the same in both, or differently, will be important in tracking it down.

There are some limitations on IMAP behaviour specific to the admin namespace, though I don't remember exactly what they are offhand. You might just be running into something like this, and it will be fine in normal use by real users. This comment for example -- I'm not sure of its ramifications offhand, but it seems plausible that it may affect the haschildren/hasnochildren computations later:

https://github.com/cyrusimap/cyrus-imapd/blob/588078f43f3fd3d060a8b3f6adf724212334a652/imap/mboxlist.c#L4042-L4043

We usually recommend that "defaultdomain" be set to something that will never be a real domain (and its default value is "internal" for this reason), and that all users are created fully qualified. I don't remember why though!

aminski316 commented 2 years ago

It looks like you've done these tests in a single session, as an admin user, correct?

That's true.

What happens if you CREATE these mailboxes as the admin user (and SETACL them properly), but then run the LIST commands as the users who own the mailboxes?

I ask, because inside LIST's implementation there is some dependency on whether or not we're in the admin namespace or a user's namespace. So whether the problem presents the same in both, or differently, will be important in tracking it down.

You're right, output is ok in user context.

. login user.one pw
. OK
. list "" "*"
* LIST (\HasNoChildren) "/" INBOX
* LIST (\HasNoChildren) "/" one
. OK Completed (0.000 secs 2 calls)

. login user.one@domain.com pw
. OK
. list "" "*"
* LIST (\HasNoChildren) "/" INBOX
* LIST (\HasNoChildren) "/" one
. OK Completed (0.000 secs 2 calls)

There are some limitations on IMAP behaviour specific to the admin namespace, though I don't remember exactly what they are offhand. You might just be running into something like this, and it will be fine in normal use by real users.

Well, looks fine in user context but not in admin context.

. login cyrus pw
. list "" "*"
* LIST (\HasNoChildren) "/" user/user.one@domain.com
* LIST (\HasNoChildren) "/" user/user.one/one@domain.com
* LIST (\HasNoChildren) "/" user/user.one/two@domain.com
* LIST (\HasNoChildren) "/" user/user.one/two/three@domain.com
* LIST (\HasNoChildren) "/" user/user.one/one
. OK Completed (0.000 secs 6 calls)
. login user.one@domain.com pw
. list "" "*"
* LIST (\HasNoChildren) "/" INBOX
* LIST (\HasNoChildren) "/" one
* LIST (\HasChildren) "/" two
* LIST (\HasNoChildren) "/" two/three
. OK Completed (0.001 secs 4 calls)

This comment for example -- I'm not sure of its ramifications offhand, but it seems plausible that it may affect the haschildren/hasnochildren computations later:

https://github.com/cyrusimap/cyrus-imapd/blob/588078f43f3fd3d060a8b3f6adf724212334a652/imap/mboxlist.c#L4042-L4043

We usually recommend that "defaultdomain" be set to something that will never be a real domain (and its default value is "internal" for this reason), and that all users are created fully qualified. I don't remember why though!

I suppose not setting defaultdomain forces using of default value "internal". Can you confirm that?

elliefm commented 2 years ago

I suppose not setting defaultdomain forces using of default value "internal". Can you confirm that?

Yep, exactly.

Generally, for any given imapd.conf option, if you don't set it yourself, it will use its default value. The default value for defaultdomain is "internal". I appreciate this specific case is a little confusing, since there's two different uses of the word "default" happening.