imapd segmentation fault when querying mailboxes

[tacerus]

Hi,

this issue occurs on 3.8.2, which I am packaging for openSUSE, however I was able to reproduce it with 3.4.x and 3.2.12 as well.

$ cyradm --auth login -u cyrus localhost
IMAP Password:
localhost> lm

# server output
Apr 10 22:30:11 phoebe cyrus/imap[7041]: starttls: TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits new) no authentication
Apr 10 22:30:12 phoebe cyrus/imap[7041]: login: localhost [::1] cyrus plaintext+TLS User logged in SESSIONID=<cyrus-1712781011-7041-1-16161939197667482871>
Apr 10 22:30:13 phoebe cyrus/imap[7041]: Falling back to using legacy location for cyrus.sub
Apr 10 22:30:13 phoebe kernel: imapd[7041]: segfault at 0 ip 00007f6bf54a673e sp 00007fffd65cab50 error 4 in libc-2.31.so[7f6bf5409000+1e8000]
Apr 10 22:30:13 phoebe kernel: Code: 84 5c ff ff ff 0f 1f 80 00 00 00 00 48 8d 0c 68 0f b7 11 66 85 d2 0f 84 45 ff ff ff 48 8d 04 e8 83 ea 01 4c 8b 80 80 00 00 00 <49> 8b 30 48 89 b0 80 00 00 00 66 89 11 4c 89 c0 49 c7 40 08 00 00
Apr 10 22:30:13 phoebe cyrus/master[2575]: process type:SERVICE name:imap path:/usr/lib/cyrus/imapd age:2.148s pid:7041 signaled to death by signal 11 (Segmentation fault, core dumped)

I am generally building with:

-O2 -g -m64 -fmessage-length=0 -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -fno-strict-aliasing -fPIC

However I also tried the CFLAGS recommended in the upstream guide (-W -Wno-unused-parameter -g -O0 -Wall -Werror -Wextra -fPIC) and collected the backtrace:

(gdb) backtrace
#0  tcache_get (tc_idx=<optimized out>) at malloc.c:2937
#1  __GI___libc_malloc (bytes=12) at malloc.c:3051
#2  0x00007f83e3edd71a in re_node_set_alloc (size=<optimized out>, set=0x7ffd20b94090) at regex_internal.c:972
#3  calc_eclosure_iter (new_set=0x7ffd20b94160, dfa=0x558e38deb5f0, node=3, root=true) at regcomp.c:1700
#4  0x00007f83e3ee4a7d in calc_eclosure (dfa=<optimized out>) at regcomp.c:1677
#5  analyze (preg=0x558e38de8d60) at regcomp.c:1204
#6  re_compile_internal (preg=0x558e38de8d60, pattern=0x558e38deb570 "(^.*)([/]|$)", length=<optimized out>, syntax=16843462) at regcomp.c:795
#7  0x00007f83e3eea2f9 in __GI___regcomp (preg=0x558e38de8d60, pattern=0x558e38deb570 "(^.*)([/]|$)", cflags=<optimized out>) at regcomp.c:491
#8  0x00007f83e4731aed in glob_init (str=0x558e38dec931 "", sep=47 '/') at lib/glob.c:114
#9  0x00007f83e48a15e6 in mboxlist_do_find (rock=0x7ffd20b953c0, patterns=0x7ffd20b955f8) at imap/mboxlist.c:4033
#10 0x00007f83e48a24d5 in mboxlist_findallmulti_withp (namespace=0x558e389244a0 <imapd_namespace>, patterns=0x7ffd20b955f8, isadmin=1, userid=0x558e38dea8c0 "cyrus",
    auth_state=0x558e38dd1d20, p=0x0, cb=0x558e388f408a <list_cb>, rock=0x7ffd20b954d0) at imap/mboxlist.c:4313
#11 0x00007f83e48a23b8 in mboxlist_findallmulti (namespace=0x558e389244a0 <imapd_namespace>, patterns=0x7ffd20b955f8, isadmin=1, userid=0x558e38dea8c0 "cyrus", auth_state=0x558e38dd1d20,
    proc=0x558e388f408a <list_cb>, rock=0x7ffd20b954d0) at imap/mboxlist.c:4281
#12 0x0000558e388f50a6 in list_data (listargs=0x7ffd20b955e0) at imap/imapd.c:13942
#13 0x0000558e388e6ec9 in cmd_list (tag=0x558e38dab020 "7", listargs=0x7ffd20b955e0) at imap/imapd.c:8504
--Type <RET> for more, q to quit, c to continue without paging--
#14 0x0000558e388d302d in cmdloop () at imap/imapd.c:1694
#15 0x0000558e388d0dd8 in service_main (argc=1, argv=0x558e38d9f960, envp=0x7ffd20b982b8) at imap/imapd.c:1009
#16 0x0000558e3890ea51 in main (argc=1, argv=0x7ffd20b982a8, envp=0x7ffd20b982b8) at master/service.c:647

Would appreciate any ideas! Georg

[elliefm]

Can you provide the configure line that was used for the build? You can usually get it from head -10 config.log in the build directory. Please feel free to censor any sensitive details if necessary.

Here's mine, for example:

ellie@debian:master:~/fastmail/cyrus-imapd$head -10 config.log 
This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.

It was created by cyrus-imapd configure 3.11.0-alpha0-337-g1bca21baf, which was
generated by GNU Autoconf 2.71.  Invocation command line was

  $ ./configure PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/home/ellie/bin:/home/ellie/fastmail/cyrus-build-tools:/usr/local/cyruslibs/bin 'CFLAGS=-g -O0 -Wall -Wextra -Werror -Wl,--as-needed -fstack-protector-all ' 'CXXFLAGS=-g -O0 -Wall -Wextra -Werror -fstack-protector-all ' --prefix=/dev/shm/cyrus/main --enable-http --enable-calalarmd --enable-unit-tests --enable-replication --with-openssl=yes --enable-nntp --enable-murder --enable-idled --enable-event-notification --enable-sieve --enable-autocreate --enable-silent-rules --enable-backup --with-lmdb --enable-xapian --enable-jmap --with-ldap

## --------- ##
## Platform. ##

If you're able to, can you also capture the "Cyrus Server configured components" information that's printed to stdout by configure? It's not logged anywhere, so to get it you'll need to run configure again, and capture it from there. It'll be the last ~80 or so lines.

I'm particularly interested in seeing what regex library you're configured to use, because it smells like that's where the issue is coming from. We mostly only test with PCRE/PCRE2, so if you're using something else, Cyrus might be doing something that's incompatible with it that we haven't noticed.

[tacerus]

Hi @elliefm, thanks for getting back, and sure!

  $ ./configure --host=x86_64-suse-linux-gnu --build=x86_64-suse-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/lib --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --disable-dependency-tracking --localstatedir=/var/lib --enable-autocreate --enable-idled --enable-murder --enable-nntp --enable-replication --enable-gssapi --enable-unit-tests --enable-xapian --with-com_err --with-cyrus-user=cyrus --prefix=/usr --libexecdir=/usr/lib/cyrus --with-gnu-ld --with-gss_impl=auto --with-ldap --with-libwrap --with-lock=fcntl --with-openssl --with-perl=perl --with-sasl --with-syslogfacility=DAEMON --with-zlib

[   42s] Cyrus Server configured components
[   42s]
[   42s]    gssapi:             yes
[   42s]    autocreate:         yes
[   42s]    idled:              yes
[   42s]    httpd:              no
[   42s]    kerberos V4:        no
[   42s]    murder:             yes
[   42s]    nntpd:              yes
[   42s]    replication:        yes
[   42s]    sieve:              yes
[   42s]    srs:                no
[   42s]    calalarmd:          no
[   42s]    jmap:               no
[   42s]    objectstore:        no
[   42s]    backup:             no
[   42s]    com_err:            yes
[   42s]
[   42s] External dependencies:
[   42s]    ldap:               yes
[   42s]    openssl:            yes
[   42s]    zlib:               yes
[   42s]    jansson:            yes
[   42s]    pcre:               yes
[   42s]    pcre2:              no
[   42s]    clamav:             yes
[   42s]    -----------------------
[   42s]    caringo:            no
[   42s]    openio:             no
[   42s]    -----------------------
[   42s]    nghttp2:            no (httpd is not built)
[   42s]    wslay:              no (httpd is not built)
[   42s]    brotli:             no (httpd is not built)
[   42s]    zstd:               no (httpd is not built)
[   42s]    xml2:               no (httpd is not built)
[   42s]    ical:               no (httpd is not built)
[   42s]    shapelib:           no (httpd is not built)
[   42s]    icu4c:              yes
[   42s]    chardet:            no
[   42s]    cld2:               no
[   42s]    guesstz:            no
[   42s]
[   42s] Database support:
[   42s]    mysql:              no
[   42s]    postgresql:         no
[   42s]    sqlite:             yes
[   42s]    zeroskip:           no
[   42s]
[   42s] Search engine:
[   42s]    squat:              yes
[   42s]    xapian:             yes
[   42s]    xapian_cjk_tokens:  ngrams
[   42s]
[   42s] Documentation dependencies:
[   42s]    sphinx-build:
[   42s]    Pod::POM::View::Restructured:  no
[   42s]    GitPython:          no
[   42s]
[   42s] Installation directories:
[   42s]    prefix:             /usr
[   42s]    sysconfdir:         /etc
[   42s]
[   42s] Build info:
[   42s]    shared:             yes
[   42s]    static:             no
[   42s]    cflags:             -W -Wno-unused-parameter -g -O0 -Wall -Wextra -Werror -fPIC
[   42s]    cxxflags:           -fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-prote
ction -g
[   42s]    libs:                -lgssapi_krb5 -lkrb5 -lk5crypto -lcom_err -lkrb5support    -lpcreposix -lpcre -lz  -lsqlite3
[   42s]    ldflags:
[   42s]    libm:               -lm
[   42s]    unit tests (cunit): yes
[   42s]

Given your hint, I now also tested building against pcre2 - same configure line and output as above, except for

[   31s]    pcre:               no
[   31s]    pcre2:              yes

However, that unfortunately does not let me use cyradm, hence I cannot test the issue at hand:

cyradm --auth login -u cyrus localhost
Can't load '/usr/lib/perl5/vendor_perl/5.26.1/x86_64-linux-thread-multi/auto/Cyrus/IMAP/IMAP.so' for module Cyrus::IMAP: /usr/lib/perl5/vendor_perl/5.26.1/x86_64-linux-thread-multi/auto/Cyrus/IMAP/IMAP.so: undefined symbol: pcre2_regexec at /usr/lib/perl5/5.26.1/x86_64-linux-thread-multi/DynaLoader.pm line 193.
 at /usr/lib/perl5/vendor_perl/5.26.1/x86_64-linux-thread-multi/Cyrus/IMAP/Admin.pm line 43.
Compilation failed in require at /usr/lib/perl5/vendor_perl/5.26.1/x86_64-linux-thread-multi/Cyrus/IMAP/Admin.pm line 43.
BEGIN failed--compilation aborted at /usr/lib/perl5/vendor_perl/5.26.1/x86_64-linux-thread-multi/Cyrus/IMAP/Admin.pm line 43.
Compilation failed in require at /usr/lib/perl5/vendor_perl/5.26.1/x86_64-linux-thread-multi/Cyrus/IMAP/Shell.pm line 59.
BEGIN failed--compilation aborted at /usr/lib/perl5/vendor_perl/5.26.1/x86_64-linux-thread-multi/Cyrus/IMAP/Shell.pm line 59.
Compilation failed in require at /usr/bin/cyradm line 66.
BEGIN failed--compilation aborted at /usr/bin/cyradm line 66.

I should probably rather investigate that separately.

[tacerus]

Now I built the Cyrus server with pcre2, and perl-Cyrus-IMAP/cyradm with pcre!

With this constellation, I can use cyradm without any segfaults on the server side.

So I think the issue in this report is only concerning legacy pcre.

Edit: upon applying a patch from Fedora to link the Perl libraries correctly, I can confirm that the full stack (server and client components) works correctly with pcre2.

[elliefm]

What's the Fedora patch? We might be able to upstream it

I think your original crash might have just been #2629 saying hello again. The old PCRE library claimed ABI compatibility with libc, but had a different size regex_t. When an application was linked against both glibc and libpcre, it would end up calling the wrong regcomp/etc with the wrong regex_t, and crash like this. The only way to make it work is to use a PCRE built with Debian's patches (which rename the incompatible functions so they don't get in each others' way anymore). If the PCRE you were using did not have Debian's patches, then that's what the problem was.

I built the Cyrus server with pcre2, and perl-Cyrus-IMAP/cyradm with pcre!

Yeahhhh don't do that, you'll just bump into the same crash again, somewhere else 😅

The current PCRE2 doesn't have the same problem as PCRE did, so none of this matters if you can build with PCRE2.

[tacerus]

What's the Fedora patch? We might be able to upstream it

I wanted to make a separate issue for it, the patch is this:

https://src.fedoraproject.org/rpms/cyrus-imapd/blob/rawhide/f/patch-cyrus-perl-linking

But it does not look quite right to me, because the server components do get correctly linked automatically, so I suppose the client ones should as well? And it requires the use of --disable-pcre to work albeit the configure script reporting "No" for legacy pcre without it.

I think your original crash might have just been https://github.com/cyrusimap/cyrus-imapd/issues/2629 saying hello again.

Interesting, I did actually find that issue, but didn't pay more attention to it .. the tests all pass in my environment. Thanks for elaborating on this!

We have some patches, but none renaming functions:

https://build.opensuse.org/package/show/openSUSE:Leap:15.5:Update/pcre

Then yes, in my case it seems "resolved" with with the use of prce2. But maybe something could be done for it to fail at build or test time?