dlopen() failed: lib21_fts_flatcurve_plugin.so: undefined symbol

[kingfisher77]

We cannot get rid of this message using compiled dovecot-fts-flatcurve:

Fatal: Couldn't load required plugin /usr/lib/dovecot/modules/lib21_fts_flatcurve_plugin.so: dlopen() failed: /usr/lib/dovecot/modules/lib21_fts_flatcurve_plugin.so: undefined symbol: _ZTIN6icu_668ByteSinkE

We looked at this and this issue.

We compile the module and copy it to Dovecot (/usr/lib/dovecot/modules) which is a dovecot bullseye repo install in a bullseye container.

What is wrong with us? ;-)

Our environment is this:

Dockerfile: .github/actions/dovecot-fts-flatcurve-test/Dockerfile

FROM debian:bullseye-slim
echo "deb https://repo.dovecot.org/ce-2.3-latest/debian/bullseye bullseye main" >> /etc/apt/sources.list.d/dovecot.list

FROM debian:bullseye-slim

ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update && apt-get install -y \
        curl \
        gpg

RUN curl https://repo.dovecot.org/DOVECOT-REPO-GPG | gpg --import && \
        gpg --export ED409DA1 > /etc/apt/trusted.gpg.d/dovecot.gpg && \
        echo "deb https://repo.dovecot.org/ce-2.3-latest/debian/bullseye bullseye main" >> /etc/apt/sources.list.d/dovecot.list

RUN apt-get update && apt-get install -y \
        git \
        automake \
        libtool \
        wget \
        make \
        gettext \
        build-essential \
        bison \
        flex \
        valgrind \
        libssl-dev \
        pkg-config \
        libstemmer-dev \
        libexttextcat-dev \
        libicu-dev \
        libxapian-dev \
        dovecot-imaptest

# We need to build Dovecot ourselves, since "standard" Dovecot does not
# come with necessary ICU libraries built-in
RUN mkdir /dovecot
RUN git clone --depth 1 --branch release-2.3 \
        https://github.com/dovecot/core.git /dovecot/core
RUN cd /dovecot/core && \
        ./autogen.sh && \
        PANDOC=false ./configure --with-stemmer --with-textcat --with-icu && \
        make install

RUN git clone --depth 1 https://github.com/slusarz/dovecot-fts-flatcurve.git \
        /dovecot/fts-flatcurve
RUN cd /dovecot/fts-flatcurve && ./autogen.sh && ./configure --with-dovecot=/usr/local/lib/dovecot && make install

# Users dovecot and dovenull are created by dovecot-imaptest package
RUN useradd vmail && \
    mkdir -p /dovecot/sdbox && \
    chown -R vmail:vmail /dovecot/sdbox

ADD configs/ /dovecot/configs
RUN chown -R vmail:vmail /dovecot/configs/virtual
ADD imaptest/ /dovecot/imaptest

ADD fts-flatcurve-test.sh /fts-flatcurve-test.sh
RUN chmod +x /fts-flatcurve-test.sh
ENTRYPOINT ["/fts-flatcurve-test.sh"]

[kingfisher77]

Dovecot version in Bullseye container ist 2.3.18:

deb http://repo.dovecot.org/ce-2.3.18/debian/bullseye bullseye main

[slusarz]

We compile the module and copy it to Dovecot (/usr/lib/dovecot/modules) which is a dovecot bullseye repo install in a bullseye container.

You have to run the module against a Dovecot core that contains all necessary libraries (stemmer/textcat/icu). The "standard" Dovecot packages do not provide this. (This is why we have to do the expensive build step of compiling Dovecot every time we do github CI - since you can't compile/execute against a generic installable Dovecot package.)

This compilation difficulty goes away in the (forthcoming) CE 2.4, where flatcurve is built-in to Dovecot core.

[kingfisher77]

Thank you for explaining again. We were expecting that copying the plugin file from a same environment would be sufficient. But it is logical - if libraries are not present, it cannot work.

We compiled now dovecot with flatcurve and all features needed by ourselve. Full text search is running like hell! :-) Thank you for your awesome work.

[slusarz]

FWIW, I believe it would be possible (with some degree of unknown efforts) to have the plugin code directly link to the libraries, so you could perform what you originally attempted to do.

The focus of this project was to create an FTS plugin that was eventually going to be incorporated into core Dovecot code, so it made sense to use the existing compilation/linking framework in Dovecot to handle these libraries (and this compilation code tends to be a bit fragile, so best not to touch unless you have to).

However, at this point, the code in this repository is being maintained for use with 2.3 until the next version of CE is coming out. My opinion is that I don't want to make drastic changes at this time since the code is working (mostly) stable for people, albeit with a bit of compilation dependencies necessary to get it to work.

[kingfisher77]

BTW Do you know approximately when version 2.4 will be released?

[slusarz]

There is no date, unfortunately.