undefined symbol: hfile_plugin_init for hfile_irods.so in 1.11

[mcshane]

iRODS support in Dockerfile below works for the version 1.10.2, but not for 1.11.

ARG htsversion=1.10.2
ARG samversion=1.10
ARG bcfversion=1.10.2

Specifically, when loading an irods file with

[D::init_add_plugin] Loaded "knetfile"
[D::init_add_plugin] Loaded "mem"
[D::init_add_plugin] Loaded "crypt4gh-needed"
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_cip.so"
[M::hfile_crypt4gh.init] version 015c142
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_crypt4gh.so"
[M::hfile_gcs.init] version 1.11
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_gcs.so"
[W::load_plugin] can't load plugin "/usr/local/libexec/htslib/hfile_irods.so": /usr/local/libexec/htslib/hfile_irods.so: undefined symbol: hfile_plugin_init
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_libcurl.so"
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_mmap.so"
[M::hfile_s3.init] version 1.11
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_s3.so"
[M::hfile_s3_write.init] version 1.11
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_s3_write.so"
[W::hts_path_itr] can't scan directory "/usr/libexec/htslib": No such file or directory
[E::hts_open_format] Failed to open file "irods:/seq/pacbio/r64125e_20210218_150723/1_A01/demultiplex.bc1016_BAK8B_OA--bc1016_BAK8B_OA.bam" : Protocol not supported

My suspicion is perhaps samtools/htslib@9dfbf8d7 from @daviesrob and then some change now required here in htslib-plugins??

The above logging looks like this with 1.10.2:

[D::init_add_plugin] Loaded "knetfile"
[D::init_add_plugin] Loaded "mem"
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_cip.so"
[M::hfile_crypt4gh.init] version 015c142
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_crypt4gh.so"
[M::hfile_gcs.init] version 1.10.2
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_gcs.so"
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_irods.so"
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_libcurl.so"
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_mmap.so"
[M::hfile_s3.init] version 1.10.2
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_s3.so"
[M::hfile_s3_write.init] version 1.10.2
[D::init_add_plugin] Loaded "/usr/local/libexec/htslib/hfile_s3_write.so"
[W::hts_path_itr] can't scan directory "/usr/libexec/htslib": No such file or directory
[M::hfile_irods.init] version d3ad3cc-dirty built against rods4.2.8(d)

The Dockerfile is a modification of one from @jmarshall 's gist (https://gist.github.com/jmarshall/adc75969306354f4398896f1fc1f4865)

ARG  ubuntu_version=18.04
FROM ubuntu:${ubuntu_version}

USER root

# ubuntu18 as no irods packages available for later versions yet
# gnuplot for plot-bamstats
# python/matplotlib/texlive for plot-vcfstats

RUN apt-get update && DEBIAN_FRONTEND="noninteractive" apt-get install -y \
        locales \
        build-essential \
        curl \
        git \
        libbz2-dev \
        libcurl4-openssl-dev \
        libgsl0-dev \
        liblzma-dev \
        libncurses5-dev \
        libperl-dev \
        libssl-dev \
        zlib1g-dev \
        gnuplot \
        python3-matplotlib \
        python3-numpy \
        texlive-latex-base \
        texlive-fonts-recommended \
        texlive-fonts-extra \
        texlive-latex-extra \
        libsodium-dev \
        autoconf \
        automake \
        lsb-release \
    && curl -L https://packages.irods.org/irods-signing-key.asc | apt-key add - && \
        echo "deb [arch=amd64] https://packages.irods.org/apt/ $(lsb_release -sc) main" | tee /etc/apt/sources.list.d/renci-irods.list && \
        apt-get update && \
        apt-get install -y irods-dev irods-externals-* irods-auth-plugin-krb \
    && sed -i '/^#.* en_GB.UTF-8 /s/^#//' /etc/locale.gen && locale-gen && update-locale LC_ALL="en_GB.UTF-8" \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /tmp

ARG htsversion=1.11
ARG samversion=1.11
ARG bcfversion=1.11
ARG libdeflateversion=1.7

# htslib-plugins hack is issue https://github.com/samtools/htslib-plugins/issues/4
RUN curl -L https://github.com/ebiggers/libdeflate/archive/v${libdeflateversion}.tar.gz | tar xz && \
    (cd libdeflate-${libdeflateversion} && make install) && \
    curl -L https://github.com/samtools/htslib/releases/download/${htsversion}/htslib-${htsversion}.tar.bz2 | tar xj && \
    (cd htslib-${htsversion} && ./configure --with-libdeflate --enable-plugins --with-plugin-path='$(libexecdir)/htslib:/usr/libexec/htslib' && make install) && \
    ldconfig && \
    curl -L https://github.com/samtools/samtools/releases/download/${samversion}/samtools-${samversion}.tar.bz2 | tar xj && \
    (cd samtools-${samversion} && ./configure --with-htslib=system && make install) && \
    curl -L https://github.com/samtools/bcftools/releases/download/${bcfversion}/bcftools-${bcfversion}.tar.bz2 | tar xj && \
    (cd bcftools-${bcfversion} && ./configure --enable-libgsl --enable-perl-filters --with-htslib=system && make install) && \
    git clone --depth 1 git://github.com/samtools/htslib-plugins && \
    (cd htslib-plugins && sed -i 's/init_client_api_table/load_client_api_plugins/g' hfile_irods.c && make install) && \
    git clone --depth 1 git://github.com/samtools/htslib-crypt4gh && \
    (cd htslib-crypt4gh && autoheader && autoconf -Wno-syntax && ./configure --with-htslib=system && cp config.h include/ && make install) && \
    rm -rf /tmp/*

WORKDIR /

[jmarshall]

Alas, this is an HTSlib bug fixed post-1.11 by samtools/htslib#1150. This is also the explanation for wtsi-npg/npg_conda#168; see also this lament.

[mcshane]

Thanks @jmarshall. Thought I was losing my mind :-)

[mcshane]

Modified Dockerfile to build from develop and, indeed, already fixed there.

[jmarshall]

Roll on 1.12 :smile:

As the links on the PR show, I've backported this to pysam (as yet unreleased; 1.12 might still beat it to release!) and bioconda and Debian's htslib packages, so they are unaffected.