search

chrisjackson-pellicle / ParaGone

GNU General Public License v3.0
2 stars 1 forks source link

TAPER errors since 1.1.0 #8

Closed TomHarrop closed 2 months ago

TomHarrop commented 2 months ago

Hi Chris,

I'm getting errors with the latest conda releases. Maybe some more dependencies need to be added to the recipe?

For 1.1.0--py39h9ee0642_1 I get

[INFO]:    Finished generating cleaned alignment for file 5620.aln.trimmed.fasta, 0/8Error raised: local variable 'result' referenced before assignment
traceback is:
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/paragone/align_and_clean.py", line 466, in run_taper
    assert utils.file_exists_and_not_empty(expected_cleaned_alignment_file)
AssertionError

For 1.1.1--py39h9ee0642_0 I get

Error raised: 
TAPER FAILED. Output is: Command 'julia $(which correction_multi.jl) -c 3 -m - -a N 03_alignments_trimmed/4527.aln.trimmed.fasta' returned non-zero exit status 1.
TAPER stdout is: 
TAPER stderr is: ERROR: Unable to load dependent library /usr/local/bin/../lib/julia/libjulia-codegen.so.1.10
Message:libz.so.1: cannot open shared object file: No such file or directory

All working fine with 1.0.0--pl5321h031d066_0

Cheers, Tom

tallnuttrbgv commented 2 months ago

Could be same as https://github.com/chrisjackson-pellicle/ParaGone/issues/9

chrisjackson-pellicle commented 2 months ago

Hi Tom,

The error you're seeing in v1.1.0 was a bug - it's the wrong error message for the problem, which is that TAPER didn't run successfully.

For v1.1.1 - from the look of the path in the error (/usr/local/bin/../lib/julia/libjulia-codegen.so.1.10) you're running ParaGone using a system Julia, rather than the conda package dependency Julia - is that right? If so, can you try again using the conda install of Julia? Sometimes I've found that unexpected $PATH precedence with conda can be resolved by deactivating all conda environements (including base) and then activating the paragone environment directly.

Let me know if there are still issues!

Cheers,

Chris

TomHarrop commented 2 months ago

No, I'm using the biocontainer with containall so I have whatever is in the conda recipe.

It often means the package was tested in an environment where some dependencies were actually on the host machine not in the conda env. It's better to include all the dependencies in the conda recipe so the package can be more portable, which helps reproducibility.

In this case something is looking for libz. I see that isn't in the conda recipe, so that's an easy fix. I don't know what package provides libjulia-codegen.so.

chrisjackson-pellicle commented 2 months ago

Huh, interesting!

On my bioconda paragone install on my Linux machine the file libjulia-codegen.so.1.10 is found at:

/home/cjackson/.conda/envs/paragone_1.1.1/lib/julia/libjulia-codegen.so.1.10

...and libz.so.1 is found at (among other locations):

/home/cjackson/.conda/envs/paragone_1.1.1/lib/libz.so.1

That particular libz.so.1 gets installed as part of the zlib package, which does get installed on my machine during the conda install of paragone:

$conda install paragone

...
The following NEW packages will be INSTALLED:

....
paragone           bioconda/linux-64::paragone-1.1.1-py39h9ee0642_0
...
xz                        5.2.6                h166bdaf_0    conda-forge
zlib                      1.3.1                h4ab18f5_1    conda-forge
zstd                      1.5.6                ha6fb4c9_0    conda-forge

However, running ldd /home/cjackson/.conda/envs/paragone_1.1.1/lib/julia/libjulia-codegen.so.1.10 returns:

...
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5e3c992000)
...

So, as you say, /home/cjackson/.conda/envs/paragone_1.1.1/lib/julia/libjulia-codegen.so.1.10 is expecting EDIT: has found libz.so.1 at /lib/x86_64-linux-gnu, rather than from somewhere within the conda environment. I've checked that this is also true for a fresh install of conda Julia within a new, clean conda environment (which also installs the zlib package). EDIT So it looks like happens with both a local build and on the bioconda build system.

Given the above, I'm not sure that adding zlib as an explicit dependency in the paragone conda recipe will solve the issue. The problem persists when doing so for a local build (https://anaconda.org/chrisjackson-pellicle/paragone_test_zlib/files).

Could be an issue withe the conda julia package, unless I'm missing something obvious?

Cheers,

Chris

chrisjackson-pellicle commented 2 months ago

Hmm, I'm looking within the Singularity BioContainer for paragone 1.1.1, and libz.so.1 is present at /usr/local/lib, but ldd /usr/local/lib/julia/libjulia-codegen.so.1.10 returns:

linux-vdso.so.1 (0x00007ffead3fd000)
libunwind.so.8 => /usr/local/lib/julia/../libunwind.so.8 (0x00007fcde84a6000)
libLLVM-15jl.so => /usr/local/lib/julia/libLLVM-15jl.so (0x00007fcde3200000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fcde84a0000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fcde849b000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fcde8496000)
libatomic.so.1 => /usr/local/lib/julia/libatomic.so.1 (0x00007fcde848a000)
libjulia.so.1.10 => /usr/local/lib/julia/../libjulia.so.1.10 (0x00007fcde8465000)
libjulia-internal.so.1.10 => /usr/local/lib/julia/libjulia-internal.so.1.10 (0x00007fcde2d60000)
libstdc++.so.6 => /usr/local/lib/julia/libstdc++.so.6 (0x00007fcde2aac000)
libgcc_s.so.1 => /usr/local/lib/julia/libgcc_s.so.1 (0x00007fcde8440000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fcde28cb000)
/lib64/ld-linux-x86-64.so.2 (0x00007fcde867e000)
libz.so.1 => not found
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fcde27ec000)
libutf8proc.so.2 => /usr/local/lib/julia/../libutf8proc.so.2 (0x00007fcde83e6000)

So, it's not that it's got the wrong path to libz.so.1, it's just not found.

chrisjackson-pellicle commented 2 months ago

Incidentally, this doesn't appear to be an issue with a conda install of Julia on macOS. The equivalent library on my mac (/Users/chrisjackson/miniforge3/envs/pg_1.1.1_test/lib/julia/libjulia-codegen.1.10.dylib) has the correct path in @rpath:

$otool -l /Users/chrisjackson/miniforge3/envs/pg_1.1.1_test/lib/julia/libjulia-codegen.1.10.dylib | grep RPATH -A2
          cmd LC_RPATH
      cmdsize 32
         path @loader_path/ (offset 12)
--
          cmd LC_RPATH
      cmdsize 272
         path /Users/chrisjackson/miniforge3/envs/pg_1.1.1_test/lib (offset 12)
--
          cmd LC_RPATH
      cmdsize 32
         path @loader_path/../ (offset 12)

...and: 

$ls /Users/chrisjackson/miniforge3/envs/pg_1.1.1_test/lib | grep libz
libz.1.3.1.dylib
libz.1.dylib
libz.a
libz.dylib
libzstd.1.5.6.dylib
libzstd.1.dylib
libzstd.dylib
chrisjackson-pellicle commented 2 months ago

Further investigations:

Looking further at /usr/local/lib/julia/libjulia-codegen.so.1.10 within the BioContainers SIngularity container:

Singularity> /usr/local/x86_64-conda-linux-gnu/bin/readelf -d /usr/local/lib/julia/libjulia-codegen.so.1.10

Dynamic section at offset 0x1b6188 contains 38 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libunwind.so.8]
 0x0000000000000001 (NEEDED)             Shared library: [libLLVM-15jl.so]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libatomic.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libjulia.so.1.10]
 0x0000000000000001 (NEEDED)             Shared library: [libjulia-internal.so.1.10]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [libjulia-codegen.so.1.10]
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN:$ORIGIN/..]
...

...it looks like the library should actually be able to search for libz.so.1 in the correct directory ($ORIGIN/.., so /usr/local/lib.

However, I'm wondering if we're running in to the issue described here.

The output of readelf -d usr/local/lib/julia/libjulia-codegen.so.1.10 shows that libz.so.1 isn't a direct dependency. Using lddtree on my Linux (it doesn't appear to be present in the Biocontainer image) shows that libz.so.1 is actually a dependency of libLLVM-15jl.so (the latter being a direct dependency of libjulia-codegen.so.1.10):

(paragone_test_zlib) cjackson@volvox_08:59:38_~/julia_env_test$lddtree /home/cjackson/.conda/envs/paragone_test_zlib/lib/julia/libjulia-codegen.so.1.10
libjulia-codegen.so.1.10 => /home/cjackson/.conda/envs/paragone_test_zlib/lib/julia/libjulia-codegen.so.1.10 (interpreter => none)
    libunwind.so.8 => /usr/lib/x86_64-linux-gnu/libunwind.so.8
        ld-linux-x86-64.so.2 => /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5
    libLLVM-15jl.so => /home/cjackson/.conda/envs/paragone_test_zlib/lib/julia/libLLVM-15jl.so
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
    libatomic.so.1 => /home/cjackson/.conda/envs/paragone_test_zlib/lib/julia/libatomic.so.1
    libjulia.so.1.10 => not found
    libjulia-internal.so.1.10 => /home/cjackson/.conda/envs/paragone_test_zlib/lib/julia/libjulia-internal.so.1.10
        libutf8proc.so.2 => not found
    libstdc++.so.6 => /home/cjackson/.conda/envs/paragone_test_zlib/lib/julia/libstdc++.so.6
    libgcc_s.so.1 => /home/cjackson/.conda/envs/paragone_test_zlib/lib/julia/libgcc_s.so.1
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6

..and, libLLVM-15jl.so doesn't have $ORIGIN/.. in its runpath - just $ORIGIN:

Singularity> /usr/local/x86_64-conda-linux-gnu/bin/readelf -d /usr/local/lib/julia/libLLVM-15jl.so

Dynamic section at offset 0x4ed7548 contains 35 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
 0x000000000000000e (SONAME)             Library soname: [libLLVM-15jl.so]
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN]

Hmmm. If this is true, I'm not sure what to do about it.

chrisjackson-pellicle commented 2 months ago

...again, it's not a problem on macOS:

otool -l /Users/chrisjackson/miniforge3/envs/pg_1.1.1_test/lib/julia/libLLVM.dylib | grep -A2 RPATH
          cmd LC_RPATH
      cmdsize 32
         path @loader_path/../lib (offset 12)

...as here libLLVM.dylib will search @loader_path/../lib (/Users/chrisjackson/miniforge3/envs/pg_1.1.1_test/lib)

chrisjackson-pellicle commented 2 months ago

In the BioContainers Singularity container, running export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib resolves the issue (i.e. I can then successfully launch Julia within the container), but I'm guessing this isn't a viable solution given the auto-generation of the BioContainers Docker container from the bioconda recipe. Unless there's a way to feed in to that process?

chrisjackson-pellicle commented 2 months ago

Ok, conda Julia v1.8.5 does not have this problem. From a paragone v1.1.1 package built specifying julia =1.8:

$ldd /home/cjackson/.conda/envs/paragone_1.1.1/lib/julia/libjulia-codegen.so.1.8
linux-vdso.so.1 (0x00007fffe67a1000)
libunwind.so.8 => /home/cjackson/.conda/envs/paragone_1.1.1/lib/julia/../libunwind.so.8 (0x00007f165ad8b000)
libLLVM-13jl.so => /home/cjackson/.conda/envs/paragone_1.1.1/lib/julia/libLLVM-13jl.so (0x00007f1656449000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f165642f000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f1656425000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1656404000)
libatomic.so.1 => /home/cjackson/.conda/envs/paragone_1.1.1/lib/julia/libatomic.so.1 (0x00007f16563f9000)
libjulia.so.1 => /home/cjackson/.conda/envs/paragone_1.1.1/lib/julia/../libjulia.so.1 (0x00007f16563d5000)
libjulia-internal.so.1 => /home/cjackson/.conda/envs/paragone_1.1.1/lib/julia/libjulia-internal.so.1 (0x00007f1655f4f000)
libstdc++.so.6 => /home/cjackson/.conda/envs/paragone_1.1.1/lib/julia/libstdc++.so.6 (0x00007f1655c8f000)
libgcc_s.so.1 => /home/cjackson/.conda/envs/paragone_1.1.1/lib/julia/libgcc_s.so.1 (0x00007f1655c6c000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1655a82000)
/lib64/ld-linux-x86-64.so.2 (0x00007f165aee0000)
libz.so.1 => /home/cjackson/.conda/envs/paragone_1.1.1/lib/julia/../libz.so.1 (0x00007f1655a67000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f16558d8000)
libutf8proc.so.2 => /home/cjackson/.conda/envs/paragone_1.1.1/lib/julia/../libutf8proc.so.2 (0x00007f1655880000)

So, libz.so.1 is now found at the correct path within the conda env (as libLLVM-13jl.so now has $ORIGIN/.. in its Library rpath).

I'll experiment a bit to see which of the available versions of Julia play nicely, push and release v1.1.2 of paragone (there's a bugfix for an issue when running paragone full_pipeline), and then update the bioconda recipe to restrict the julia depedency to working versions. Hopefully that solves the BioContainter issue!

Cheers,

Chris

chrisjackson-pellicle commented 2 months ago

A final (ha) note:

For julia v1.10.4 downloaded directly (https://julialang.org/downloads/#current_stable_release - Generic Linux on x86, glibc binaries), lib/julia/libLLVM-15jl.so also only has $ORIGIN for the Library runpath, but libz.so.1 is present in lib/julia/, so it's not an issue.

chrisjackson-pellicle commented 2 months ago

Hi @TomHarrop - I've updated the Bioconda recipe and made a release for ParaGone v1.1.2. I've just tested the Singularity BioContainer for v1.1.2 and it seems to be running correctly.

Let me know if there are any further issues!

Cheers,

Chris

TomHarrop commented 2 months ago

Thanks @chrisjackson-pellicle! My module passed testing with the new container and I have some jobs in the queue to run it on a full dataset .