search

jodyphelan / TBProfiler

Profiling tool for Mycobacterium tuberculosis to detect ressistance and strain type from WGS data
GNU General Public License v3.0
102 stars 42 forks source link

ERROR MESSAGE #371

Open tuelomogashoa opened 1 week ago

tuelomogashoa commented 1 week ago

BTB001_S140.errlog.txt

Please help resolve this error. I have attached one f the errlog.txt files

jodyphelan commented 1 week ago

Did you install with conda? Can you try run the command delly on your commandline and paste the output here?

tuelomogashoa commented 1 week ago

Hi,

I am not sure I understand how to run the command delly, please clarify

Tuelo

On 24 Jun 2024, at 10:12, Jody Phelan @.***> wrote:

Can you try run the command delly on your commandline and paste the output here?

— Reply to this email directly, view it on GitHub https://github.com/jodyphelan/TBProfiler/issues/371#issuecomment-2185881852, or unsubscribe https://github.com/notifications/unsubscribe-auth/A4LCKKCLUCQWGGZ5F5EJ5ZDZI7IHJAVCNFSM6AAAAABJZJY526VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBVHA4DCOBVGI. You are receiving this because you authored the thread.

Sent from Tuelo’s MacBook Pro @.***

tuelomogashoa commented 1 week ago

Did you install with conda? Can you try run the command delly on your commandline and paste the output here?

tb-profiler error report

jodyphelan commented 1 week ago

If you open up your terminal, type in delly and then hit enter.

Did you install with conda? I think delly is not working.

tuelomogashoa commented 1 week ago

Yes I installed with conda

On 24 Jun 2024, at 11:06, Jody Phelan @.***> wrote:

If you open up your terminal, type in delly and then hit enter.

Did you install with conda? I think delly is not working.

— Reply to this email directly, view it on GitHub https://github.com/jodyphelan/TBProfiler/issues/371#issuecomment-2185988868, or unsubscribe https://github.com/notifications/unsubscribe-auth/A4LCKKA7IEX64PF4ZI3BBS3ZI7OSNAVCNFSM6AAAAABJZJY526VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBVHE4DQOBWHA. You are receiving this because you authored the thread.

Sent from Tuelo’s MacBook Pro @.***

tuelomogashoa commented 1 week ago

When I type delly I get an error that reads “Error while loading shared libraries: lib boost_iostreams.so.1.85.0: cannot open shared object file: No such file or directory”

I think this shows that delly is not working?

On 24 Jun 2024, at 11:06, Jody Phelan @.***> wrote:

If you open up your terminal, type in delly and then hit enter.

Did you install with conda? I think delly is not working.

— Reply to this email directly, view it on GitHub https://github.com/jodyphelan/TBProfiler/issues/371#issuecomment-2185988868, or unsubscribe https://github.com/notifications/unsubscribe-auth/A4LCKKA7IEX64PF4ZI3BBS3ZI7OSNAVCNFSM6AAAAABJZJY526VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBVHE4DQOBWHA. You are receiving this because you authored the thread.

Sent from Tuelo’s MacBook Pro @.***

jodyphelan commented 1 week ago

I suspect the conda environment has not correctly been solved. You could download the conda environment file from here: https://raw.githubusercontent.com/jodyphelan/TBProfiler/dev/conda/linux-latest.txt to rebuild your environment. After downloading just run the following command to create a new environment with all the correct dependancies:

wget https://raw.githubusercontent.com/jodyphelan/TBProfiler/dev/conda/linux-latest.txt
conda create --name tb-profiler --file linux-latest.txt
conda activate tb-profiler
tb-profiler
jloubser commented 1 week ago

I apologise for chipping in here....getting some strange errors here @jodyphelan. Please see below. I just removed the openjdk line from the txt file but I don't think this is why we are getting these errors.

(base) [tuelomogashoa@khaos Final_TBP_run]$ mamba create --name tb-profiler --file linux-latest.txt

Downloading and Extracting Packages: pillow-10.0.1 | #########################5 | 19% /home/tuelomogashoa/miniforge3/lib/python3.10/site-packages/tqdm/std.py:636: TqdmWarning: clamping frac to range [0, 1]####################################3 | 98%

Error with archive /home/tuelomogashoa/miniforge3/pkgs/mash-2.3-ha9a2dd8_3.tar.bz2. You probably need to delete and re-download or re-create this file. Message was:

failed with error: Invalid data stream HTTP 404 NOT FOUND for url https://conda.anaconda.org/t/no-15126f85-4dce-4462-b41b1e2_0.conda Elapsed: 00:01.005817 CF-RAY: 898e4289d8690718-CPT

An HTTP error occurred when trying to retrieve this URL. HTTP errors are often intermittent, and a simple retry will get you on your way.

HTTP 404 NOT FOUND for url https://conda.anaconda.org/t/no-15126f85-4dce-4462-b41b1e2_0.conda Elapsed: 00:01.005817 CF-RAY: 898e4289d8690718-CPT

An HTTP error occurred when trying to retrieve this URL. HTTP errors are often intermittent, and a simple retry will get you on your way.

InvalidArchiveError('Error with archive /home/tuelomogashoa/miniforge3/pkgs/mash-2.3-ha9a2dd8_3.tar.bz2. You probably need to delete and re-download or re-create this file. Message was:\n\nfailed with error: Invalid data stream') CondaHTTPError: HTTP 404 NOT FOUND for url https://conda.anaconda.org/t/no-15126f85-4dce-4462-b41b1e2_0.conda Elapsed: 00:01.005817 CF-RAY: 898e4289d8690718-CPT

An HTTP error occurred when trying to retrieve this URL. HTTP errors are often intermittent, and a simple retry will get you on your way.

CondaHTTPError: HTTP 404 NOT FOUND for url https://conda.anaconda.org/t/no-15126f85-4dce-4462-b41b1e2_0.conda Elapsed: 00:01.005817 CF-RAY: 898e4289d8690718-CPT

An HTTP error occurred when trying to retrieve this URL. HTTP errors are often intermittent, and a simple retry will get you on your way.

jloubser commented 1 week ago

Some context for removing the openjdk line... We have seen that it installed a 'strange' internal Java build that has given me issues in the past. Would this openjdk version be suitable for the latest version of TB Profiler: (base) [tuelomogashoa@khaos Final_TBP_run]$ java -version openjdk version "11.0.23" 2024-04-16 LTS OpenJDK Runtime Environment (Red_Hat-11.0.23.0.9-2.el7_9) (build 11.0.23+9-LTS) OpenJDK 64-Bit Server VM (Red_Hat-11.0.23.0.9-2.el7_9) (build 11.0.23+9-LTS, mixed mode, sharing)

jloubser commented 1 week ago

I've restarted it all. Used micromamba. Got an error line for each of the package: warning libmamba Could not validate package '/home/tuelomogashoa/micromamba/pkgs/xorg-recordproto-1.14.2-h7f98852_1002/info/repodata_record.json': md5 and sha256 sum unknown. Final line: Set safety_checks to disabled to override this warning.

But, delly seems to be working...

jodyphelan commented 1 week ago

Some context for removing the openjdk line... We have seen that it installed a 'strange' internal Java build that has given me issues in the past. Would this openjdk version be suitable for the latest version of TB Profiler: (base) [tuelomogashoa@khaos Final_TBP_run]$ java -version openjdk version "11.0.23" 2024-04-16 LTS OpenJDK Runtime Environment (Red_Hat-11.0.23.0.9-2.el7_9) (build 11.0.23+9-LTS) OpenJDK 64-Bit Server VM (Red_Hat-11.0.23.0.9-2.el7_9) (build 11.0.23+9-LTS, mixed mode, sharing)

Yes I think that should be fine - as long as trimmomatic works then that should be ok.

I've restarted it all. Used micromamba. Got an error line for each of the package: warning libmamba Could not validate package '/home/tuelomogashoa/micromamba/pkgs/xorg-recordproto-1.14.2-h7f98852_1002/info/repodata_record.json': md5 and sha256 sum unknown. Final line: Set safety_checks to disabled to override this warning.

I'm not sure why this is happening but seems to be an error specific to the system rather than the environment file?

Now that delly works ok, does tb-profiler run ok also?

taranewman commented 1 week ago

Hi @jodyphelan

Just a note that we've encountered the same delly issue relating to libboost_iostreams.so.1.85.0 and know a few others that have as well for v6.2.0 and 6.2.1 : https://github.com/bioconda/bioconda-recipes/issues/48755

I did not experience this issue when I updated tbprofliler within an existing conda environment using the pypi channel.

@dfornika has worked to pin boost-cpp to v1.85.0 on the delly recipe, however we are still experiencing this issue.

This may get updated, but we are currently seeing this error in our pipeline checks here: https://github.com/BCCDC-PHL/tbprofiler-nf/pull/38/checks

dfornika commented 1 week ago

I hope this is helpful, I'm still a bit confused about what is going on with all of this dependency resolution. But at this point, if I try to create a conda env for TBProfiler:

conda create -n tb-profiler-6.2.1-test tb-profiler=6.2.1

...I'm still getting these builds of boost-cpp and delly:

image

that is:

boost-cpp          conda-forge/linux-64::boost-cpp-1.78.0-h2c5509c_4
delly              bioconda/linux-64::delly-1.2.6-h6dccd9a_2

...and if I try to specify the new build of delly that requires boost-cpp v1.85.0 as follows:

conda create -n tb-profiler-6.2.1-test tb-profiler=6.2.1 delly=1.2.6=hdcf5f25_3

...then I get this:

image

...saying that usher wants an older version of boost-cpp.

But I'm confused because the usher recipe doesn't pin to any specific version of boost-cpp:

https://github.com/bioconda/bioconda-recipes/blob/4b4544a052e8fa802a19d793e7cdcdef88cc55b8/recipes/usher/meta.yaml#L32

...so I don't understand why conda thinks it needs those version ranges. I'll try to ask around on the bioconda gitter to see if anyone there understands what's going wrong.

jodyphelan commented 1 week ago

Interesting, not too sure what's going on here either. For now I could pin a lower version of delly to the tb-profiler recipe as it seems to work ok when delly=v1.1.6 is used?

apetkau commented 6 days ago

Installation seems to work for me for delly 1.2.6 build hb7e2ac5_1, but not for build h6dccd9a_2, which is currently the default that is installed.

Installation that doesn't work

That is, if I install tb-profiler using the below:

conda create --name tb-profiler 'tb-profiler=6.2.1' -y

I will get:

conda list --name tb-profiler 'delly|boost-cpp'

# Name                    Version                   Build  Channel
boost-cpp                 1.78.0               h2c5509c_4    conda-forge
delly                     1.2.6                h6dccd9a_2    bioconda

This installation doesn't work, since delly is linked to libboost 1.85.0 libraries:

ldd envs/tb-profiler/bin/delly

...
        libboost_iostreams.so.1.85.0 => not found
        libboost_filesystem.so.1.85.0 => not found
        libboost_program_options.so.1.85.0 => not found

Installation that is working

But, if I instead install tb-profiler and specify an older build of delly:

conda create --name tb-profiler 'tb-profiler=6.2.1' 'delly=1.2.6=hb7e2ac5_1' -y

I get:

conda list --name tb-profiler 'delly|boost-cpp'

# Name                    Version                   Build  Channel
boost-cpp                 1.78.0               h2c5509c_4    conda-forge
delly                     1.2.6                hb7e2ac5_1    bioconda

This installation does work, since the older build of delly seems linked to the boost 1.78.0 libraries:

ldd tb-profiler/bin/delly

...
        libboost_iostreams.so.1.78.0 => [...]/envs/tb-profiler/bin/../lib/libboost_iostreams.so.1.78.0 (0x00007ff211d8d000)
        libboost_filesystem.so.1.78.0 => [...]/envs/tb-profiler/bin/../lib/libboost_filesystem.so.1.78.0 (0x00007ff211d6d000)
        libboost_program_options.so.1.78.0 => [...]/envs/tb-profiler/bin/../lib/libboost_program_options.so.1.78.0 (0x00007ff211d1b000)

So the difference between delly 1.2.6 build hb7e2ac5_1 and h6dccd9a_2 must have been building against different versions of boost.

dfornika commented 6 days ago

Thanks for doing those tests @apetkau, I hadn't found a way to get a successful dependency resolution and installation.

Based on what I'm seeing, I do think targeting an earlier version of delly (one that uses boost v1.78.0) would make sense. It looks like we need a way to get usher and delly aligned on which version of boost they want when they come together in the tb-profiler environment. I think technically the place to do that is probably the pathogen-profiler recipe because that's where delly is listed as a dependency.

I really hope I haven't made things worse by pinning the delly recipe against boost v1.85.0 or later. Please let me know if you think that was a bad move. I don't have much experience with C++ or the details of how linking is done with .so files. So it's not entirely clear to me how flexible delly is with which versions of boost that it's compatible with. But my error logs were saying that delly wanted boost v1.85.0, and my conda environment had boost v1.78.0, so it seemed sensible to me to pin boost to v1.85.0 or later on the delly recipe.

jodyphelan commented 3 days ago

Thanks for all the help debugging this. I think for now I'll specify delly <=1.1.6 in the pathogen-profiler recipe.

jloubser commented 2 days ago

@jodyphelan all seems to working now, yes. Thank you!