search

biocorecrg / master_of_pores

Nextflow pipeline for analysis of direct RNA Nanopore reads
https://biocorecrg.github.io/master_of_pores/
MIT License
93 stars 16 forks source link

How to install extra apt packages easily in biocorecrg-mopbasecall #91

Closed eylenth closed 3 years ago

eylenth commented 3 years ago

Hello A bioscientist is trying to execute the following command from the NanoPreprocess directory: nextflow run nanopreprocess.nf -with-singularity

But then we see the following error guppy_basecaller: error while loading shared libraries: libssl.so.10: cannot open shared object file: No such file or directory which means that some apt packages are missing in the biocorecrg-mopbasecall-0.1 container It seems the guppy_basecaller is missing the following libraries in the singularity container:

How can I easily install extra apt packages in biocorecrg-mopbasecall container ? I am not a nextflow export, so I don't know if there is even an 'easy' way to do this.

lucacozzuto commented 3 years ago

Hi, I don't know if we tested that version. Can you try with the version we described in the paper? If you still have the problem is likely another kind of problem.

@ADelgadoT did you tested the version 4.0.11?

ADelgadoT commented 3 years ago

I tested version 4.0.15 and it worked. I did not test 4.0.11.

lucacozzuto commented 3 years ago

If someone can pass me the newest version I can test it

L

Il mar 8 dic 2020, 20:38 ADelgadoT notifications@github.com ha scritto:

I tested version 4.0.15 and it worked. I did not test 4.0.11.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/biocorecrg/master_of_pores/issues/91#issuecomment-740916977, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADZ5FPKQPXBL5QP6X5MME23STZ6FBANCNFSM4USFZ3YQ .

eylenth commented 3 years ago

I'm using the rpm installation to install guppy on my client server. I don't know if this matter?

eylenth commented 3 years ago

Hi it seems to be working when I use the tarball of guppy instead of the rpm version. But I also needed to modifiy the following file: $MASTEROFPORESINSTALL/NanoPreprocess/nanopreprocess.nf modified gpu_prefix = 'export LD_LIBRARY_PATH="/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs"' to gpu_prefix = 'export LD_LIBRARY_PATH="/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs:${GUPPYSOFTAWAREPATH}/4.2.2/lib"' so it could also read the guppy libraries inside the singularity container.

lucacozzuto commented 3 years ago

I don't understand. Have you placed the guppy folder inside bin in NanoPreprocess? Like here:

ls ont-guppy_xx.xx.xx_linux64/ont-guppy/
bin data    lib

you should have bin, data e lib folders. Then once you make links like here:

ls
guppy_aligner -> ont-guppy_xx.xx.xx_linux64/ont-guppy/bin/guppy_aligner
guppy_barcoder -> ont-guppy_xx.xx.xx_linux64/ont-guppy/bin/guppy_barcoder
guppy_basecaller -> ont-guppy_xx.xx.xx_linux64/ont-guppy/bin/guppy_basecaller

it should automatically mount the lib ...

eylenth commented 3 years ago

It didn't work for me..

$ pwd
${MASTEROFPORESINSTALLDIR}/20201104-guppy-4.2.2/NanoPreprocess/bin
$ ls ont-guppy
bin  data  lib
$ ls -ls
total 128
  0 -rwxr-xr-x 1 easybuild easybuild  3220 Dec 10 17:39 bam2stats.py
128 -rwxr-xr-x 1 easybuild easybuild 21907 Dec 10 17:39 deeplexicon.py
  0 -rwxr-xr-x 1 easybuild easybuild  1117 Dec 10 17:39 extract_sequence_from_fastq.py
  0 -rwxr-xr-x 1 easybuild easybuild  1416 Dec 10 17:39 fast5_to_fastq.py
  0 -rwxr-xr-x 1 easybuild easybuild   642 Dec 10 17:39 fast5_type.py
  0 lrwxrwxrwx 1 easybuild easybuild    27 Dec 10 17:41 guppy_aligner -> ont-guppy/bin/guppy_aligner
  0 lrwxrwxrwx 1 easybuild easybuild    28 Dec 10 17:41 guppy_barcoder -> ont-guppy/bin/guppy_barcoder
  0 lrwxrwxrwx 1 easybuild easybuild    35 Dec 10 17:41 guppy_basecall_client -> ont-guppy/bin/guppy_basecall_client
  0 lrwxrwxrwx 1 easybuild easybuild    30 Dec 10 17:41 guppy_basecaller -> ont-guppy/bin/guppy_basecaller
  0 lrwxrwxrwx 1 easybuild easybuild    41 Dec 10 17:41 guppy_basecaller_supervisor -> ont-guppy/bin/guppy_basecaller_supervisor
  0 lrwxrwxrwx 1 easybuild easybuild    35 Dec 10 17:41 guppy_basecall_server -> ont-guppy/bin/guppy_basecall_server
  0 drwxr-xr-x 5 easybuild easybuild  4096 Dec 10 14:53 ont-guppy
  0 -rwxr-xr-x 1 easybuild easybuild   899 Dec 10 17:39 reorganize.sh
$ ldd guppy_barcoder
    linux-vdso.so.1 =>  (0x00007fffdb3b0000)
    libboost_thread.so.1.66.0 => not found
    libboost_chrono.so.1.66.0 => not found
    libboost_system.so.1.66.0 => not found
    libboost_date_time.so.1.66.0 => not found
    libboost_atomic.so.1.66.0 => not found
    libz.so.1 => /lib64/libz.so.1 (0x00007fec8af31000)
    libboost_filesystem.so.1.66.0 => not found
    libboost_iostreams.so.1.66.0 => not found
    libboost_log.so.1.66.0 => not found
    libboost_log_setup.so.1.66.0 => not found
    libboost_program_options.so.1.66.0 => not found
    libboost_regex.so.1.66.0 => not found
    libcuda.so.1 => /lib64/libcuda.so.1 (0x00007fec89ddb000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fec89bd7000)
    librt.so.1 => /lib64/librt.so.1 (0x00007fec899cf000)
    libm.so.6 => /lib64/libm.so.6 (0x00007fec896cd000)
    libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fec893c6000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fec891b0000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fec88f94000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fec88bc7000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fec8b147000)
    libnvidia-fatbinaryloader.so.418.39 => /lib64/libnvidia-fatbinaryloader.so.418.39 (0x00007fec88979000)
lucacozzuto commented 3 years ago

what is inside ont-guppy folder?

L

eylenth commented 3 years ago 
$ ls ont-guppy
bin  data  lib
lucacozzuto commented 3 years ago

and inside lib?

lucacozzuto commented 3 years ago

so the structure I have is:

master_of_pores/NanoPreprocess/bin/ont-guppy_3.1.5_linux64/ont-guppy/lib

is the same you have for the newer version?

eylenth commented 3 years ago 
$ ls -ls ont-guppy/lib/
total 20352
   0 lrwxrwxrwx 1 easybuild easybuild      25 Sep 21 16:26 libboost_atomic.so -> libboost_atomic.so.1.66.0
 128 -rw-r--r-- 1 easybuild easybuild    9019 Oct 22  2019 libboost_atomic.so.1.66.0
   0 lrwxrwxrwx 1 easybuild easybuild      25 Sep 21 16:26 libboost_chrono.so -> libboost_chrono.so.1.66.0
 128 -rw-r--r-- 1 easybuild easybuild   37603 Oct 22  2019 libboost_chrono.so.1.66.0
   0 lrwxrwxrwx 1 easybuild easybuild      28 Sep 21 16:26 libboost_date_time.so -> libboost_date_time.so.1.66.0
 128 -rw-r--r-- 1 easybuild easybuild  108155 Oct 22  2019 libboost_date_time.so.1.66.0
   0 lrwxrwxrwx 1 easybuild easybuild      29 Sep 21 16:26 libboost_filesystem.so -> libboost_filesystem.so.1.66.0
 128 -rw-r--r-- 1 easybuild easybuild  122153 Oct 22  2019 libboost_filesystem.so.1.66.0
   0 lrwxrwxrwx 1 easybuild easybuild      28 Sep 21 16:26 libboost_iostreams.so -> libboost_iostreams.so.1.66.0
 256 -rw-r--r-- 1 easybuild easybuild  139806 Oct 22  2019 libboost_iostreams.so.1.66.0
   0 lrwxrwxrwx 1 easybuild easybuild      28 Sep 21 16:26 libboost_log_setup.so -> libboost_log_setup.so.1.66.0
1152 -rw-r--r-- 1 easybuild easybuild 1141501 Oct 22  2019 libboost_log_setup.so.1.66.0
   0 lrwxrwxrwx 1 easybuild easybuild      22 Sep 21 16:26 libboost_log.so -> libboost_log.so.1.66.0
1280 -rw-r--r-- 1 easybuild easybuild 1218520 Oct 22  2019 libboost_log.so.1.66.0
   0 lrwxrwxrwx 1 easybuild easybuild      34 Sep 21 16:26 libboost_program_options.so -> libboost_program_options.so.1.66.0
 640 -rw-r--r-- 1 easybuild easybuild  554344 Oct 22  2019 libboost_program_options.so.1.66.0
   0 lrwxrwxrwx 1 easybuild easybuild      24 Sep 21 16:26 libboost_regex.so -> libboost_regex.so.1.66.0
1280 -rw-r--r-- 1 easybuild easybuild 1272462 Oct 22  2019 libboost_regex.so.1.66.0
   0 lrwxrwxrwx 1 easybuild easybuild      25 Sep 21 16:26 libboost_system.so -> libboost_system.so.1.66.0
 128 -rw-r--r-- 1 easybuild easybuild   18513 Oct 22  2019 libboost_system.so.1.66.0
   0 lrwxrwxrwx 1 easybuild easybuild      25 Sep 21 16:26 libboost_thread.so -> libboost_thread.so.1.66.0
 256 -rw-r--r-- 1 easybuild easybuild  199538 Oct 22  2019 libboost_thread.so.1.66.0
   0 lrwxrwxrwx 1 easybuild easybuild      19 Sep 21 16:26 libcrypto.so -> libcrypto.so.1.0.1e
   0 lrwxrwxrwx 1 easybuild easybuild      19 Sep 21 16:26 libcrypto.so.10 -> libcrypto.so.1.0.1e
2048 -rw-r--r-- 1 easybuild easybuild 1971488 Aug 14  2019 libcrypto.so.1.0.1e
 768 -rw-r--r-- 1 easybuild easybuild  751872 Oct 22  2019 libcurl.so
   0 lrwxrwxrwx 1 easybuild easybuild      19 Sep 21 16:26 libhdf5_hl.so -> libhdf5_hl.so.8.0.0
 384 -rw-r--r-- 1 easybuild easybuild  335623 Oct 22  2019 libhdf5_hl.so.1.8.11
   0 lrwxrwxrwx 1 easybuild easybuild      20 Sep 21 16:26 libhdf5_hl.so.8.0.0 -> libhdf5_hl.so.1.8.11
   0 lrwxrwxrwx 1 easybuild easybuild      16 Sep 21 16:26 libhdf5.so -> libhdf5.so.8.0.0
9344 -rw-r--r-- 1 easybuild easybuild 9487318 Oct 22  2019 libhdf5.so.1.8.11
   0 lrwxrwxrwx 1 easybuild easybuild      17 Sep 21 16:26 libhdf5.so.8.0.0 -> libhdf5.so.1.8.11
   0 lrwxrwxrwx 1 easybuild easybuild      25 Sep 21 16:26 libont_minimap2.so.2 -> libont_minimap2.so.2.17.0
 256 -rw-r--r-- 1 easybuild easybuild  192368 Sep 21 16:12 libont_minimap2.so.2.17.0
   0 lrwxrwxrwx 1 easybuild easybuild      21 Sep 21 16:26 libpgm-5.1.so.0 -> libpgm-5.1.so.0.0.118
 384 -rw-r--r-- 1 easybuild easybuild  294848 Dec 22  2012 libpgm-5.1.so.0.0.118
   0 lrwxrwxrwx 1 easybuild easybuild      16 Sep 21 16:26 libssl.so -> libssl.so.1.0.1e
   0 lrwxrwxrwx 1 easybuild easybuild      16 Sep 21 16:26 libssl.so.10 -> libssl.so.1.0.1e
 512 -rw-r--r-- 1 easybuild easybuild  443416 Aug 14  2019 libssl.so.1.0.1e
   0 lrwxrwxrwx 1 easybuild easybuild      22 Sep 21 16:26 libvbz_hdf_plugin.so -> libvbz_hdf_plugin.so.1
   0 lrwxrwxrwx 1 easybuild easybuild      26 Sep 21 16:26 libvbz_hdf_plugin.so.1 -> libvbz_hdf_plugin.so.1.0.0
 640 -rwxr-xr-x 1 easybuild easybuild  591992 Sep 21 16:26 libvbz_hdf_plugin.so.1.0.0
   0 lrwxrwxrwx 1 easybuild easybuild      15 Sep 21 16:26 libzmq.so -> libzmq.so.3.0.0
   0 lrwxrwxrwx 1 easybuild easybuild      15 Sep 21 16:26 libzmq.so.3 -> libzmq.so.3.0.0
 384 -rw-r--r-- 1 easybuild easybuild  316688 Feb 21  2015 libzmq.so.3.0.0
   0 lrwxrwxrwx 1 easybuild easybuild      13 Dec 10 14:56 libz.so -> libz.so.1.2.3
   0 lrwxrwxrwx 1 easybuild easybuild      13 Sep 21 16:26 libz.so.1 -> libz.so.1.2.3
 128 -rw-r--r-- 1 easybuild easybuild   88600 Feb 22  2013 libz.so.1.2.3
   0 -rw-r--r-- 1 easybuild easybuild    1235 Sep 21 16:12 MINIMAP2_LICENSE
eylenth commented 3 years ago

my structure looks like this ls NanoPreprocess/bin/ont-guppy/lib/ and We have guppy 4.2.2

lucacozzuto commented 3 years ago

ok so this is the problem I think. Make another folder named ont-guppy_4.2.2_linux64 and put ont-guppy inside to recreate the structure we have. Then make the soft links and try again

L

eylenth commented 3 years ago

It won't work either:

$ ls -ls
total 128
-rwxr-xr-x 1 easybuild easybuild  3220 Dec 10 17:39 bam2stats.py
-rwxr-xr-x 1 easybuild easybuild 21907 Dec 10 17:39 deeplexicon.py
-rwxr-xr-x 1 easybuild easybuild  1117 Dec 10 17:39 extract_sequence_from_fastq.py
-rwxr-xr-x 1 easybuild easybuild  1416 Dec 10 17:39 fast5_to_fastq.py
-rwxr-xr-x 1 easybuild easybuild   642 Dec 10 17:39 fast5_type.py
lrwxrwxrwx 1 easybuild easybuild    51 Dec 10 18:03 guppy_aligner -> ont-guppy_4.2.2_linux64/ont-guppy/bin/guppy_aligner
lrwxrwxrwx 1 easybuild easybuild    52 Dec 10 18:03 guppy_barcoder -> ont-guppy_4.2.2_linux64/ont-guppy/bin/guppy_barcoder
lrwxrwxrwx 1 easybuild easybuild    59 Dec 10 18:03 guppy_basecall_client -> ont-guppy_4.2.2_linux64/ont-guppy/bin/guppy_basecall_client
lrwxrwxrwx 1 easybuild easybuild    54 Dec 10 18:03 guppy_basecaller -> ont-guppy_4.2.2_linux64/ont-guppy/bin/guppy_basecaller
lrwxrwxrwx 1 easybuild easybuild    65 Dec 10 18:03 guppy_basecaller_supervisor -> ont-guppy_4.2.2_linux64/ont-guppy/bin/guppy_basecaller_supervisor
lrwxrwxrwx 1 easybuild easybuild    59 Dec 10 18:03 guppy_basecall_server -> ont-guppy_4.2.2_linux64/ont-guppy/bin/guppy_basecall_server
drwxrwxr-x 3 easybuild easybuild  4096 Dec 10 18:03 ont-guppy_4.2.2_linux64
-rwxr-xr-x 1 easybuild easybuild   899 Dec 10 17:39 reorganize.sh
 $ ldd guppy_barcoder
    linux-vdso.so.1 =>  (0x00007ffcda121000)
    libboost_thread.so.1.66.0 => not found
    libboost_chrono.so.1.66.0 => not found
    libboost_system.so.1.66.0 => not found
    libboost_date_time.so.1.66.0 => not found
    libboost_atomic.so.1.66.0 => not found
    libz.so.1 => /lib64/libz.so.1 (0x00007f6cc5f1c000)
    libboost_filesystem.so.1.66.0 => not found
    libboost_iostreams.so.1.66.0 => not found
    libboost_log.so.1.66.0 => not found
    libboost_log_setup.so.1.66.0 => not found
    libboost_program_options.so.1.66.0 => not found
    libboost_regex.so.1.66.0 => not found
    libcuda.so.1 => /lib64/libcuda.so.1 (0x00007f6cc4dc6000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f6cc4bc2000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f6cc49ba000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f6cc46b8000)
    libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f6cc43b1000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f6cc419b000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6cc3f7f000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f6cc3bb2000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f6cc6132000)
    libnvidia-fatbinaryloader.so.418.39 => /lib64/libnvidia-fatbinaryloader.so.418.39 (0x00007f6cc3964000)
lucacozzuto commented 3 years ago

No but the missing libraries are in the container, so it is should be ok. Try to launch MoP. My ldd shows that these libraries are missing for me too.

eylenth commented 3 years ago

Seems to be working with this procedure.

eylenth commented 3 years ago

maybe can you clearly document on the installation procedure that you have to use the tarball of guppy and not the rpm one. and copy the tarball to this exact location and not another location.

lucacozzuto commented 3 years ago

Hi, here the documentation about the installation. https://biocorecrg.github.io/master_of_pores/install.html Feel free to suggest any modifications.

eylenth commented 3 years ago

c) Guppy Please note Guppy versions older than 3.1 (e.g. 3.0.3) only runs on CPUs. Newer versions (e.g. 3.1.5 and above) works on both CPUs and GPUs. The difference of speed between CPUs and GPU is more than 10 times. Please do not use the rpm of Guppy, you have to use the tarball of Guppy.

cd master_of_pores/NanoPreprocess/bin
tar -zvxf ont-guppy_3.1.5_linux64.tar.gz #do not put the guppy tarball at another location.
ln -s ont-guppy_3.1.5_linux64/ont-guppy/bin/guppy_* .
lucacozzuto commented 3 years ago

Ok, I added a note. Thanks for this advice! https://github.com/biocorecrg/master_of_pores/commit/f02a076f03d25060b9f57d977a5c349c9c8b108d