search

ratschlab / metagraph

Scalable annotated de Bruijn graphs for DNA indexing, alignment, and assembly
http://metagraph.ethz.ch
GNU General Public License v3.0
110 stars 17 forks source link

Create conda package (to omit brew issues) #231

Closed michalkowalski94 closed 3 years ago

michalkowalski94 commented 3 years ago

Currently build and deploy of metagraph is limited only to docker and macOS users (instructions in README and wiki are not sufficient). Install with gcc on linux servers requires sudo.

karasikov commented 3 years ago

Hi Michal,

I have been able to install and currently actively using MetaGraph with gcc and all dependencies on Linux without root, using Linuxbrew (see details in the README).

Could you please specify what issue you're facing?

michalkowalski94 commented 3 years ago

I cannot compile dependencies on CentOS(7), Ubuntu (18.04, 20.04), Debian(9) with this instruction. I managed to compile fully functioning Docker image with this Dockerfile (compilation of KMC inside).


#Good practice
LABEL maintainer="m.kowalski@doctoral.uj.edu.pl"
#Environmental variables set for all image to get along with UTF-8
ENV LANG=C.UTF-8 LC_ALL=C.UTF-8
#Essential libraries and software
COPY ./metagraph /usr/bin/metagraph
SHELL ["/bin/bash", "-c"]
RUN mkdir /home/datadir && \ 
    apt-get update -y && \
    apt-get upgrade -y && \
    apt-get install -y apt-utils software-properties-common -y && \
    add-apt-repository ppa:jonathonf/gcc-9.0 -y && \
    apt-get update -y && \
    apt-get install gcc-9 g++-9 -y && \
    apt-get install curl wget git ccache doxygen python3-venv -y && \
    apt-get install gcc g++ autoconf automake libtool make -y && \
    apt-get install libomp-dev libcurl4-gnutls-dev libbz2-dev -y && \
    apt-get install zlib1g-dev libncurses5-dev libncursesw5-dev -y && \
    apt-get install libgoogle-glog-dev liblzma-dev libboost-all-dev -y && \
    apt-get install libevent-dev libdouble-conversion-dev libgflags-dev -y && \
    apt-get install libiberty-dev liblz4-dev libsnappy-dev binutils-dev -y && \
    apt-get install libjemalloc-dev libssl-dev pkg-config libunwind-dev -y && \
    apt-get install libunwind8-dev libelf-dev libdwarf-dev -y && \
    export CC=/usr/bin/gcc-9 && export CXX=/usr/bin/g++-9 
#Upgrade of cmake version to 3.12
RUN apt-get purge cmake -y && \
    version=3.12 && \
    build=3 && \
    mkdir /temp && cd /temp && \
    wget https://cmake.org/files/v$version/cmake-$version.$build.tar.gz && \
    tar -xzvf cmake-$version.$build.tar.gz && \
    cd cmake-$version.$build/ && \
    ./bootstrap && \
    make -j4 && make install && \
    cd / && rm -rf /temp && \
    cmake --version
#Installation of folly and its dependencies
RUN cd /usr/bin && \
    wget https://github.com/google/googletest/archive/release-1.8.0.tar.gz && \
    tar zxf release-1.8.0.tar.gz && \
    rm -f release-1.8.0.tar.gz && \
    cd googletest-release-1.8.0 && \
    cmake . && \
    make && \
    make install && \
    cd /bin && \
    git clone https://github.com/fmtlib/fmt.git && cd fmt && \
    mkdir _build && cd _build && \
    cmake .. && make -j$(nproc) && \
    make install && \
    cd /bin && \
    git clone https://github.com/facebook/folly.git && \
    cd folly && \
    mkdir _build && cd _build && \
    cmake .. && make -j $(nproc) && \
    make install
#Installation of htslib
RUN cd /usr/bin && \
    wget https://github.com/samtools/htslib/releases/download/1.10.2/htslib-1.10.2.tar.bz2 && \
    tar -vxjf htslib-1.10.2.tar.bz2 && \
    rm htslib-1.10.2.tar.bz2 && \
    cd htslib-1.10.2 && \
    make && make install
#Installation of metagraph itself
RUN add-apt-repository ppa:deadsnakes/ppa -y && \
    apt-get update -y && \
    apt-get install python3.4 python3.4-venv -y && \
    cd /usr/bin && \
    git clone --recursive https://github.com/ratschlab/metagraph.git && \
    cd metagraph && \
    git submodule sync && \
    git submodule update --init --recursive && \
    pushd metagraph/external-libraries/sdsl-lite && \
    ./install.sh $PWD && \
    popd && \
    pushd metagraph/external-libraries/libmaus2 && \
    cmake -DCMAKE_INSTALL_PREFIX:PATH=$PWD . && \
    make -j $(($(getconf _NPROCESSORS_ONLN) - 1)) && \
    make install && \
    popd && \
    mkdir -p metagraph/build && cd metagraph/build && \
    cmake .. && make -j $(($(getconf _NPROCESSORS_ONLN) - 1)) && \
    cp /usr/local/lib/libhts.so.3 /usr/lib/  && \
    ./unit_tests && \
    make check && \
    echo 'export PATH="/usr/bin/metagraph/metagraph/build":$PATH' ~/.bash_profile```
And that's the only sudo free solution available for me right now
michalkowalski94 commented 3 years ago

Currently my efforts for building conda pa are facing this issue:

[  1%] Building CXX object external-libraries/googletest/googletest/CMakeFiles/gtest.dir/src/gtest-all.cc.o
x86_64-conda-linux-gnu-c++: error: =strict-aliasing: No such file or directory
x86_64-conda-linux-gnu-c++: error: =strict-aliasing: No such file or directory
make[2]: *** [external-libraries/jsoncpp/src/lib_json/CMakeFiles/jsoncpp_lib.dir/build.make:76: external-libraries/jsoncpp/src/lib_json/CMakeFiles/jsoncpp_lib.dir/json_value.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
Scanning dependencies of target spdlog
[  1%] Building C object zlib/CMakeFiles/zlib.dir/adler32.c.o
make[2]: *** [external-libraries/jsoncpp/src/lib_json/CMakeFiles/jsoncpp_lib.dir/build.make:63: external-libraries/jsoncpp/src/lib_json/CMakeFiles/jsoncpp_lib.dir/json_reader.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:1534: external-libraries/jsoncpp/src/lib_json/CMakeFiles/jsoncpp_lib.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

with end result in 

[ 24%] Building CXX object external-libraries/benchmark/src/CMakeFiles/benchmark.dir/reporter.cc.o
[ 25%] Building CXX object external-libraries/benchmark/src/CMakeFiles/benchmark.dir/sleep.cc.o
[ 25%] Building CXX object external-libraries/benchmark/src/CMakeFiles/benchmark.dir/statistics.cc.o
[ 25%] Building CXX object external-libraries/benchmark/src/CMakeFiles/benchmark.dir/string_util.cc.o
[ 26%] Building CXX object external-libraries/benchmark/src/CMakeFiles/benchmark.dir/sysinfo.cc.o
[ 26%] Building CXX object external-libraries/benchmark/src/CMakeFiles/benchmark.dir/timers.cc.o
[ 28%] Linking CXX static library libspdlog.a
[ 28%] Built target spdlog
[ 28%] Linking CXX static library ../../../lib/libgtest.a
[ 28%] Built target gtest
[ 28%] Linking CXX static library libbenchmark.a
[ 28%] Built target benchmark
make: *** [Makefile:141: all] Error 2
Traceback (most recent call last):

I also had managed to install metagraph with brew on debian, so probably in near future I'll create pull request for update of wiki. Unfortunately result of integration tests looks like this:

[==========] 811 tests ran. (881.44 sec total)
[  PASSED  ] 811 tests.
[  ERRORS  ] 22 test(s), listed below:
[  ERRORS  ] setUpClass (test_query.TestQueryPrimary_12_succinct_rbfish)
[  ERRORS  ] setUpClass (test_query.TestQueryPrimary_16_bitmap_rb_brwt)
[  ERRORS  ] setUpClass (test_query.TestQueryPrimary_1_hashfast_row)
[  ERRORS  ] setUpClass (test_query.TestQueryPrimary_23_hash_brwt)
[  ERRORS  ] setUpClass (test_query.TestQueryPrimary_27_succinct_bloom_row)
[  ERRORS  ] setUpClass (test_query.TestQueryPrimary_30_succinct_bloom_rbfish)
[  ERRORS  ] setUpClass (test_query.TestQueryPrimary_34_succinct_mask_rb_brwt)
[  ERRORS  ] setUpClass (test_query.TestQueryPrimary_3_hashfast_brwt)
[  ERRORS  ] setUpClass (test_query.TestQueryPrimary_7_succinct_row)
[  ERRORS  ] setUpClass (test_query.TestQueryPrimary_8_succinct_row_diff)
[  ERRORS  ] setUpClass (test_query.TestQueryPrimary_9_succinct_row_diff_brwt)
[  ERRORS  ] setUpClass (test_query.TestQuery_12_bitmap_rbfish)
[  ERRORS  ] setUpClass (test_query.TestQuery_16_hash_rb_brwt)
[  ERRORS  ] setUpClass (test_query.TestQuery_1_succinct_row)
[  ERRORS  ] setUpClass (test_query.TestQuery_23_hashfast_brwt)
[  ERRORS  ] setUpClass (test_query.TestQuery_27_hashstr_row)
[  ERRORS  ] setUpClass (test_query.TestQuery_2_succinct_row_diff)
[  ERRORS  ] setUpClass (test_query.TestQuery_33_succinct_bloom_row)
[  ERRORS  ] setUpClass (test_query.TestQuery_37_succinct_bloom_flat)
[  ERRORS  ] setUpClass (test_query.TestQuery_3_succinct_row_diff_brwt)
[  ERRORS  ] setUpClass (test_query.TestQuery_43_succinct_mask_flat)
[  ERRORS  ] setUpClass (test_query.TestQuery_7_succinct_flat)
----------------------------------------------------------------------
Ran 811 tests in 946.098s
karasikov commented 3 years ago

Hi Michal,

The error log looks incomplete. Could you send a full listing?

I wonder if these issues are caused by a conflict with other libraries you have installed on the system. Can you try to reset environment variables to make sure you're using all dependencies installed with Linuxbrew?

I also had managed to install metagraph with brew on debian, so probably in near future I'll create pull request for update of wiki.

Your contribution will be very much appreciated, thanks.

hmusta commented 3 years ago

We have simplified the installation procedure for various Linux distributions to no longer require non-standard packages. Could you please try it out to see if it works for you now?

michalkowalski94 commented 3 years ago

Sure, on it 😊

From: Harun Mustafa notifications@github.com Sent: Monday, January 18, 2021 2:13 PM To: ratschlab/metagraph metagraph@noreply.github.com Cc: Michał Kowalski michal.bozydar.kowalski@gmail.com; Author author@noreply.github.com Subject: Re: [ratschlab/metagraph] Create conda package (to omit brew issues) (#231)

We have simplified the installation procedure for various Linux distributions to no longer require non-standard packages. Could you please try it out to see if it works for you now?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ratschlab/metagraph/issues/231#issuecomment-762242399 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AJSADOCUOIIP5INRR6YW5E3S2QXWLANCNFSM4TRJBQ6A . https://github.com/notifications/beacon/AJSADOFFSEVIZFAKDS7GZFDS2QXWLA5CNFSM4TRJBQ6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFVXOKXY.gif

michalkowalski94 commented 3 years ago

Hi there Sorry for late response Docker looks great, everything is working on own docker image. Dockerfile provided by you has some bugs regarding libmaus. Conda build unfortunately is still crashing:

[ 39%] Linking CXX static library ../../../lib/libgtest.a
[ 39%] Built target gtest
Scanning dependencies of target gmock
[ 39%] Building CXX object external-libraries/googletest/googlemock/CMakeFiles/gmock.dir/src/gmock-all.cc.o
[ 40%] Building CXX object CMakeFiles/metagraph-core.dir/src/graph/alignment/cigar.cpp.o
In file included from /home/mkowalski/conda-bld/metagraph_1612780921057/work/metagraph/src/graph/representation/bitmap/dbg_bitmap.cpp:1:
/home/mkowalski/conda-bld/metagraph_1612780921057/work/metagraph/src/graph/representation/bitmap/dbg_bitmap.hpp:23:7: error: 'mtg::graph::DBGBitmap' declared with greater visibility than the type of its field 'mtg::graph::DBGBitmap::kmers_' [-Werror=attributes]
   23 | class DBGBitmap : public DeBruijnGraph {
      |       ^~~~~~~~~
In file included from /home/mkowalski/conda-bld/metagraph_1612780921057/work/metagraph/src/graph/representation/bitmap/dbg_bitmap.cpp:7:
/home/mkowalski/conda-bld/metagraph_1612780921057/work/metagraph/src/graph/representation/bitmap/dbg_bitmap_construct.hpp:11:7: error: 'mtg::graph::IBitmapChunkConstructor' declared with greater visibility than its base 'mtg::graph::IGraphChunkConstructor<bit_vector_adaptive_stat<smart_representation> >' [-Werror=attributes]
   11 | class IBitmapChunkConstructor : public IGraphChunkConstructor<DBGBitmap::Chunk> {
      |       ^~~~~~~~~~~~~~~~~~~~~~~
In file included from /home/mkowalski/conda-bld/metagraph_1612780921057/work/metagraph/src/graph/representation/bitmap/dbg_bitmap_construct.hpp:4,
                 from /home/mkowalski/conda-bld/metagraph_1612780921057/work/metagraph/src/graph/representation/bitmap/dbg_bitmap_construct.cpp:1:
/home/mkowalski/conda-bld/metagraph_1612780921057/work/metagraph/src/graph/representation/bitmap/dbg_bitmap.hpp:23:7: error: 'mtg::graph::DBGBitmap' declared with greater visibility than the type of its field 'mtg::graph::DBGBitmap::kmers_' [-Werror=attributes]
   23 | class DBGBitmap : public DeBruijnGraph {
      |       ^~~~~~~~~
In file included from /home/mkowalski/conda-bld/metagraph_1612780921057/work/metagraph/src/graph/representation/bitmap/dbg_bitmap_construct.cpp:1:
/home/mkowalski/conda-bld/metagraph_1612780921057/work/metagraph/src/graph/representation/bitmap/dbg_bitmap_construct.hpp:11:7: error: 'mtg::graph::IBitmapChunkConstructor' declared with greater visibility than its base 'mtg::graph::IGraphChunkConstructor<bit_vector_adaptive_stat<smart_representation> >' [-Werror=attributes]
   11 | class IBitmapChunkConstructor : public IGraphChunkConstructor<DBGBitmap::Chunk> {
      |       ^~~~~~~~~~~~~~~~~~~~~~~
[ 40%] Building CXX object CMakeFiles/metagraph-core.dir/src/graph/alignment/aligner_seeder_methods.cpp.o
[ 40%] Building CXX object CMakeFiles/metagraph-core.dir/src/graph/alignment/aligner_helper.cpp.o
[ 41%] Building CXX object CMakeFiles/metagraph-core.dir/src/graph/alignment/aligner_extender_methods.cpp.o
[ 41%] Building CXX object CMakeFiles/metagraph-core.dir/src/graph/alignment/aligner_config.cpp.o
[ 41%] Building CXX object CMakeFiles/metagraph-core.dir/src/common/vectors/wavelet_tree.cpp.o
[ 43%] Building CXX object CMakeFiles/metagraph-core.dir/src/common/vectors/vector_algorithm.cpp.o
[ 43%] Building CXX object CMakeFiles/metagraph-core.dir/src/common/vectors/bitmap_mergers.cpp.o
Scanning dependencies of target gtest_main
[ 43%] Building CXX object external-libraries/googletest/googletest/CMakeFiles/gtest_main.dir/src/gtest_main.cc.o
[ 44%] Linking CXX static library ../../../lib/libgmock.a
[ 44%] Built target gmock
[ 44%] Building CXX object CMakeFiles/metagraph-core.dir/src/common/vectors/bitmap.cpp.o
[ 45%] Building CXX object CMakeFiles/metagraph-core.dir/src/common/vectors/bit_vector.cpp.o
[ 46%] Linking CXX static library ../../../lib/libgtest_main.a
[ 46%] Building CXX object CMakeFiles/metagraph-core.dir/src/common/utils/string_utils.cpp.o
[ 46%] Built target gtest_main
[ 46%] Building CXX object CMakeFiles/metagraph-core.dir/src/common/utils/file_utils.cpp.o
[ 48%] Building CXX object CMakeFiles/metagraph-core.dir/src/common/unix_tools.cpp.o
[ 48%] Building CXX object CMakeFiles/metagraph-core.dir/src/common/threads/threading.cpp.o
[ 48%] Building CXX object CMakeFiles/metagraph-core.dir/src/common/sorted_sets/sorted_set_disk_base.cpp.o
[ 49%] Building CXX object CMakeFiles/metagraph-core.dir/src/common/sorted_sets/sorted_set_disk.cpp.o
[ 49%] Building CXX object CMakeFiles/metagraph-core.dir/src/common/sorted_sets/sorted_set.cpp.o
[ 49%] Building CXX object CMakeFiles/metagraph-core.dir/src/common/sorted_sets/sorted_multiset_disk.cpp.o
[ 50%] Building CXX object CMakeFiles/metagraph-core.dir/src/common/sorted_sets/sorted_multiset.cpp.o
[ 50%] Building CXX object CMakeFiles/metagraph-core.dir/src/common/serialization.cpp.o
[ 50%] Building CXX object CMakeFiles/metagraph-core.dir/src/common/range_partition.cpp.o
cc1plus: all warnings being treated as errors
make[2]: *** [CMakeFiles/metagraph-core.dir/build.make:342: CMakeFiles/metagraph-core.dir/src/graph/representation/bitmap/dbg_bitmap.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
cc1plus: all warnings being treated as errors
make[2]: *** [CMakeFiles/metagraph-core.dir/build.make:329: CMakeFiles/metagraph-core.dir/src/graph/representation/bitmap/dbg_bitmap_construct.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:622: CMakeFiles/metagraph-core.dir/all] Error 2
make: *** [Makefile:160: all] Error 2
Traceback (most recent call last):
  File "/opt/anaconda3/bin/conda-build", line 11, in <module>
    sys.exit(main())
  File "/opt/anaconda3/lib/python3.8/site-packages/conda_build/cli/main_build.py", line 481, in main
    execute(sys.argv[1:])
  File "/opt/anaconda3/lib/python3.8/site-packages/conda_build/cli/main_build.py", line 470, in execute
    outputs = api.build(args.recipe, post=args.post, test_run_post=args.test_run_post,
  File "/opt/anaconda3/lib/python3.8/site-packages/conda_build/api.py", line 186, in build
    return build_tree(
  File "/opt/anaconda3/lib/python3.8/site-packages/conda_build/build.py", line 3068, in build_tree
    packages_from_this = build(metadata, stats,
  File "/opt/anaconda3/lib/python3.8/site-packages/conda_build/build.py", line 2191, in build
    utils.check_call_env(cmd, env=env, rewrite_stdout_env=rewrite_env,
  File "/opt/anaconda3/lib/python3.8/site-packages/conda_build/utils.py", line 412, in check_call_env
    return _func_defaulting_env_to_os_environ('call', *popenargs, **kwargs)
  File "/opt/anaconda3/lib/python3.8/site-packages/conda_build/utils.py", line 392, in _func_defaulting_env_to_os_environ
    raise subprocess.CalledProcessError(proc.returncode, _args)
subprocess.CalledProcessError: Command '['/bin/bash', '-o', 'errexit', '/home/mkowalski/conda-bld/metagraph_1612780921057/work/conda_build.sh']' returned non-zero exit status 2.

Also, may i ask for docker-compose file?

michalkowalski94 commented 3 years ago

There is also problem with integration tests, no matter what they do not see "parameterized" module

karasikov commented 3 years ago

Hi Michal,

There were some residuals of libmaus remaining in the docker config, I cleaned that up. (branch dev) It should cause no issues now.

As for the integration tests, they should work fine if you follow the instructions and set up a python environment with cmake .., which calls https://github.com/ratschlab/metagraph/blob/10da1c0c102880ae6b0fd26e501857b9917cfe4d/metagraph/integration_tests/install_test_environment.sh)

Regarding the conda build, I suppose you wrote your own config file? Could you please post it here?

Have you checked the CI build file? https://github.com/ratschlab/metagraph/blob/10da1c0c102880ae6b0fd26e501857b9917cfe4d/.github/workflows/main.yml It installs everything, including the tests in github actions. For other systems, the process should be similar.

michalkowalski94 commented 3 years ago

Yes i had checked it. About CI, i want to say that You are the best :D I checked recent release and managed to create conda package for linux-64 out of it. It is available in here

michalkowalski94 commented 3 years ago

@karasikov ping me in this thread when releases for macOS and linux will be in production phase, I'll test anaconda recipe and share it with You, so You could post it to Your channel or bioconda one

izcram commented 3 years ago

@michalkowalski94 thanks for your effort and this first step! :) I've just started looking into the conda packaging for metagraph and will start working on writing a recipe which builds the binaries from source.

izcram commented 3 years ago

conda package was created on bioconda It can be installed using conda install -c bioconda metagraph