search

google / deepvariant

DeepVariant is an analysis pipeline that uses a deep neural network to call genetic variants from next-generation DNA sequencing data.
BSD 3-Clause "New" or "Revised" License
3.17k stars 716 forks source link

Turning off Intel SSE instructions #123

Closed anands-repo closed 5 years ago

anands-repo commented 5 years ago

Hello, I am trying to install DeepVariant on an IBM Power 8 machine within a docker container.

I get the following error during ./build_and_test.sh, which I understand is tied to Intel SSE2 instruction set.

external/libssw/src/ssw.c:38:23: fatal error: emmintrin.h: No such file or directory

I did export DV_USE_GCP_OPTIMIZED_TF_WHL=0 from the command line before running the compile. I also changed DV_COPT_FLAGS to --copt=-Wno-sign-compare --copt=-Wno-write-strings within settings.sh (removing the corei7 option).

I am using bazel version '0.15.0-' (settings.sh is changed to reflect this)

I am using scikit-learn=0.20 (run-prereq.sh changed to reflect this)

pyclif was compiled from source

Is there a way to circumvent this error? The complete error message is as follows.

ERROR: /root/.cache/bazel/_bazel_root/8422bf851bfac3671a35809acde131a7/external/libssw/BUILD.bazel:11:1: C++ compilation of rule '@libssw//:ssw' failed (Exit 1): gcc failed: error executing command (cd /root/.cache/bazel/_bazel_root/8422bf851bfac3671a35809acde131a7/execroot/com_google_deepvariant && \ exec env - \ LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 \ OMP_NUM_THREADS=1 \ PATH=/root/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ PWD=/proc/self/cwd \ PYTHON_BIN_PATH=/usr/bin/python \ PYTHON_LIB_PATH=/usr/local/lib/python2.7/dist-packages \ TF_DOWNLOAD_CLANG=0 \ TF_NEED_CUDA=0 \ TF_NEED_OPENCL_SYCL=0 \ /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -B/usr/bin -B/usr/bin -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction -sections -fdata-sections -MD -MF bazel-out/ppc-opt/bin/external/libssw/_objs/ssw/external/libssw/src/ssw.pic.d -fPIC -iquote external/libssw -iquote bazel-out/ppc-opt/genfiles/external/libssw -iquote ext ernal/bazel_tools -iquote bazel-out/ppc-opt/genfiles/external/bazel_tools -Wno-maybe-uninitialized -Wno-unused-function -Wno-sign-compare -Wno-write-strings -fno-inline -fno-canonical-system-headers -Wno- builtin-macro-redefined '-DDATE="redacted"' '-DTIMESTAMP="redacted"' '-DTIME="redacted"' -c external/libssw/src/ssw.c -o bazel-out/ppc-opt/bin/external/libssw/_objs/ssw/external/libssw/src/ssw .pic.o) external/libssw/src/ssw.c:38:23: fatal error: emmintrin.h: No such file or directory compilation terminated. Target //:binaries failed to build

depristo commented 5 years ago

Hi @DiableJambe,

We use libssw (https://github.com/mengyao/Complete-Striped-Smith-Waterman-Library) for our local alignment. This code directly uses emmintrin.h in ssw.c so there's no easy way to remove it for the powerpc architecture. That said, there appears to be a pull request to add PPC64 support (https://github.com/mengyao/Complete-Striped-Smith-Waterman-Library/pull/35) but this has been pending for more than a year. We'd be more than happy to update libssw if you can get an implementation in there that doesn't require SSE intrinsics. Let us know if you want to go down that route.

anands-repo commented 5 years ago

Hi @depristo

Thanks for the clarification and the resource!

Right now, my goal is to get the flow working, so I may need to use an Intel machine for the time being. I will need to revisit this later.

I wonder whether this affects only make_examples. If so, it may be good to separate the build into two parts, one for make_examples, and one for the rest. I am not sure whether this is worth supporting from your side. If I am constrained to use Power machines for my subsequent runs, I will be able to look into this aspect.

pichuan commented 5 years ago

Hi @DiableJambe , thanks for reporting the issue. I can keep this bug open for now in case you want to give us more updates. Just note that our team is not actively looking into this right now.

(I also receive notification emails on updated GitHub issues even after they're closed. So keeping this open or closed doesn't really matter much in terms of communication)

qili93 commented 5 years ago

Hi @DiableJambe and @pichuan ,

This issue can be resolved by IBM Advance Toolchain 11.0 :)

Some brief steps for your references of how to build DeepVariant on Power8 & Redhat 7.5

  1. Build the following packages with "/usr/bin/gcc" cmake 3.13.3 Protobuf 3.6.1 C++ (static build with --enable-static for bazel) bazel 0.15.0

  2. Install Advance Toolchain 11.0 and build the following packages with /opt/at11.0/bin/gcc Python 2 and Pip 19.0.2 Protobuf 3.6.1 C++ (uninstall static and build shared) Protobuf 3.6.1 Python (should build and install from source or CLIF will fail) TensorFlow 1.12.0 (fix floatn.h error with the link Floatn.h error: https://gist.github.com/nonchip/2c93ff2d9bc1bf2cd12bc6e76010da0f) CLIF Opencv-python 3.4.5.20 (for tensor2tensor install)

Then build DeepVariant will pass, and results here (excepted //deepvariant/labeler:haplotype_labeler_test tracked in issue 154)

================================================================================
(05:42:50) INFO: Elapsed time: 715.015s, Critical Path: 689.68s
(05:42:50) INFO: 1835 processes: 1835 local.
(05:42:50) INFO: Build completed, 1 test FAILED, 2433 total actions
//deepvariant:allelecounter_test                                         PASSED in 0.1s
//deepvariant:call_variants_test                                         PASSED in 59.8s
//deepvariant:data_providers_test                                        PASSED in 11.8s
//deepvariant:dv_vcf_constants_test                                      PASSED in 0.5s
//deepvariant:exclude_contigs_test                                       PASSED in 1.6s
//deepvariant:haplotypes_test                                            PASSED in 1.7s

▽
//deepvariant:modeling_test                                              PASSED in 48.2s
//deepvariant:pileup_image_test                                          PASSED in 1.8s
//deepvariant:postprocess_variants_lib_test                              PASSED in 0.1s
//deepvariant:postprocess_variants_test                                  PASSED in 4.8s
//deepvariant:resources_test                                             PASSED in 1.8s
//deepvariant:tf_utils_test                                              PASSED in 3.8s
//deepvariant:utils_test                                                 PASSED in 0.1s
//deepvariant:variant_caller_test                                        PASSED in 2.4s
//deepvariant:variant_calling_test                                       PASSED in 0.1s
//deepvariant/environment_tests:env_smoke_test                           PASSED in 0.4s
//deepvariant/environment_tests:protobuf_implementation_test             PASSED in 0.8s
//deepvariant/labeler:customized_classes_labeler_test                    PASSED in 1.8s
//deepvariant/labeler:labeled_examples_to_vcf_test                       PASSED in 1.9s
//deepvariant/labeler:positional_labeler_test                            PASSED in 1.8s
//deepvariant/labeler:variant_labeler_test                               PASSED in 1.8s
//deepvariant/python:allelecounter_wrap_test                             PASSED in 1.7s
//deepvariant/python:variant_calling_wrap_test                           PASSED in 1.8s
//deepvariant/realigner:aligner_test                                     PASSED in 1.7s
//deepvariant/realigner:fast_pass_aligner_test                           PASSED in 0.1s
//deepvariant/realigner:realigner_test                                   PASSED in 3.1s
//deepvariant/realigner:ssw_test                                         PASSED in 0.1s
//deepvariant/realigner:window_selector_test                             PASSED in 1.8s
//deepvariant/realigner/allele_count_linear:generate_trained_model_test  PASSED in 2.7s
//deepvariant/realigner/allele_count_linear:model_evaluation_test        PASSED in 4.3s
//deepvariant/realigner/python:debruijn_graph_wrap_test                  PASSED in 1.8s
//deepvariant/realigner/python:ssw_misc_test                             PASSED in 0.3s
//deepvariant/realigner/python:ssw_wrap_test                             PASSED in 0.3s
//deepvariant/vendor:timer_test                                          PASSED in 0.5s
//deepvariant/labeler:haplotype_labeler_test                             FAILED in 2.6s
  /root/.cache/bazel/_bazel_root/dc155a991b1776fcc65387121539d20a/execroot/com_google_deepvariant/bazel-out/ppc-opt/testlogs/deepvariant/labeler/haplotype_labeler_test/test.log
//deepvariant:make_examples_test                                         PASSED in 13.4s
  Stats over 2 runs: max = 13.4s, min = 12.8s, avg = 13.1s, dev = 0.3s
//deepvariant:model_eval_test                                            PASSED in 40.9s
  Stats over 10 runs: max = 40.9s, min = 2.8s, avg = 7.3s, dev = 11.3s
//deepvariant:model_train_test                                           PASSED in 120.0s
  Stats over 10 runs: max = 120.0s, min = 3.0s, avg = 38.6s, dev = 44.3s
anands-repo commented 5 years ago

Hey @qili93

Thanks for letting me know! Is there a docker image available for this build (DeepVariant or prereqs)? If not, I will try to do it myself.

Thanks!

qili93 commented 5 years ago

Hi @DiableJambe ,

Sorry for my late response, I do not have a docker image yet. And I attached the detailed steps here for your reference. It's a bit long :)

DeepVariant

Environment

# Power8 environment
export CPU=power8
export HOMEPATH=/home/qilibj
export PATH=$HOMEPATH/inst/bin:/usr/local/cuda-10.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin

# AT 11.0 environment
source /etc/profile
module load at11.0
export PATH=/opt/at11.0/bin:/opt/at11.0/sbin:$PATH

# python2 and pip environment
export LD_LIBRARY_PATH=/home/qilibj/inst/lib:$LD_LIBRARY_PATH
export PYTHONPATH=/home/qilibj/inst/lib/python2.7/site-packages
export PYTHON_BIN_PATH=/home/qilibj/inst/bin/python
export PYTHON_LIB_PATH=/home/qilibj/inst/lib/python2.7/site-packages

Cmake 3.13.3

Build and install doc: https://cmake.org/install/

Note: only can be built with system gcc, AT11.0 will cause build failure

# download source code
wget https://github.com/Kitware/CMake/releases/download/v3.13.3/cmake-3.13.3.tar.gz
tar -zxvf cmake-3.13.3.tar.gz
cd cmake-3.13.3

# build scirpt
./bootstrap
make -j20
make -j20 install
export PATH=/usr/local/bin:$PATH

Protobuf 3.6.1 C++ static

C++: https://github.com/protocolbuffers/protobuf/blob/master/src/README.md

Note: Protobuf 3.6.1 should be built from /usr/bin/gcc, static for bazel

# download source code
wget https://github.com/protocolbuffers/protobuf/releases/download/v3.6.1/protobuf-all-3.6.1.tar.gz
tar -zxvf protobuf-all-3.6.1.tar.gz
cd protobuf-3.6.1/

# clean static protobuf build
make uninstall
make distclean

# environment
export CPU=power8
export HOMEPATH=/home/qilibj
export PATH=$HOMEPATH/inst/bin:/usr/local/cuda-10.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin

# share build for C++
# install under /usr instead of /usr/local
CFLAGS="-mcpu=$CPU -mtune=$CPU -O3" CXXFLAGS="-mcpu=$CPU -mtune=$CPU -O3" ./configure --prefix=$HOMEPATH/inst --disable-shared --enable-static
make clean
make -j20
make -j20 check # check if protobuf build is good
# install
make install
sudo ldconfig # refresh shared library cache - /opt/at11.0/sbin/ldconfig
#cd ..

# verify
echo "$(which protoc)"
echo "$(protoc --version)"

Bazel 0.15.0

Build instructions: https://docs.bazel.build/versions/master/install-compile-source.html#bootstrap-bazel

Edit crosstool: http://biophysics.med.jhmi.edu/~yliu120/tensorflow.html

Note: Bazel 0.15.0 should be built from /usr/bin/gcc

# download source code
wget https://github.com/bazelbuild/bazel/releases/download/0.15.0/bazel-0.15.0-dist.zip
mkdir bazel-0.15.0
unzip -n bazel-0.15.0-dist.zip -d ./bazel-0.15.0
cd bazel-0.15.0

# environment
export CPU=power8
export HOMEPATH=/home/qilibj
export JAVA_HOME=/usr/lib/jvm/java-1.8.0
export PATH=$HOMEPATH/inst/bin:$JAVA_HOME/bin:/usr/local/cuda-10.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin

# build from scratch
PROTOC=$HOMEPATH/inst/bin/protoc ./compile.sh
rsync -avP output/bazel $HOMEPATH/inst/bin/
# verification
bazel info

Advance Toolchain 11.0

Doc: https://developer.ibm.com/linuxonpower/advance-toolchain/advtool-installation/

# gpg public key
wget ftp://ftp.unicamp.br/pub/linuxpatch/toolchain/at/redhat/RHEL7/gpg-pubkey-6976a827-5164221b
rpm --import gpg-pubkey-6976a827-5164221b

# Configure the Advance Toolchain repositories
[root@cit1074 deepvariant]# cat /etc/yum.repos.d/advance-toolchain.repo
#Begin of configuration file
[advance-toolchain]
name=Advance Toolchain Unicamp FTP
baseurl=ftp://ftp.unicamp.br/pub/linuxpatch/toolchain/at/redhat/RHEL7
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=ftp://ftp.unicamp.br/pub/linuxpatch/toolchain/at/redhat/RHEL7/gpg-pubkey-6976a827-5164221b
# End of configuration file

# Install the Advance Toolchain
yum install advance-toolchain-at11.0-runtime
yum install advance-toolchain-at11.0-devel
yum install advance-toolchain-at11.0-perf
yum install advance-toolchain-at11.0-mcore-libs

# Load enviroment
export PATH=/opt/at11.0/bin:/opt/at11.0/sbin:$PATH
# Do not need to export
# export LD_LIBRARY_PATH=/opt/at11.0/lib64:/lib64:$LD_LIBRARY_PATH 
# Load environment
sudo yum install environment-modules
source /etc/profile
module load at11.0
module unload at11.0

Python 2 and Pip 19.0.1

Build instructions: https://docs.python.org/2/using/unix.html

Note: Python 2 should be built from AT 11.0

# download source code 
wget https://www.python.org/ftp/python/2.7.15/Python-2.7.15.tgz
tar -zxvf Python-2.7.15.tgz
cd Python-2.7.15

# environment
export HOMEPATH=/home/qilibj
export CPU=power8

# check gcc before build, should be AT11.0
which gcc

# build
CFLAGS="-mcpu=$CPU -mtune=$CPU -O3" CPPFLAGS="-mcpu=$CPU -mtune=$CPU -O3" ./configure --prefix=$HOMEPATH/inst --enable-shared --disable-static
make -j20
make install

# set environment
export PATH=$HOMEPATH/inst/bin:/usr/local/cuda-10.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
export LD_LIBRARY_PATH=/home/qilibj/inst/lib:$LD_LIBRARY_PATH
export PYTHONPATH=/home/qilibj/inst/lib/python2.7/site-packages

# verify 2.7.15
echo "$(python --version)"

# Pip 19.0.2
wget -qc https://bootstrap.pypa.io/get-pip.py --no-check-certificate
$HOMEPATH/inst/bin/python ./get-pip.py --prefix $HOMEPATH/inst
#pip install --upgrade --force-reinstall pip
echo "$(pip --version)"
pip install setuptools nose asv cython future protobuf==3.6.1 six mock
pip install --upgrade setuptools

Protobuf 3.6.1 C++ shared

C++: https://github.com/protocolbuffers/protobuf/blob/master/src/README.md

Note: Protobuf 3.6.1 should be built from AT 11.0, C++ shared for TF 1.12

# download source code
wget https://github.com/protocolbuffers/protobuf/releases/download/v3.6.1/protobuf-all-3.6.1.tar.gz
tar -zxvf protobuf-all-3.6.1.tar.gz
cd protobuf-3.6.1/

# clean static protobuf build
make uninstall
make distclean

# share build for C++
# install under /usr instead of /usr/local
CFLAGS="-mcpu=$CPU -mtune=$CPU -O3" CXXFLAGS="-mcpu=$CPU -mtune=$CPU -O3" ./configure --prefix=$HOMEPATH/inst --enable-shared --disable-static
make clean
make -j20
# optional
make -j20 check # check if protobuf build is good
# install
make install
sudo ldconfig # refresh shared library cache - /opt/at11.0/sbin/ldconfig
#cd ..

# verify
echo "$(which protoc)"
echo "$(protoc --version)"

Protobuf 3.6.1 Python from source code

Python: https://github.com/protocolbuffers/protobuf/blob/master/python/README.md

Note: Protobuf 3.6.1 should be built from source code or CLIF cannot find protobuf

# share build for Python
python --version # python 2.7 or newer
protoc --version
# build
cd protobuf-3.6.1/python/
python setup.py build
python setup.py test
# install from source as deepvariant needed
python setup.py install
# install from wheel
python setup.py bdist_wheel
pip install protobuf-3.6.1-py2.py3-none-any.whl --force-reinstall
# verify
python -c "import google.protobuf"

OpenBLAS 0.3.5

git clone -b v0.3.5 https://github.com/xianyi/OpenBLAS.git OpenBLAS-0.3.5
cd OpenBLAS-0.3.5
make TARGET=power8
make TARGET=power8 PREFIX=$HOMEPATH/inst install

Boost 1.66.0

wget -qc https://dl.bintray.com/boostorg/release/1.66.0/source/boost_1_66_0.tar.gz
tar xzf boost_1_66_0.tar.gz
cd boost_1_66_0
./bootstrap.sh --with-toolset=gcc --prefix=$HOMEPATH/inst
./b2 dll-path="$HOMEPATH/inst/lib" install

TensorFlow 1.12.0

Note: Tensorflow 1.12.0 can be built with AT 11.0

Build instructions: https://www.tensorflow.org/install/source

Edit instructions: http://biophysics.med.jhmi.edu/~yliu120/tensorflow.html

Floatn.h error: https://gist.github.com/nonchip/2c93ff2d9bc1bf2cd12bc6e76010da0f

# development packages
yum install python-devel python-pip -y

# dependency of numpy 1.14.6
OPT="-D_XLOCALE_H=1 -O3 -mcpu=$CPU -mtune=$CPU -maltivec -mvsx -ffast-math -fpeel-loops -funroll-loops -ftree-vectorize -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free" BLAS=$HOMEPATH/inst/lib/libopenblas.so LAPACK=$HOMEPATH/inst/lib/libopenblas.so ATLAS=$HOMEPATH/inst/lib/libopenblas.so pip install numpy==1.14.6
# verify
python -c "import numpy"

# dependecy of scipy 1.2.0
OPT="-O3 -mcpu=$CPU -mtune=$CPU -maltivec -mvsx -ffast-math -fpeel-loops -funroll-loops -ftree-vectorize -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free" BLAS=$HOMEPATH/inst/lib/libopenblas.so LAPACK=$HOMEPATH/inst/lib/libopenblas.so ATLAS=$HOMEPATH/inst/lib/libopenblas.so pip install scipy==1.2.0
# verify
python -c "import scipy"

# pip package dependencies
# pip install pip six wheel mock
pip install wheel autograd h5py==2.9.0 enum34
pip install keras_applications==1.0.6 keras_preprocessing==1.0.5

# download source code
git clone -b r1.12 https://github.com/tensorflow/tensorflow.git tensorflow-1.12
cd tensorflow-1.12

# configure in tensorflow/.tf_configure.bazelrc
PYTHON_BIN_PATH="$HOMEPATH/inst/bin/python" \
PYTHON_LIB_PATH="$HOMEPATH/inst/lib/python2.7/site-packages" \
TF_NEED_IGNITE="0" \
TF_ENABLE_XLA="0" \
TF_NEED_OPENCL_SYCL="0" \
TF_NEED_ROCM="0" \
TF_NEED_MPI="0" \
TF_NEED_TENSORRT="0" \
TF_NEED_CUDA="1" \
TF_CUDA_VERSION="10.0" \
CUDA_TOOLKIT_PATH="/usr/local/cuda" \
TF_CUDNN_VERSION="7" \
CUDNN_INSTALL_PATH="/usr/local/cuda-10.0" \
TF_NCCL_VERSION="2" \
NCCL_INSTALL_PATH="/usr/local/cuda-10.0/targets/ppc64le-linux/lib" \
NCCL_HDR_PATH="/usr/local/cuda-10.0/targets/ppc64le-linux/lib/../include" \
TF_CUDA_COMPUTE_CAPABILITIES="3.7" \
TF_CUDA_CLANG="0" \
GCC_HOST_COMPILER_PATH="/opt/at11.0/bin/gcc" \
CC_OPT_FLAGS="-mcpu=$CPU -mtune=$CPU -O3" \
TF_SET_ANDROID_WORKSPACE=0 \
./configure

# fix build error
vim /opt/at11.0/include/bits/floatn.h
-------------------------------------
#include <features.h>

/* Defined to 1 if the current compiler invocation provides a
   floating-point type with the IEEE 754 binary128 format, and this glibc
   includes corresponding *f128 interfaces for it.  */
#if defined _ARCH_PWR8 && defined __LITTLE_ENDIAN__ && (_CALL_ELF == 2) \
&& defined __FLOAT128__
# define __HAVE_FLOAT128 1
#else
# define __HAVE_FLOAT128 0
#endif

/* add the following block of fix tensorflow build error */
#if CUDART_VERSION
#undef __HAVE_FLOAT128
#define __HAVE_FLOAT128 0
#endif

/* Defined to 1 if __HAVE_FLOAT128 is 1 and the type is ABI-distinct
   from the default float, double and long double types in this glibc.  */
#if __HAVE_FLOAT128
-------------------------------------

# build
bazel clean

export PYTHON_BIN_PATH="$HOMEPATH/inst/bin/python"
bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" --action_env=PYTHON_BIN_PATH=$PYTHON_BIN_PATH --test_env=PYTHON_BIN_PATH=$PYTHON_BIN_PATH --distinct_host_configuration=false

# generate package
bazel-bin/tensorflow/tools/pip_package/build_pip_package $HOMEPATH/tensorflow_package

# install
pip install $HOMEPATH/tensorflow_package/tensorflow-1.12.0-*.whl

# verification
python -c "import tensorflow as tf; print(tf.__version__)"

CLIF

Note: CLIF can be built with AT 11.0

Git Repository: https://github.com/google/clif

# Prerequisites
cmake --version #3.5+
protoc --version # 3.2.0+ build from source code for both C++ and Python
pip install virtualenv
pip install pyparsing
yum install subversion
yum install ocaml
pip install 'pyparsing>=2.2.0'
pkg-config --libs python # workable

# download source code
cd $HOMEPATH
git clone https://github.com/google/clif.git
cd clif

# set environment
export INSTALL_DIR="$HOMEPATH/inst"
export CLIFSRC_DIR="$HOMEPATH/clif"
export LLVM_DIR="$CLIFSRC_DIR/../clif_backend"
export BUILD_DIR="$LLVM_DIR/build_matcher"

export PYTHON_BIN_PATH="$HOMEPATH/inst/bin/python"
export PYTHON_LIB_PATH="$HOMEPATH/inst/lib/python2.7/site-packages"
export PROTOC_PREFIX_PATH="$(dirname "$(dirname "$(which protoc)")")"

export CLIF_VIRTUALENV="$INSTALL_DIR"/clif
export CLIF_PIP="$CLIF_VIRTUALENV/bin/pip"

virtualenv -p "$PYTHON_BIN_PATH" "$CLIF_VIRTUALENV"

$CLIF_PIP install --upgrade pip
$CLIF_PIP install --upgrade setuptools

# Checkout LLVM and Clang source trees
mkdir -p $LLVM_DIR
cd $LLVM_DIR
svn co https://llvm.org/svn/llvm-project/llvm/trunk@307315 llvm
cd llvm/tools
svn co https://llvm.org/svn/llvm-project/cfe/trunk@307315 clang
ln -s -f -n $CLIFSRC_DIR/clif clif

# Builds must be done outside of the LLVM tree.
mkdir -p $BUILD_DIR
cd $BUILD_DIR
# Note to remove -DLLVM_TARGETS_TO_BUILD=X86
# "rm CMakeCache.txt" to remove cmake cache
cmake -DCMAKE_INSTALL_PREFIX="$CLIF_VIRTUALENV/clang" \
  -DCMAKE_PREFIX_PATH="$PROTOC_PREFIX_PATH" \
  -DLLVM_INSTALL_TOOLCHAIN_ONLY=true \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLVM_BUILD_DOCS=false \
  -DLLVM_TARGETS_TO_BUILD=PowerPC \
  -DPYTHON_INCLUDE_DIR="$HOMEPATH/inst/include/python2.7" \
  -DPYTHON_LIBRARY="$HOMEPATH/inst/lib/libpython2.7.so" \
  -DPYTHON_EXECUTABLE="$HOMEPATH/inst/bin/python" \
  "$LLVM_DIR/llvm"
make -j20 clif-matcher
# export LD_LIBRARY_PATH=/home/qilibj/inst/lib:$LD_LIBRARY_PATH
make -j20 clif_python_utils_proto_util
make -j20 install

## Get back to the CLIF Python directory and have pip run setup.py.
cd "$CLIFSRC_DIR"
# Grab the python compiled .proto
cp "$BUILD_DIR/tools/clif/protos/ast_pb2.py" clif/protos/
# Grab CLIF generated wrapper implementation for proto_util.
cp "$BUILD_DIR/tools/clif/python/utils/proto_util.cc" clif/python/utils/
cp "$BUILD_DIR/tools/clif/python/utils/proto_util.h" clif/python/utils/
cp "$BUILD_DIR/tools/clif/python/utils/proto_util.init.cc" clif/python/utils/
# install
export C_INCLUDE_PATH=/home/qilibj/inst/include
export CPLUS_INCLUDE_PATH=/home/qilibj/inst/include
"$CLIF_PIP" install .
# echo "SUCCESS - To use pyclif, run $CLIF_VIRTUALENV/bin/pyclif."
python setup.py bdist_wheel
# Note: pyclif should be installed into virtualenv
"$CLIF_PIP" install pyclif-0.3-cp27-none-linux_ppc64le.whl
pip install dist/pyclif-0.3-cp27-cp27m-linux_ppc64le.whl

# verify
python -c "from clif.python.proto import start"

# link for deepvariant
ln -s /home/qilibj/inst/clif /usr/local/

Opencv-python 3.4.5.20

Git repository: https://github.com/skvark/opencv-python

# Checkout repository and submodules
git clone https://github.com/skvark/opencv-python.git
cd opencv-python/
# fetch the tags to your local repository
git fetch --all --tags --prune
# check out tag 3.4.5.20
git checkout tags/20
# load submoduel
git submodule update --init --recursive

# Dependency
pip install pyparsing
yum install qt-devel
# Build
python setup.py bdist_wheel

# Insatll
pip install dist/opencv_python-3.4.5.20-cp27-cp27mu-linux_ppc64le.whl

# Verify in a new session
python -c "import cv2"

DV Prerequisite

####################################################################
# misc setup
####################################################################

# development packages
yum install python2-pkgconfig zip zlib-devel unzip curl -y
# python packages
yum install python-devel python-pip python-wheel -y

####################################################################
# python packages
####################################################################

# python 2 required
echo "$(python --version)"
echo "$(pip --version)"

# Install python packages
pip install contextlib2
pip install enum34
pip install intervaltree
pip install 'mock>=2.0.0'

# pip install 'numpy==1.14' => skip as installed in TF

pip install 'requests>=2.18'
# pip install 'scipy==1.0' => skip as installed in TF
pip install 'oauth2client>=4.0.0'
pip install 'crcmod>=1.7'
pip install six
pip install sklearn
pip install pandas
pip install psutil
pip install --upgrade google-api-python-client

####################################################################
# depend on opencv-python wheel - build from source
####################################################################
pip install 'tensor2tensor>=1.9.0'

####################################################################
# depend on - TensorFlow - 1.12 build from source
####################################################################
pip install tensorflow-1.12.0-cp27-cp27mu-linux_ppc64le.whl

####################################################################
# Misc dependencies
####################################################################

yum install openssl-devel curl-devel zlib-devel bzip2-devel xz-devel
yum install boost-devel

DeepVariant

Git Repository https://github.com/google/deepvariant

# check out source code
git clone https://github.com/google/deepvariant.git
cd deepvariant
# fetch all tags
git fetch --all --tags --prune
# check out tag
git checkout tags/v0.7.2

# Edit ctx.action with use_default_shell_env=True
vim ./third_party/clif.bzl

# Build and test
export LD_LIBRARY_PATH=/home/qilibj/inst/lib:/qilibj/inst/lib/python2.7/site-packages:$LD_LIBRARY_PATH
export PYTHONPATH=/home/qilibj/inst/lib/python2.7/site-packages
export PYTHON_BIN_PATH=/home/qilibj/inst/bin/python
export PYTHON_LIB_PATH=/home/qilibj/inst/lib/python2.7/site-packages
export BAZEL_PYTHON=/home/qilibj/inst/bin/python
export DV_COPT_FLAGS="--copt=-maltivec --copt=-Wno-sign-compare --copt=-Wno-write-strings --copt=-DNO_WARN_X86_INTRINSICS --cxxopt=-std=gnu++11"
# export DV_COPT_FLAGS="--copt=-maltivec --copt=-Wno-sign-compare --copt=-Wno-write-strings --copt=-DNO_WARN_X86_INTRINSICS --cxxopt=-std=gnu++11 --copt=-fsigned-char --cxxopt=-fsigned-char"

# for GPU enabled
# fix "ImportError: No module named google.protobuf" by install protobuf from source
bazel clean
bazel shutdown
bazel test -c opt --local_test_jobs=1 ${DV_COPT_FLAGS} "$@" deepvariant/... \
--action_env=LD_LIBRARY_PATH --test_env=LD_LIBRARY_PATH \
--action_env=PYTHONPATH=$PYTHONPATH --test_env=PYTHONPATH=$PYTHONPATH \
--action_env=PYTHON_BIN_PATH=$PYTHON_BIN_PATH --test_env=PYTHON_BIN_PATH=$PYTHON_BIN_PATH \
--action_env=PYTHON_LIB_PATH=$PYTHON_LIB_PATH --test_env=PYTHON_LIB_PATH=$PYTHON_LIB_PATH \
--action_env=BAZEL_PYTHON=$BAZEL_PYTHON --test_env=BAZEL_PYTHON=$BAZEL_PYTHON >& output.log 2>&1 &
bazel test -c opt --local_test_jobs=1 ${DV_COPT_FLAGS} "$@" deepvariant:gpu_tests --action_env=PYTHONPATH=$PYTHONPATH --test_env=PYTHONPATH=$PYTHONPATH

# for CPU only
bazel test -c opt ${DV_COPT_FLAGS} "$@" deepvariant/...

# build binary
bazel build -c opt ${DV_COPT_FLAGS} "$@" deepvariant:binaries --build_python_zip --action_env=PYTHONPATH=$PYTHONPATH --test_env=PYTHONPATH=$PYTHONPATH
echo 'Expect a usage message:'
(bazel-bin/deepvariant/call_variants --help || : ) | grep '/call_variants.py:'

bazel build :licenses_zip --action_env=PYTHONPATH=$PYTHONPATH --test_env=PYTHONPATH=$PYTHONPATH

Fix DV Error

################################################################################
# bazel test -c opt --local_test_jobs=1 ${DV_COPT_FLAGS} "$@"//deepvariant:resources_test
# use lscpu to show the actual CPU number
################################################################################
python -c "import multiprocessing; print(multiprocessing.cpu_count())" #160
python -c "import psutil;print(p/sutil.cpu_count
())" #160

vim deepvariant/resources.py
--------------------------------
def _get_cpu_count():
  """Gets the number of physical cores in this machine.
  Returns:
    int >= 1 if the call to get the cpu_count succeeded, or 0 if not.
  """
  # return psutil.cpu_count(logical=False) or 0 ==> comment
  return 20
--------------------------------

vim deepvariant/resources_test.py
--------------------------------
  def test_metrics_is_ok_when_cpu_count_returns_none(self):
    # Some psutil functions, such as cpu_freq(), can return None depending on
    # the environment; make sure we don't crash when that occurs.
    with mock.patch.object(resources.psutil, 'cpu_count', return_value=None):
      with resources.ResourceMonitor() as monitor:
        #self.assertEqual(monitor.metrics().physical_core_count, 0) ==> comment
        self.assertEqual(monitor.metrics().physical_core_count, 20)
--------------------------------

##########################################################################
# //deepvariant/realigner/allele_count_linear:generate_trained_model_test
# ImportError: /root/.local/lib/python2.7/site-packages/sklearn/__check_build/_check_build.so: undefined symbol: PyUnicodeUCS4_DecodeUTF8
##########################################################################
# reinstall numpy, scipy, Cpython, scikit-learn to fix with AT built Python
python -c "import numpy" # prequests of TF 1.12.0
python -c "import scipy" # prequests of TF 1.12.0
pip install Cython --force-reinstall --no-deos
pip install scikit-learn --force-reinstall --no-deos
# build from source
wget https://github.com/scikit-learn/scikit-learn/archive/0.20.2.tar.gz
tar zxvf 0.20.2.tar.gz
cd scikit-learn-0.20.2
python setup.py bdist_wheel
# verify
python -c "from sklearn.externals import joblib"

##########################################################################
# //deepvariant/labeler:haplotype_labeler_test
# bazel test -c opt --local_test_jobs=1 ${DV_COPT_FLAGS} "$@"//deepvariant/labeler:haplotype_labeler_test --action_env=PYTHONPATH=$PYTHONPATH --test_env=PYTHONPATH=$PYTHONPATH
##########################################################################
# fail due to mock data, open an issue in github
https://github.com/google/deepvariant/issues/154

##########################################################################
# //deepvariant:make_examples_test
# bazel test -c opt --local_test_jobs=1 ${DV_COPT_FLAGS} "$@"//deepvariant:make_examples_test --action_env=PYTHONPATH=$PYTHONPATH --test_env=PYTHONPATH=$PYTHONPATH
# internvaltree v3 has some API changes with v2
##########################################################################
pip install 'intervaltree==2.1.0'
pichuan commented 5 years ago

Hi @DiableJambe , I noticed this issue is still open from a while ago. So I want to give a quick update and close this issue. DeepVariant 0.8.0 is out yesterday. It might still not solve your original issue, but I just want to give you a heads up in case you want to try out the new version. From @qili93 's last response, many versions of the dependencies will be different now. But hopefully it helped resolved your original problem. I'll close this issue now. If you have more questions about the new version, please feel to open another issue.

anands-repo commented 4 years ago

@qili93 I am trying your method above. I understand it's from a long time ago, but do you remember having issues with glibc and bazel? I am running into some issues because on CentOS7 (and RHEL7 I guess?) the default system gcc is very old, while Advance Toolchain has a much newer gcc version.

Thanks!

anands-repo commented 4 years ago

The above issue was due to a problem with numpy compilation. Newer versions of numpy need to be compiled from source for POWER platforms.