Bazel build error

[yanyang907]

Hi, I was working on update the deepvariant source code from ubuntu 16.04 to 18.04 and python 3.6 to python 3.8. Now I met a problem in build_release_binaries.shell scripts.

bazel build -c opt \ --output_filter=DONT_MATCH_ANYTHING \ --noshow_loading_progress \ --show_result=0 \ ${DV_COPT_FLAGS} \ --build_python_zip \ :binaries

The error is below: [1,442 / 1,802] Compiling third_party/nucleus/protos/struct.pb.cc; 1s local ... (128 actions, 48 running) (17:42:57) [1,544 / 1,802] Compiling external/org_tensorflow/tensorflow/core/util/test_log.pb.cc; 6s local ... (128 actions, 47 running) (17:43:03) ERROR: /opt/deepvariant/deepvariant/realigner/python/BUILD:54:1: C++ compilation of rule '//deepvariant/realigner/python:ssw_cclib' failed (Exit 1): gcc failed: error executing command (cd /root/.cache/bazel/_bazel_root/617054f44dc1f1e9b3fe3174f8eb2580/execroot/com_google_deepvariant && \ exec env - \ PATH=/root/bin:/root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ PWD=/proc/self/cwd \ PYTHON_BIN_PATH=/usr/bin/python3.8 \ PYTHON_LIB_PATH=/usr/local/lib/python3.8/dist-packages \ TF2_BEHAVIOR=1 \ TF_CONFIGURE_IOS=0 \ TF_ENABLE_XLA=1 \ /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/k8-opt/bin/deepvariant/realigner/python/_objs/ssw_cclib/ssw.pic.d '-frandom-seed=bazel-out/k8-opt/bin/deepvariant/realigner/python/_objs/ssw_cclib/ssw.pic.o' -fPIC -DTF_USE_SNAPPY -DEIGEN_MPL2_ONLY '-DEIGEN_MAX_ALIGN_BYTES=64' '-DEIGEN_HAS_TYPE_TRAITS=0' -DHAVE_SYS_UIO_H -D__CLANG_SUPPORT_DYN_ANNOTATION -iquote . -iquote bazel-out/k8-opt/bin -iquote external/libssw -iquote bazel-out/k8-opt/bin/external/libssw -iquote external/org_tensorflow -iquote bazel-out/k8-opt/bin/external/org_tensorflow -iquote external/com_google_absl -iquote bazel-out/k8-opt/bin/external/com_google_absl -iquote external/nsync -iquote bazel-out/k8-opt/bin/external/nsync -iquote external/eigen_archive -iquote bazel-out/k8-opt/bin/external/eigen_archive -iquote external/local_config_sycl -iquote bazel-out/k8-opt/bin/external/local_config_sycl -iquote external/gif -iquote bazel-out/k8-opt/bin/external/gif -iquote external/libjpeg_turbo -iquote bazel-out/k8-opt/bin/external/libjpeg_turbo -iquote external/com_google_protobuf -iquote bazel-out/k8-opt/bin/external/com_google_protobuf -iquote external/zlib -iquote bazel-out/k8-opt/bin/external/zlib -iquote external/com_googlesource_code_re2 -iquote bazel-out/k8-opt/bin/external/com_googlesource_code_re2 -iquote external/farmhash_archive -iquote bazel-out/k8-opt/bin/external/farmhash_archive -iquote external/fft2d -iquote bazel-out/k8-opt/bin/external/fft2d -iquote external/highwayhash -iquote bazel-out/k8-opt/bin/external/highwayhash -iquote external/double_conversion -iquote bazel-out/k8-opt/bin/external/double_conversion -iquote external/snappy -iquote bazel-out/k8-opt/bin/external/snappy -iquote external/clif -iquote bazel-out/k8-opt/bin/external/clif -iquote external/local_config_python -iquote bazel-out/k8-opt/bin/external/local_config_python -isystem external/nsync/public -isystem bazel-out/k8-opt/bin/external/nsync/public -isystem external/eigen_archive -isystem bazel-out/k8-opt/bin/external/eigen_archive -isystem external/gif -isystem bazel-out/k8-opt/bin/external/gif -isystem external/com_google_protobuf/src -isystem bazel-out/k8-opt/bin/external/com_google_protobuf/src -isystem external/zlib -isystem bazel-out/k8-opt/bin/external/zlib -isystem external/farmhash_archive/src -isystem bazel-out/k8-opt/bin/external/farmhash_archive/src -isystem external/double_conversion -isystem bazel-out/k8-opt/bin/external/double_conversion -isystem external/local_config_python/python_include -isystem bazel-out/k8-opt/bin/external/local_config_python/python_include -w -DAUTOLOAD_DYNAMIC_KERNELS -Wno-maybe-uninitialized -Wno-unused-function '-march=corei7' -Wno-sign-compare -Wno-write-strings '-std=c++14' '-std=c++11' -fno-canonical-system-headers -Wno-builtin-macro-redefined '-DDATE="redacted"' '-DTIMESTAMP="redacted"' '-DTIME__="redacted"' -c bazel-out/k8-opt/bin/deepvariant/realigner/python/ssw.cc -o bazel-out/k8-opt/bin/deepvariant/realigner/python/_objs/ssw_cclib/ssw.pic.o) Execution platform: @local_execution_config_platform//:platform bazel-out/k8-opt/bin/deepvariant/realigner/python/ssw.cc:283:1: error: cannot convert 'std::nullptr_t' to 'Py_ssize_t {aka long int}' in initialization }; ^ bazel-out/k8-opt/bin/deepvariant/realigner/python/ssw.cc:534:1: error: cannot convert 'std::nullptr_t' to 'Py_ssize_t {aka long int}' in initialization }; ^ bazel-out/k8-opt/bin/deepvariant/realigner/python/ssw.cc:815:1: error: cannot convert 'std::nullptr_t' to 'Py_ssize_t {aka long int}' in initialization };

I knew that "nullptr" might be change to "NULL" in pyth3.8. But how should I change the specific files?

[pichuan]

Hi @yanyang1989 , internally we've updated to Ubuntu18.04 and made a bunch of updates accordingly too. It will come out in the next release.

We're still using Python 3.6 though. Can you tell us why you're considering using 3.8?

[ASLeonard]

Python 3.6 is meant to be "unsupported" by the end of this year (pep 494), so would be decent future-proofing or ensuring security.

[yanyang907]

Hi @pichuan , I am planning to integrate deepvariant into a whole germline variant calling pipeline which is using python3.8 and built everything into a Docker. So I need to build every prereq relating deepvariant inside it.

[aliceseaborn]

I have been experiencing the same error when trying to build DeepVariant from the Dockerfile they provided but I managed to pull working images from Docker by following the steps below:

BIN_VERSION="1.1.0-gpu"
docker image pull google/deepvariant:$BIN_VERSION

Pulling the image from remote is not ideal for us as I would prefer to use the Dockerfile to increase transparency in how the image is created. However, having a working DeepVariant is better than nothing.

We are using Ubuntu 16.04 but we would greatly prefer to run Ubuntu 18.04 as our OS for running DeepVariant. Does DeepVariant 1.1.0 now support Ubuntu 18.04 hosts?

Thanks!

[pichuan]

Thanks @ASLeonard for the pointer on the timeline for Python 3.6 and 3.8. I'll take a look to update the Python version. @yanyang1989 thanks for sharing your use case! We'll look into updating to Python 3.8 this but it will take a while. If you can try pulling our image directly like @aliceseaborn , that might work for you now?

@aliceseaborn My experience is that google/deepvariant:1.1.0 and google/deepvariant:1.1.0-gpu work with no issues directly on Ubuntu 18.04 machines. Are you having any issues when you pull our images?

[yanyang907]

is it possible to make this available https://storage.googleapis.com/deepvariant/packages/oss_clif_py3/oss_clif.ubuntu-20.latest.tgz

Since I have another problem about "Install CLIF binary" using ubuntu 20.04. Quote from build-prereq.sh script:

Figure out which linux installation we are on to fetch an appropriate # version of the pre-built CLIF binary. Note that we only support now Ubuntu # 14, 16, and 18.

[pichuan]

@yanyang1989 in our next release (which we have internally updated already), we'll be building clif from scratch. However, in our next release we will only support Ubuntu18.

For any other versions, please directly refer to https://github.com/google/clif to build clif binaries.

[pichuan]

Please see https://github.com/google/deepvariant/issues/443#issuecomment-821755400 for how to update to Ubuntu 18.04. I haven't tried updating to Python 3.8 yet. But I'll create an internal issue to track it. Closing this issue now, but feel free to add more questions/comments if related to this one, or feel free to open another issue for any other questions.

[yanyang907]

@pichuan Hi! When I downloaded all the pre-built binaries and run the run-prereq.sh scripts. How I suppose to run the make_example parallel? I use the following command: seq 0 47 | parallel -q --halt 2 --line-buffer python3 make_examples.zip --mode calling --ref /data/input/human_g1k_v37.fasta --reads /data/input/c6c4c1db-4328-4aa9-b038-074c9a453117.dedup.bam --examples make_examples.tfrecord@12.gz

showing error:

E0430 18:57:45.160124 140015706085184 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_9oblbsyi/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '33']". E0430 18:57:45.128387 140717112878912 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfilesgd9fj22/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '2']". E0430 18:57:45.149224 139802704738112 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_2hbfxd92/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '16']". E0430 18:57:45.196900 140351706515264 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles__ytoy0a6/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '4']". E0430 18:57:45.304501 140371551012672 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_w7jwxlqo/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '28']". E0430 18:57:45.337900 139621533210432 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_22p2hp0z/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '45']". E0430 18:57:45.424449 139874966882112 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_wv1oakms/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '46']". E0430 18:57:45.268234 140085723301696 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_g62feq4g/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '10']". E0430 18:57:45.190006 140494428116800 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_gsw00kpo/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '12']". E0430 18:57:45.278714 140080891017024 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles__lr697ii/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '20']". E0430 18:57:45.366031 140648053425984 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_9e_e2zvn/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '38']". E0430 18:57:45.197047 140371481966400 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_9fc41k4h/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '14']". E0430 18:57:45.279598 140654497855296 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_rybr_dmc/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '34']". E0430 18:57:45.291190 140144580835136 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_xwtzfol0/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '15']". E0430 18:57:45.354358 139648338528064 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_3baynv6p/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '47']". E0430 18:57:45.249558 140521800120128 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_hpf7iban/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '21']". E0430 18:57:45.245847 140159308461888 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_0ekpfvin/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '24']". E0430 18:57:45.283955 140581469652800 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_brxc6_sa/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '7']". E0430 18:57:45.303462 139795832272704 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfilessfy61tb/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '29']". E0430 18:57:45.331091 140537432155968 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_y60taw80/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '31']". E0430 18:57:45.239560 139714767980352 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_v3df_cxj/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '17']". E0430 18:57:45.262542 140488697198400 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_1k7ckjbn/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '44']". E0430 18:57:45.264230 139873562601280 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_qtxrg7dl/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '6']". E0430 18:57:45.193969 139692519348032 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_05qe3qax/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '36']". E0430 18:57:45.197033 140638292031296 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_75j2ynwd/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '5']". E0430 18:57:45.264279 139811152422720 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_i060jaf0/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '27']". E0430 18:57:45.313135 139973217675072 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_p96gjj9f/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '11']". E0430 18:57:45.286854 139660079433536 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_fhgr5den/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '26']". E0430 18:57:45.266479 139743940249408 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_09rk_ns4/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '9']". E0430 18:57:45.246133 140107102431040 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_i98_1ues/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '13']". E0430 18:57:45.303056 140048653686592 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_yk1hqf6a/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '39']". E0430 18:57:45.368143 139769855092544 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_3ci75isq/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '30']". E0430 18:57:45.312847 140640366352192 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_3bfx8qjn/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '3']". E0430 18:57:45.285222 140677590058816 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_pbohkh7d/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '40']". E0430 18:57:45.251970 140221024618304 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_r241voor/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '0']". E0430 18:57:45.291264 139938218350400 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_q2sdkm67/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '23']". E0430 18:57:45.246095 140479227369280 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_uphv_tdy/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '37']". E0430 18:57:45.317537 139819477477184 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_4750gtic/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '43']". E0430 18:57:45.318566 140493027030848 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_mw222qvc/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '19']". E0430 18:57:45.281981 139640920373056 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_087sqql0/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '18']". E0430 18:57:45.269598 140672023549760 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_prpwm4tu/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '41']". E0430 18:57:45.343238 140414174025536 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_dqd3ut4s/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '32']". E0430 18:57:45.247818 140240365713216 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_8s9w7qaa/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '25']". E0430 18:57:45.247906 139736525375296 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_8kqng5_c/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '1']". E0430 18:57:45.354531 139703252227904 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_47rk8xc1/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '22']". E0430 18:57:45.318170 140515109386048 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_4p5rc3ja/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '35']". E0430 18:57:45.306068 140062873229120 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_xavizfpc/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '42']". E0430 18:57:45.268234 140590012761920 errors.py:61] Command line parsing failure: make_examples does not accept positional arguments but some are present on the command line: "['/tmp/Bazel.runfiles_ovmu_l59/runfiles/com_google_deepvariant/deepvariant/make_examples.py', '8']". parallel: This job failed:

could you provide any guidance on how to run the make_example.zip working on parallel (like create a 48 jobs)? Thanks

[pichuan]

Hi @yanyang1989 If you're specifying a flag like : --examples make_examples.tfrecord@12.gz  , you'll need to run make_examples for each of the 12 shards (0 to 11) by specifying --task {} in your parallel command.

Usually we recommend people to directly use the run_deepvariant one-step command (see : https://github.com/google/deepvariant/blob/r1.1/docs/deepvariant-quick-start.md#run-deepvariant-with-one-command ) which will take care of this for you.

If you want an example of how to run the 3-step approach (and an example of how to use parallel with make_examples), you can see this example from r0.7 documentation: https://github.com/google/deepvariant/blob/r0.7/docs/deepvariant-quick-start.md#make_examples

[yanyang907]

Hi @pichuan , if not using pipe in the command to run make_example.zip. How to run this script: I am using

seq 0 47 | parallel --halt 2 \ python3 make_examples.zip \ --mode calling \ --ref /data/input/human_g1k_v37.fasta \ --reads /data/input/dedup.bam \ --examples make_examples.tfrecord@48.gz \ --gvcf gvcf.tfrecord@48.gz \ --task {}

I don't want to use pipe. How to edit the code? (seq 0 47 | parallel --halt 2)

[pichuan]

Hi @yanyang1989 , you mean you don't want to use "|"? You could use a for loop in bash that iterate from 0 to 47, instead of seq and |.

Basically, you want to run

python3 make_examples.zip
--mode calling
--ref /data/input/human_g1k_v37.fasta
--reads /data/input/dedup.bam
--examples make_examples.tfrecord@48.gz
--gvcf gvcf.tfrecord@48.gz
--task $IDX

where $IDX ranges from 0 to 47 to cover all 48 shards.