Bazel failed to build behind proxy

NearLinHere commented 6 years ago

Description of the problem / feature request:

I would like to follow tensorflow example to build generate_streaming_test_wav to generate test wav. The Bazel version on my machine is 0.16.1.

The problem is when I use the command bazel run tensorflow/examples/speech_commands:generate_streaming_test_wav , the following error message shown up:

xxx@xxx:~/kws/tensorflow-0911$ bazel run tensorflow/examples/speech_commands:generate_streaming_test_wav
Starting local Bazel server and connecting to it...
ERROR: error loading package '': Encountered error while reading extension file 'closure/defs.bzl': no such package '@io_bazel_rules_closure//closure': Error downloading [https://mirror.bazel.build/github.com/bazelbuild/rules_closure/archive/dbb96841cc0a5fb2664c37822803b06dab20c7d1.tar.gz, https://github.com/bazelbuild/rules_closure/archive/dbb96841cc0a5fb2664c37822803b06dab20c7d1.tar.gz] to /home/janet/.cache/bazel/_bazel_janet/2d14dc1ff5782da202e00efcc3cd86bc/external/io_bazel_rules_closure/dbb96841cc0a5fb2664c37822803b06dab20c7d1.tar.gz: All mirrors are down:  [Unable to tunnel through proxy. Proxy returns "HTTP/1.0 407 Proxy Authentication Required"]
ERROR: error loading package '': Encountered error while reading extension file 'closure/defs.bzl': no such package '@io_bazel_rules_closure//closure': Error downloading [https://mirror.bazel.build/github.com/bazelbuild/rules_closure/archive/dbb96841cc0a5fb2664c37822803b06dab20c7d1.tar.gz, https://github.com/bazelbuild/rules_closure/archive/dbb96841cc0a5fb2664c37822803b06dab20c7d1.tar.gz] to /home/janet/.cache/bazel/_bazel_janet/2d14dc1ff5782da202e00efcc3cd86bc/external/io_bazel_rules_closure/dbb96841cc0a5fb2664c37822803b06dab20c7d1.tar.gz: All mirrors are down:  [Unable to tunnel through proxy. Proxy returns "HTTP/1.0 407 Proxy Authentication Required"]
INFO: Elapsed time: 57.573s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
FAILED: Build did NOT complete successfully (0 packages loaded)

However, I can use wget and curl to download those two packages. 1.https://mirror.bazel.build/github.com/bazelbuild/rules_closure/archive/dbb96841cc0a5fb2664c37822803b06dab20c7d1.tar.gz 2.https://github.com/bazelbuild/rules_closure/archive/dbb96841cc0a5fb2664c37822803b06dab20c7d1.tar.gz Since the file is able to be downloaded on my machine, I think the proxy settings are fine.

A workable workaround method is change the http_archive to local_repository. However, the build process still stopped because tensorflow apply their rule (tf_http_archive) to fetch other files in the following step. It's not reasonable to download all the file manually and replace all the `tf_http_archive' in each file.

Feature requests: what underlying problem are you trying to solve with this feature?

Is it possible to make bazel download files through proxy?

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Please download tensorflow and build it behind proxy.

What operating system are you running Bazel on?

Ubuntu 16.04.4 LTS

What's the output of `bazel info release`?

release 0.16.1

What's the output of `git remote get-url origin ; git rev-parse master ; git rev-parse HEAD` ?

https://github.com/tensorflow/tensorflow.git e18f84a394bcbde62b344a3b32e8d8fd248fea58 e18f84a394bcbde62b344a3b32e8d8fd248fea58

Have you found anything relevant by searching the web?

likely issue raised before: https://github.com/bazelbuild/bazel/issues/587#issuecomment-412531604

after applied the settings still not work

Question raised on stackoverflow: https://stackoverflow.com/questions/52311486/bazel-build-behind-proxy

dslomov commented 6 years ago

@aehlig why may we fail to authenticate to a proxy?

NearLinHere commented 5 years ago

@aehlig why may we fail to authenticate to a proxy?

Hello Aehlig, Would you please help us with this issue?

aehlig commented 5 years ago

Would you please help us with this issue?

A work around is to fetch the files manually and put them in a directory specified via the --distdir option.

aehlig commented 5 years ago

@aehlig why may we fail to authenticate to a proxy?

Most likely because the HttpDownloader is not aware of the particular authentication mechanism. However the downloader is a complex piece of code not written by me, so I did not yet have time to verify which authentication meachanisms are supported.

rjmccabe3701 commented 5 years ago

This is my workaround (when attempting to build the tensorflow_serving repo):

Repeatedly try the bazel build and capture the packages it cannot download

tools/run_in_docker.sh bazel build --distdir /src/tools/dl --remote_timeout=2 \
--keep_going -c opt tensorflow_serving/... 2>&1 | tee build_failure.log

Then run this python script to download the packages:

import re
import subprocess
import os
dl_dir='tools/dl'
p = re.compile('Error downloading \[([^\]]*)')
downloaded = set()
with open('build_failure.log', 'r') as f:
   os.chdir(dl_dir)
   for line in f.readlines():
      if not 'Error downloading' in line:
         continue
      m = p.search(line)
      if not m:
         print('error finding urls in ', line)
         continue
      urls = m.group(1).split(',')
      file_name = urls[0].split('/')[-1]
      if file_name in downloaded:
         continue
      print('downloading file ', file_name)
      for url in urls:
         print('   trying ', url)
         proc = subprocess.Popen('wget ' + url, stdout=subprocess.DEVNULL,
               stderr=subprocess.DEVNULL, shell=True)
         proc.communicate()
         if proc.returncode == 0:
            print('OK')
            downloaded.add(file_name)
            break
         else:
            print('FAILED')

This is incredibly annoying, I would be grateful for whomever fixes this.

pcj commented 5 years ago

I've also seen ?similar issues building behind a proxy (bazel build not working on an internal jenkins). We reasoned (without much evidence) that perhaps the proxy was overwhelmed by what bazel was trying to do. I've wanted to experiment with bazel fetch --loading_phase_threads={SOME_VALUE} or something to try and slow bazel down a bit to see if that helps but have not had time to do so.

At the very least (if I'm interpreting it correctly), your script shows that this issue is not about authentication, but failure to download some of the dependencies. Are you observing that you have to wget all dependencies or just some of them? If the latter, is the list of dependencies for wget the same every time, or is it different?

Hope also this issue get solved, because this issue is affecting socialization/acceptance of bazel within my organization.

rjmccabe3701 commented 5 years ago

@pcj My organization has proxy password authentication requirements that are annoying to workaround. Every bazel package fails to download. I've had similar issues with tools developed by companies with the luxury of having transparent proxy.

NearLinHere commented 5 years ago

Hi @pcj, My script fails at the same line every time so that I can't tell you that if they will download all the files or just some of them. If you need more information, please feel free to ask. I am also hoping to see this issue got solved.

the7threvival commented 5 years ago

There have been various requests over various months, if not years now, for Bazel to allow build behind a proxy. Will anything be done about this? This is really frustrating.

dviettu134 commented 4 years ago

Would you please help us with this issue?

A work around is to fetch the files manually and put them in a directory specified via the --distdir option.

@aehlig I'm building TensorFlow version 1.9, which require bazel 0.11.0 (https://www.tensorflow.org/install/source#tested_build_configurations) But it seems that --distdir is not available for bazel 0.11.0 yet, do you have any work around of this problem for that version?

meisterT commented 4 years ago

@dviettu134 Can't you use a more modern Tensorflow?

General question for the people that are frustrated to build Bazel behind a proxy. Did you configure the proxy like described in https://docs.bazel.build/versions/master/external.html#using-proxies ?

verdurin commented 3 years ago

@meisterT I'm seeing this exact problem, and have the environment variables set as recommended by the link.

This is when trying to build TensorFlow 2.3.1

verdurin commented 3 years ago

Here are the logs:

INFO: Found applicable config definition build:v2 in file /dev/shm/TensorFlow/2.3.1/fosscuda-2019b-Python-3.7.4/TensorFlow/tensorflow-2.3.1/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:cuda in file /dev/shm/TensorFlow/2.3.1/fosscuda-2019b-Python-3.7.4/TensorFlow/tensorflow-2.3.1/.bazelrc: --config=using_cuda --define=using_cuda_nvcc=true
INFO: Found applicable config definition build:using_cuda in file /dev/shm/TensorFlow/2.3.1/fosscuda-2019b-Python-3.7.4/TensorFlow/tensorflow-2.3.1/.bazelrc: --define=using_cuda=true --action_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain
INFO: Found applicable config definition build:opt in file /dev/shm/TensorFlow/2.3.1/fosscuda-2019b-Python-3.7.4/TensorFlow/tensorflow-2.3.1/.tf_configure.bazelrc: --copt=-O2 --copt=-ftree-vectorize --copt=-march=native --copt=-fno-math-errno --copt=-fPIC --host_copt=-march=native --define with_default_optimizations=true
INFO: Found applicable config definition build:noaws in file /dev/shm/TensorFlow/2.3.1/fosscuda-2019b-Python-3.7.4/TensorFlow/tensorflow-2.3.1/.bazelrc: --define=no_aws_support=true
INFO: Found applicable config definition build:mkl in file /dev/shm/TensorFlow/2.3.1/fosscuda-2019b-Python-3.7.4/TensorFlow/tensorflow-2.3.1/.bazelrc: --define=build_with_mkl=true --define=enable_mkl=true --define=tensorflow_mkldnn_contraction_kernel=0 --define=build_with_mkl_dnn_v1_only=true -c opt
INFO: Found applicable config definition build:linux in file /dev/shm/TensorFlow/2.3.1/fosscuda-2019b-Python-3.7.4/TensorFlow/tensorflow-2.3.1/.bazelrc: --copt=-w --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --cxxopt=-std=c++14 --host_cxxopt=-std=c++14 --config=dynamic_kernels
INFO: Found applicable config definition build:dynamic_kernels in file /dev/shm/TensorFlow/2.3.1/fosscuda-2019b-Python-3.7.4/TensorFlow/tensorflow-2.3.1/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
Loading:
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
INFO: Repository io_bazel_rules_closure instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule http_archive defined at:
  /dev/shm/TensorFlow/2.3.1/fosscuda-2019b-Python-3.7.4/tmpIk97Hr-bazel-tf/output_base/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/bazelbuild/rules_closure/archive/308b05b2419edb5c8ee0471b67a40403df940149.tar.gz failed: class java.io.IOException Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required"
WARNING: Download from https://github.com/bazelbuild/rules_closure/archive/308b05b2419edb5c8ee0471b67a40403df940149.tar.gz failed: class java.io.IOException Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required"
ERROR: An error occurred during the fetch of repository 'io_bazel_rules_closure':
   java.io.IOException: Error downloading [https://storage.googleapis.com/mirror.tensorflow.org/github.com/bazelbuild/rules_closure/archive/308b05b2419edb5c8ee0471b67a40403df940149.tar.gz, https://github.com/bazelbuild/rules_closure/archive/308b05b2419edb5c8ee0471b67a40403df940149.tar.gz] to /dev/shm/TensorFlow/2.3.1/fosscuda-2019b-Python-3.7.4/tmpIk97Hr-bazel-tf/output_base/external/io_bazel_rules_closure/308b05b2419edb5c8ee0471b67a40403df940149.tar.gz: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required"
ERROR: no such package '@io_bazel_rules_closure//closure': java.io.IOException: Error downloading [https://storage.googleapis.com/mirror.tensorflow.org/github.com/bazelbuild/rules_closure/archive/308b05b2419edb5c8ee0471b67a40403df940149.tar.gz, https://github.com/bazelbuild/rules_closure/archive/308b05b2419edb5c8ee0471b67a40403df940149.tar.gz] to /dev/shm/TensorFlow/2.3.1/fosscuda-2019b-Python-3.7.4/tmpIk97Hr-bazel-tf/output_base/external/io_bazel_rules_closure/308b05b2419edb5c8ee0471b67a40403df940149.tar.gz: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required"
INFO: Elapsed time: 27.701s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
FAILED: Build did NOT complete successfully (0 packages loaded)

meisterT commented 3 years ago

@Wyverald do you whether the httpdownloader respects authentication as configured via HTTP_PROXY (and similar)? That whole issue will become moot after https://docs.google.com/document/d/1moQfNcEIttsk6vYanNKIy3ZuK53hQUFq1b1r0rmsYVg/edit will be implemented.

qiuxiafei commented 2 years ago

Any update with this issue? It's 2022 now and still frustrating.

nomanfound commented 2 years ago

squid can convert an authenticated proxy into a normal one without anthentication, which can be used by bazel fetch.

matawed commented 2 years ago

Is this really still an issue? According to https://github.com/bazelbuild/bazel/blob/6578b5b808eac6aa22aa325a38e59d364ac2fea3/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/ProxyHelper.java authenticated proxies set by env vars should be supported, shouldn't it?

michael-etzkorn commented 1 year ago

Does not work for socks either #15652

bazelbuild / bazel