Build: failed to connect to some link, how to use manually downloaded file?

siahuat0727 commented 4 years ago

Hi, my server can't connect to https://github.com/ApolloAuto/apollo-contrib/releases/download/v3.0.0/plat-sw-3.0.0.1.zip when executing ./apollo.sh build_opt_gpu. But I can manually download this zip file and upload it to the server. How can I modify the script or where to put this zip file so that it will use the file I manually uploaded instead of downloading from the link? Or are there any better solutions?

The command corresponding to this error is:

https://github.com/ApolloAuto/apollo/blob/a7f171f4e59c44afa5695a880279c66679d3d43a/apollo.sh#L147

The output is as follows:

System check passed. Build continue ...
[WARNING] ESD CAN library supplied by ESD Electronics does not exist. If you need ESD CAN, please refer to third_party/can_card_library/esd_can/README.md.
Running build under GPU mode. GPU is required to run the build.
[INFO] Start building, please wait ...
INFO: Reading 'startup' options from /apollo/tools/bazel.rc: --batch_cpu_scheduling --host_jvm_args=-XX:-UseParallelGC
INFO: Reading 'startup' options from /apollo/tools/bazel.rc: --batch_cpu_scheduling --host_jvm_args=-XX:-UseParallelGC
[INFO] Building on x86_64...
[INFO] Building with --jobs=80 --ram_utilization_factor 80 for x86_64
INFO: Reading 'startup' options from /apollo/tools/bazel.rc: --batch_cpu_scheduling --host_jvm_args=-XX:-UseParallelGC
INFO: Reading 'startup' options from /apollo/tools/bazel.rc: --batch_cpu_scheduling --host_jvm_args=-XX:-UseParallelGC
ERROR: (04-17 06:33:15.848) /apollo/modules/drivers/camera/BUILD:49:1: no such package '@adv_plat//': Error downloading [https://github.com/ApolloAuto/apollo-contrib/releases/download/v3.0.0/plat-sw-3.0.0.1.zip] to /home/user/cache/bazel/_bazel_user/540135163923dd7d5820f3ee4b306b32/external/adv_plat/plat-sw-3.0.0.1.zip: Connection refused (Connection refused) and referenced by '//modules/drivers/camera:camera'.
ERROR: (04-17 06:33:15.970) Analysis of target '//modules/drivers/camera:libcamera_component.so' failed; build aborted.
INFO: (04-17 06:33:15.970) Elapsed time: 15.525s
============================
[ERROR] Build failed!
[INFO] Took 16 seconds
============================

muzhou-yu commented 4 years ago

good question

daohu527 commented 4 years ago

You can try in file "apollo\WORKSPACE.in", and change

#adv_plat
new_http_archive(
    name = "adv_plat",
    build_file = "third_party/adv_plat.BUILD",
    sha256 = "0a58dadab924b520d5b5a58ef82fc0f76c2aa4feaaabd49ec9873228c125d513",
    url = "https://github.com/ApolloAuto/apollo-contrib/releases/download/v3.0.0/plat-sw-3.0.0.1.zip",
)

to

now

change in "apollo\WORKSPACE.in"

#adv_plat
new_local_repository(
name = "adv_plat",
build_file = "third_party/adv_plat.BUILD",
path = "/usr/local/apollo/adv_plat",
)

Put the download file in "/usr/local/apollo/adv_plat", unzip it

siahuat0727 commented 4 years ago

@daohu527 Hi, thanks a lot! But this raises another problem.

I run the following commands:

$ sudo mkdir -p  /usr/local/apollo/adv_plat
$ sudo mv plat-sw-3.0.0.1.zip  /usr/local/apollo/adv_plat
$ cd /usr/local/apollo/adv_plat
$ sudo unzip plat-sw-3.0.0.1.zip
Archive:  plat-sw-3.0.0.1.zip
   creating: bin/
  inflating: bin/adv_trigger         
   creating: include/
  inflating: include/bcan.h          
  inflating: include/adv_trigger.h   
   creating: include/linux/
  inflating: include/linux/bcan_defs.h  
  inflating: include/linux/zynq_api.h  
   creating: lib/
  inflating: lib/libadv_trigger.so.3.0.0.1  
  inflating: lib/libadv_trigger.a    
  inflating: lib/libadv_bcan.a       
    linking: lib/libadv_bcan.so.1    -> libadv_bcan.so.3.0.0.1 
    linking: lib/libadv_trigger.so.1  -> libadv_trigger.so.3.0.0.1 
  inflating: lib/libadv_bcan.so.3.0.0.1  
finishing deferred symbolic links:
  lib/libadv_bcan.so.1   -> libadv_bcan.so.3.0.0.1
  lib/libadv_trigger.so.1 -> libadv_trigger.so.3.0.0.1

Then run build again

in_dev_docker$ ./apollo.sh build
System check passed. Build continue ...
[WARNING] ESD CAN library supplied by ESD Electronics does not exist. If you need ESD CAN, please refer to third_party/can_card_library/esd_can/README.md.
Running build under GPU mode. GPU is required to run the build.
[INFO] Start building, please wait ...
INFO: Reading 'startup' options from /apollo/tools/bazel.rc: --batch_cpu_scheduling --host_jvm_args=-XX:-UseParallelGC
INFO: Reading 'startup' options from /apollo/tools/bazel.rc: --batch_cpu_scheduling --host_jvm_args=-XX:-UseParallelGC
[INFO] Building on x86_64...
[INFO] Building with --jobs=80 --ram_utilization_factor 80 for x86_64
INFO: Reading 'startup' options from /apollo/tools/bazel.rc: --batch_cpu_scheduling --host_jvm_args=-XX:-UseParallelGC
INFO: Reading 'startup' options from /apollo/tools/bazel.rc: --batch_cpu_scheduling --host_jvm_args=-XX:-UseParallelGC
INFO: (04-18 14:24:54.337) Found 4479 targets...
ERROR: (04-18 14:24:54.686) missing input file '@adv_plat//:include/linux/zynq_api.h'.
ERROR: (04-18 14:24:54.699) /home/user/.cache/bazel/_bazel_user/540135163923dd7d5820f3ee4b306b32/external/adv_plat/BUILD.bazel:5:1: @adv_plat//:adv_plat: missing input file '@adv_plat//:include/linux/zynq_api.h'.
ERROR: (04-18 14:24:54.719) /home/user/.cache/bazel/_bazel_user/540135163923dd7d5820f3ee4b306b32/external/adv_plat/BUILD.bazel:5:1 1 input file(s) do not exist.
INFO: (04-18 14:24:54.720) Elapsed time: 4.303s, Critical Path: 0.11s
============================
[ERROR] Build failed!
[INFO] Took 6 seconds
============================

I thought that's because I didn't have read permission for some files (e.g. files in adv_plat/include/linux/), so I changed the permission:

$ sudo chmod -R +r /usr/local/apollo

But it still can't build. (Similar error, not always the same missing input file)

Did I do something wrong? Have I provided enough information?

storypku commented 4 years ago

There is an option "--distdir=/path/to/predownloaded/packages" for bazel commands. Maybe you can have a try.

daohu527 commented 4 years ago

@siahuat0727

As @storypku metioned you can mofify the file https://github.com/ApolloAuto/apollo/blob/a7f171f4e59c44afa5695a880279c66679d3d43a/apollo.sh#L147

to (both 147 and 159 line)

 bazel build $JOB_ARG $DEFINES -c $@ $BUILD_TARGETS --distdir=/media/download/bazel

Be sure to check the sha256 of download package is equal in the "WORKSPACE.in"

sha256sum plat-sw-3.0.0.1.zip

https://github.com/ApolloAuto/apollo/blob/a7f171f4e59c44afa5695a880279c66679d3d43a/WORKSPACE.in#L248-L253

I will check the new_local_repository method later.

siahuat0727 commented 4 years ago

@storypku @daohu527 Thanks! !

Now comes:

ERROR: Unrecognized option: --distdir=/media/download/bazel

The version of bazel installed in the docker is 1.0.1.

Is it the WORKSPACE.in should be revert to the original version? And, where should I put the downloaded zip file? Since I have no idea why distdir is set to /media/download/bazel, how can it find the downloaded file?

siahuat0727 commented 4 years ago

@daohu527 Sorry, the version looks like 0.5.3.

in_dev_docker:/apollo$ bazel version
INFO: Reading 'startup' options from /apollo/tools/bazel.rc: --batch_cpu_scheduling --host_jvm_args=-XX:-UseParallelGC
INFO: Reading 'startup' options from /apollo/tools/bazel.rc: --batch_cpu_scheduling --host_jvm_args=-XX:-UseParallelGC
Build label: 0.5.3
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Fri Jul 28 08:34:59 2017 (1501230899)
Build timestamp: 1501230899
Build timestamp as int: 1501230899

But if I try to upgrade with apt, the output is as follows:

in_dev_docker:/apollo$ sudo apt install  bazel --upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
bazel is already the newest version (1.0.1).
The following packages were automatically installed and are no longer required:
  libgail-common libgail18 libgtk2.0-0 libgtk2.0-bin libgtk2.0-common
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 144 not upgraded.

daohu527 commented 4 years ago

Here is a question about the problem, If the bazel version update will cause other problems, I will check the problem and provide a solution later

siahuat0727 commented 4 years ago

@daohu527

Hi, I got into a lot of trouble...

First, if I apt-key add follow step 1 here and then sudo apt update, I will get connection timed out.

Then, I maybe remove the key, sudo apt update and sudo apt install bazel --upgrade -f (Sorry for not being able to describe it accurately)

Now if I execute

in_dev_docker:/apollo$ sudo apt install bazel --upgrade
Building dependency tree       
Reading state information... Done
bazel is already the newest version (3.0.0).
0 upgraded, 0 newly installed, 0 to remove and 5 not upgraded.

It shows that bazel is already the newest version 3.0.0. But in_dev_docker:/apollo$ bazel version returns 0.5.3.

If execute

in_dev_docker:/apollo$ sudo apt upgrade bazel
Reading package lists... Done
Building dependency tree       
Reading state information... Done
bazel is already the newest version (3.0.0).
Calculating upgrade... Done
The following packages have been kept back:
  libcudnn7 libcudnn7-dev libnccl-dev libnccl2
The following packages will be upgraded:
  tzdata
1 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
Need to get 190 kB of archives.
After this operation, 3072 B of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 tzdata all 2019c-0ubuntu0.18.04 [190 kB]
Fetched 190 kB in 2s (109 kB/s) 
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 88906 files and directories currently installed.)
Preparing to unpack .../tzdata_2019c-0ubuntu0.18.04_all.deb ...
Unpacking tzdata (2019c-0ubuntu0.18.04) over (2018i-0ubuntu0.18.04) ...
dpkg: error processing archive /var/cache/apt/archives/tzdata_2019c-0ubuntu0.18.04_all.deb (--unpack):
 unable to make backup link of './usr/share/zoneinfo/UTC' before installing new version: Invalid cross-device link
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)
debconf: falling back to frontend: Readline
Errors were encountered while processing:
 /var/cache/apt/archives/tzdata_2019c-0ubuntu0.18.04_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

I tried several solutions from some posts, likes

in_dev_docker:/apollo$ sudo apt autoremove
in_dev_docker:/apollo$ sudo apt clean
in_dev_docker:/apollo$ sudo dpkg -i --force-overwrite /var/.../..._all.deb
in_dev_docker:/apollo$ sudo dpkg --configure -a
in_dev_docker:/apollo$ sudo apt -o Dpkg::Options::="--force-overwrite" upgrade bazel

All failed. The file /var/cache/apt/archives/tzdata_2019c-0ubuntu0.18.04_all.deb doesn't exist.

Then, I tried to build from source follows bazel doc

in_dev_docker:/apollo$ wget https://codeload.github.com/bazelbuild/bazel/zip/3.0.0
in_dev_docker:/apollo$ unzip "3.0.0"
in_dev_docker:/apollo$ cd bazel-3.0.0/
in_dev_docker:/apollo$ env EXTRA_BAZEL_ARGS="--host_javabase=@local_jdk//:jdk" bash ./compile.sh
🍃  Building Bazel from scratchfind: 'derived/jars': No such file or directory

ERROR: Must specify PROTOC if not bootstrapping from the distribution artifact

--------------------------------------------------------------------------------
NOTE: This failure is likely occuring if you are trying to bootstrap bazel from
a developer checkout. Those checkouts do not include the generated output of
the protoc compiler (as we prefer not to version generated files).

* To build a developer version of bazel, do

    bazel build //src:bazel

* To bootstrap your first bazel binary, please download a dist archive from our
  release page at https://github.com/bazelbuild/bazel/releases and run
  compile.sh on the unpacked archive.

The full install instructions to install a release version of bazel can be found
at https://docs.bazel.build/install-compile-source.html
For a rationale, why the bootstrap process is organized in this way, see
https://bazel.build/designs/2016/10/11/distribution-artifact.html
--------------------------------------------------------------------------------

How can I do now? Sorry for the trouble...

daohu527 commented 4 years ago

@siahuat0727 I found anyother method, pls ignore the above method and follow this steps. I tried compiling and it seems works.

modify the WORKSPACE.in

new_http_archive(
name = "adv_plat",
build_file = "third_party/adv_plat.BUILD",
sha256 = "0a58dadab924b520d5b5a58ef82fc0f76c2aa4feaaabd49ec9873228c125d513",
url = "file:///home/tmp/plat-sw-3.0.0.1.zip",
)

download the file and put it in docker "/home/tmp"
Then run ./apollo.sh build

siahuat0727 commented 4 years ago

@daohu527 It works, thanks!

Indeed, after puttting it in docker /home/tmp, I ran

in_dev_docker:/apollo$ exit
$ ./docker/scripts/dev_start.sh
$ ./docker/scripts/dev_into.sh
in_dev_docker:/apollo$ ./apollo.sh build

Just a reference to someone who get into the same situation later.

Again, thank you very much!

daohu527 commented 4 years ago

I'm glad the problem is solved, and I'll add some to the first method.

new_local_repository

The adv package is already install by "apollo\docker\build\installers\install_adv_plat.sh" when build the docker file, the path is "/usr/local/apollo/adv_plat". But the version is older.
So bazel repull the package through network and use the new version. here is a step to build offline.

modify the WORKSPACE.in

#adv_plat
new_local_repository(
name = "adv_plat",
build_file = "third_party/adv_plat.BUILD",
path = "/usr/local/apollo/adv_plat",
)

download the file and unzip it in "/usr/local/apollo/adv_plat"(remove the older version),
```
sudo chmod 755 -R /usr/local/apollo/adv_plat
```
then use "./apollo.sh build"

new_http_archive

modify the WORKSPACE.in

new_http_archive(
name = "adv_plat",
build_file = "third_party/adv_plat.BUILD",
sha256 = "0a58dadab924b520d5b5a58ef82fc0f76c2aa4feaaabd49ec9873228c125d513",
url = "file:///home/tmp/plat-sw-3.0.0.1.zip",
)

download the file and put it in docker "/home/tmp"
Then run ./apollo.sh build

ApolloAuto / apollo

Build: failed to connect to some link, how to use manually downloaded file? #11062

now

new_local_repository

new_http_archive