bazelbuild / bazel-toolchains

Repository that hosts Bazel toolchain configs for remote execution and related support tools.
Apache License 2.0
186 stars 96 forks source link

rbe_ubuntu_20.04 based docker image? #870

Open davido opened 4 years ago

davido commented 4 years ago

On 23.04.202, Ubuntu 20.04 was released: [1]. What is the ETA for the Ubuntu 20.04 based RBE image?

The reason I'm asking is that for Gerrit Code Review project we have intergration tests based on recent Git package release (to test Gerrit with Git wire protocol v2). And Git version is outdated in 16.04 and 18.04 Ubuntu releases.

Yes, I know that we could easily build our own custom RBE docker image and publish it somewhere. E.g. on https://gcr.io or on https://hub.docker.com registries.

smukherj1 commented 4 years ago

We don't currently have plans for an Ubuntu 20.04 image. We would first need an Ubuntu 20.04 image to use as the base image from the managed base images maintained in https://github.com/GoogleContainerTools/base-images-docker.

After that, I'm open to PRs in https://github.com/GoogleCloudPlatform/layer-definitions and https://github.com/GoogleCloudPlatform/container-definitions adding the image.

davido commented 4 years ago

@smukherj1 Thanks for clarifying. I can look into sending a PR once the prerequisites are met.

davido commented 3 years ago

@philwo Are you working on it?

I also tried to add custom docker image from Bazel in Gerrit, in patch set 4 of this change: [1] but I have not managed to make it work. The error was:

/home/davido/.cache/bazel/_bazel_davido/27a001f4182820ef315d8d2d4f1edafe/external/remote_java_tools/BUILD:273:11: Compiling java_tools/src/main/cpp/util/file_posix.cc failed: (Exit 34): INVALID_ARGUMENT: Invalid arguments: 
  "command.ValidateSpec": Invalid spec - docker container must be specified

The reason I am asking: Gerrit switched to Java language level 11 recently, but unfortunately, RBE wasn't switched yet, so on gerrit@HEAD:

  $ bazel build --config=remote --remote_instance_name=<project> java/com/google/gerrit/common:server
 [...]
INFO: Found 1 target...
Target //java/com/google/gerrit/common:server up-to-date:
  bazel-bin/java/com/google/gerrit/common/libserver.jar
INFO: Elapsed time: 33.154s, Critical Path: 23.67s
INFO: 253 processes: 236 remote cache hit, 16 internal, 1 remote.
INFO: Build completed successfully, 253 total actions

  $ javap -v -cp bazel-bin/java/com/google/gerrit/common/libserver.jar com.google.gerrit.common.Version | grep major
  major version: 52

Local build works as expected and produce Java language level 11:

  $ bazel build java/com/google/gerrit/common:server
  [...]
INFO: Found 1 target...
Target //java/com/google/gerrit/common:server up-to-date:
  bazel-bin/java/com/google/gerrit/common/libserver.jar
INFO: Elapsed time: 7.942s, Critical Path: 5.34s
INFO: 253 processes: 236 remote cache hit, 16 internal, 1 linux-sandbox.
INFO: Build completed successfully, 253 total actions

  $ javap -v -cp bazel-bin/java/com/google/gerrit/common/libserver.jar com.google.gerrit.common.Version | grep major
  major version: 55

[1] https://gerrit-review.googlesource.com/c/gerrit/+/291864/4

davido commented 3 years ago

Update: I figured out how to build with JDK 11 and even produce major bytecode version 55 with RBE: [1].

[1] https://gerrit-review.googlesource.com/c/gerrit/+/291943

davido commented 2 years ago

Update: it seems that Ubuntu20.04 based docker image was added to: [1].

[1] https://github.com/GoogleContainerTools/base-images-docker/blob/master/ubuntu2004/

davido commented 2 years ago

I also figured, that we could build Ubuntu 20.04 compatible image by just saying:

Dockerfile

FROM gcr.io/cloud-marketplace/google/ubuntu2004:latest

RUN apt-get -y update && \
    apt-get -y install git && \
    apt-get -y install golang && \
    apt-get -y install openjdk-17-jdk openjdk-17-jre openjdk-17-jdk-headless && \
    apt-get -y install unzip zip && \
    apt-get -y install python3 && \
    apt-get -y install wget && \
    apt-get -y install vim && \
    apt-get clean

That way built Docker image works as expected locally (bazel build headless). I have not tried to use it on RBE, though.

bduffany commented 2 years ago

@smukherj1 I am interested in contributing to this -- one thing I don't understand is how to build ubuntu20.04-specific releases of clang. For example, here: https://github.com/GoogleCloudPlatform/layer-definitions/blob/ef7ab83fe8f7813ceeef5bb8dc283a08bed47151/layers/ubuntu1804/clang/deps.bzl#L29-L34

How is the object at ["https://storage.googleapis.com/clang-builds-stable/clang-ubuntu18_04/clang_r" + CLANG_INSTALLER.revision + ".tar.gz"] built and deployed?

More generally, since I will have the same questions for the Java layers, etc. -- is there any documentation that can be shared about how the source files in https://github.com/GoogleCloudPlatform/layer-definitions are generated?

bduffany commented 1 year ago

Friendly ping on this, since ubuntu 18.04 includes some dependencies that are out of date. Are you still open to PRs here? If so I would love to contribute, but could use some help with the questions in my above comment.

steve-261370 commented 1 year ago

Just as another alternative solution here since this repo seems abandoned:

I found that bazel's own continuous build has built an ubuntu 20.04 image that you can use (and also a 22.04 one too).

The Dockerfile: https://github.com/bazelbuild/continuous-integration/blob/08fa4f8b1648acab1726e1354c3b56719cfba417/buildkite/docker/ubuntu2004/Dockerfile The custom cpp_env.json: https://github.com/bazelbuild/continuous-integration/blob/672777d1ae17d993a5ca3de3cb9a9e2fab6531ee/rbe-configs/cpp_env/ubuntu2004.json

Which you can use as such:

./rbe_configs_gen \
    --bazel_version=6.4.0 \
    --toolchain_container=gcr.io/bazel-public/ubuntu2004-java11@sha256:833b20afd7b624693160c8e9f8aafaee46657d29d32e61012256aa3d045d8662 \
    --output_src_root=/path/to/repo \
    --output_config_path=path/to/config-directory \
    --exec_os=linux \
    --target_os=linux \
    --cpp_env_json=ubuntu2004.json