search

bazelbuild / bazel

a fast, scalable, multi-language and extensible build system
https://bazel.build
Apache License 2.0
22.69k stars 3.98k forks source link

Embedded JDK ignores system cacerts #5741

Open kivoli opened 5 years ago

kivoli commented 5 years ago

Description of the problem / feature request:

Before Bazel 0.16 at least the Debian packages depended on the system jdk.

Depends: google-jdk | java8-sdk-headless | java8-jdk | java8-sdk | oracle-java8-installer, g++, zlib1g-dev, bash-completion

Bazel 0.16 apparently embeds the JDK which fundamentally changes the behaviour at least in regards to cacerts. The custom CA certificates that have been added to the system certificate store are of course missing in the embedded JDK’s cacerts:

ERROR: /project/BUILD:13:1: no such package '@some_stuff//': Error cloning repository: https://gitlab.example.com/example/some_stuff.git: Secure connection to https://gitlab.example.com/example/some_stuff.git could not be stablished because of SSL problems caused by https://gitlab.example.com/example/some_stuff.git: Secure connection to https://gitlab.example.com/example/some_stuff.git could not be stablished because of SSL problems caused by sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target caused by PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target caused by unable to find valid certification path to requested target and referenced by '//some_other_stuff:some_files'

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Set up a Vagrant box with Debian Stretch and install Bazel 0.16 from the APT repository (http://storage.googleapis.com/bazel-apt stable jdk1.8). Also install the CACert root CA

apt install ca-certificates
cd /usr/local/share/ca-certificates
wget http://www.cacert.org/certs/root.crt
update-ca-certificates

In a folder of your choosing, create

# WORKSPACE:
workspace(name = "test_case")
git_repository(
  name = 'cacert',
  remote = 'https://git.cacert.org/cacert.git',
  tag = 'v0.0.1',
)

and

# BUILD
load("@bazel_tools//tools/build_defs/pkg:pkg.bzl", "pkg_tar")
pkg_tar(
  name = "test_case",
  srcs = ["@cacert//:some_files"],
)

Note: This has nothing to do with CACert but they are the only public site I know that uses a certificate from CA that’s not in most default certificate bundles.

Now try to “build” the package.

bazel build //:all
Starting local Bazel server and connecting to it...
ERROR: /tmp/testcase/BUILD:3:1: no such package '@cacert//': Error cloning repository: https://git.cacert.org/cacert.git: Secure connection to https://git.cacert.org/cacert.git could not be stablished because of SSL problems caused by https://git.cacert.org/cacert.git: Secure connection to https://git.cacert.org/cacert.git could not be stablished because of SSL problems caused by sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target caused by PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target caused by unable to find valid certification path to requested target and referenced by '//:test_case'
ERROR: Analysis of target '//:test_case' failed; build aborted: no such package '@cacert//': Error cloning repository: https://git.cacert.org/cacert.git: Secure connection to https://git.cacert.org/cacert.git could not be stablished because of SSL problems caused by https://git.cacert.org/cacert.git: Secure connection to https://git.cacert.org/cacert.git could not be stablished because of SSL problems caused by sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target caused by PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target caused by unable to find valid certification path to requested target
INFO: Elapsed time: 1.811s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (5 packages loaded)

If you use Bazel 0.15 instead the error message will be different:

bazel build //:all
Starting local Bazel server and connecting to it...
........
ERROR: /mnt/containers/testcase/BUILD:3:1: no such package '@cacert//': Invalid Git repository URI: Invalid remote: origin and referenced by '//:test_case'
ERROR: Analysis of target '//:test_case' failed; build aborted: no such package '@cacert//': Invalid Git repository URI: Invalid remote: origin
INFO: Elapsed time: 1.873s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (5 packages loaded)

Note: If the CACert CA is not installed in the system the error message will be the same as with 0.16.

What operating system are you running Bazel on?

Debian 8 Stretch

What's the output of bazel info release?

release 0.16.0
# old:
release 0.15.0

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

N/A

Have you found anything relevant by searching the web?

!3915 tells me how to override the path to cacerts but it would be better if Bazel just uses the system’s cacerts instead. Personally I think this is a regression breaking backwards compatibility.

Any other information, logs, or outputs that you want to share?

0.16 definitely brings its own cacerts while 0.15 does not even have the jdk folder.

# 0.16
find ~/.cache/bazel -name cacerts
/home/vagrant/.cache/bazel/_bazel_vagrant/install/c25ea2c3043bcba07b93dde10595066c/_embedded_binaries/embedded_tools/jdk/lib/security/cacerts
ls /home/vagrant/.cache/bazel/_bazel_vagrant/install/c25ea2c3043bcba07b93dde10595066c/_embedded_binaries/embedded_tools
jdk  platforms  src  third_party  tools  WORKSPACE
# 0.15
ls /home/vagrant/.cache/bazel/_bazel_vagrant/install/ce085f519b017357185750fe457b4648/_embedded_binaries/embedded_tools/
platforms/   src/         third_party/ tools/       WORKSPACE
kivoli commented 5 years ago

Replacing the cacerts file with a symlink as a work around doesn’t work either:

$ /home/vagrant/.cache/bazel/_bazel_vagrant/install/c25ea2c3043bcba07b93dde10595066c/_embedded_binaries/embedded_tools/jdk/lib/security/cacerts -> /etc/ssl/certs/java/cacerts
$ bazel build //:all
FATAL: corrupt installation: file '/home/vagrant/.cache/bazel/_bazel_vagrant/install/c25ea2c3043bcba07b93dde10595066c/_embedded_binaries/embedded_tools/jdk/lib/security/cacerts' is missing or modified.  Please remove '/home/vagrant/.cache/bazel/_bazel_vagrant/install/c25ea2c3043bcba07b93dde10595066c' and try again.

Also found this announcement from last April that states:

Note:

Homebrew and debian packages do not contain the embedded JDK. This change only affects the shell installers.

Is the embedded JDK a packaging bug?

gertvdijk commented 5 years ago

Also found this announcement from last April that states:

That's from April 2017, quite a long while ago.

Is the embedded JDK a packaging bug?

Could be that it's embedded unintentionally as it's not mentioned in the 0.16.0 release notes. Anyway, you might be interested in related issue #5753.

kivoli commented 5 years ago

@gertvdijk Indeed — on both accounts. Nevertheless the announcement declared the intent back then. I have not yet found an indicator that the intent has changed. So together with #5753 I guess it’s up for debate what route should be taken.

Personally I would prefer following the Debian way (not embedding dependencies), which, of course, would also help with my requirement of using the system cacerts.

cushon commented 5 years ago

This issue describes behaviour of the OpenJDK build that Bazel is using, I'm not sure there's a good fix in Bazel itself.

If you don't want to use the embedded JDK, you can override it by setting a custom --host_javabase.

The issue with https://git.cacert.org/cacert.git reproduces independently of Bazel, for any OpenJDK build from 8 through 11. (I didn't try earlier versions.)

Repro:

import java.io.*;
import java.net.*;
import javax.net.ssl.*;

public class UrlConnect {
  public static void main(String[] args) {
    String[] defaultArgs = {"https://git.cacert.org/cacert.git"};

    if (args.length == 0) {
      args = defaultArgs;
    }
    for (String url : args) {
      try (InputStream stream = new URL(url).openStream()) {
        System.out.println("OK: " + url);
      } catch (SSLHandshakeException ex) {
        // This is bad
        System.err.println("ERROR: " + url + ": " + ex.toString());
      } catch (IOException ex) {
        System.out.println("Warning: " + url + ": " + ex.toString());
      }
    }
  }
}
$ java -version
java version "9.0.4"
$ javac UrlConnect.java
$ java UrlConnect 
ERROR: https://git.cacert.org/cacert.git: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Note that there were some changes related to cacerts starting in 9: http://www.oracle.com/technetwork/java/javase/9all-relnotes-3704433.html#JDK-8189131

The OpenJDK 9 binary for Linux x64 contains an empty cacerts keystore. This prevents TLS connections from being established because there are no Trusted Root Certificate Authorities installed. As a workaround for OpenJDK 9 binaries, users had to set the javax.net.ssl.trustStore System Property to use a different keystore.

"JEP 319: Root Certificates" [1] addresses this problem by populating the cacerts keystore with a set of root certificates issued by the CAs of Oracle's Java SE Root CA Program. As a prerequisite, each CA must sign the Oracle Contributor Agreement (OCA) http://www.oracle.com/technetwork/community/oca-486395.html, or an equivalent agreement, to grant Oracle the right to open-source their certificates.

[1] JDK-8191486

cushon commented 5 years ago

cc @philwo

related: https://groups.google.com/forum/#!topic/bazel-discuss/13uPDObyfQg

kivoli commented 5 years ago

@cushon Please note that I do not use git.cacerts.org. You may have missed my note in the description:

Note: This has nothing to do with CACert but they are the only public site I know that uses a certificate from [a] CA that’s not in most default certificate bundles.

Our problem is with a private company CA.

The reason this is an issue opened with Bazel is because we are using their official Debian packages and these suddenly behave differently. Before 0.16 these packages used the JDK provided by Debian which integrate with the ca-certificates package managing system level certificates including private or personal CAs via /usr/local/share/ca-certificates and update-ca-certificates, generating /etc/ssl/certs/java/cacerts which in turn is used by Debian’s JDKs and JREs. This is exactly the way we deploy the company CA and it has been working perfectly fine with Bazel, too, so far. Although an announcement from a year ago may be considered outdated by some I’m missing an announcement to the opposite.

Last but not least I understand that there are work arounds but I still have to ask the question if this is a change of the stance taken by the project or, in fact, a bug in packaging. Personally I would obviously prefer for a Debian package to not bring it’s on JDK to the party and break the nice integration features along the way. If the project however thinks that times have changed, I kindly ask it to document the change(s) and ideally also provide work arounds for such issues as part of the official documentation.

You may ask why I don’t just use the work around and be done with it? Since it means hard coding a path in several locations instead of using an “automagic” solution managed by the distribution. Who knows what other stuff will break down the line because our setup is suddenly “non-standard”?

Thank you for linking the Google Groups discussion, matthew indeed has the same issue.

buchgr commented 5 years ago

Personally I would obviously prefer for a Debian package to not bring it’s on JDK to the party and break the nice integration features along the way.

Rest assured that the Bazel team feels the same. This will be a temporary measure. Java has recently switched to a six month release schedule and we found that many older but still popular distributions don't ship with a sufficiently recent JDK that we decided to go the embedded way for now.

kivoli commented 5 years ago

@buchgr It’s of course nice to hear that the Bazel shares that sentiment but now I’m confused as to what happens next. Even if it is a “temporary measure” it still means it breaks this integration “for now”. Also I would assume that there always will be popular distributions with outdated dependencies.

Or is one of the ideas to have separate repositories per distribution? Actually I just realised that the APT source “component” we use right now is “jdk1.8” which would seem to imply that the Bazel version provided depends on JDK1.8 and already has this kind of differentiation.

philwo commented 5 years ago

This will be a temporary measure. Java has recently switched to a six month release schedule and we found that many older but still popular distributions don't ship with a sufficiently recent JDK that we decided to go the embedded way for now.

RHEL / CentOS, Debian stable, Ubuntu LTS, ... will probably always ship way too outdated JDKs for our needs. It's not like this will change in the future. The idea of using an embedded JDK was to become independent of what JDK versions the popular distributions ship and not break our users when we upgrade to a newer one.

Of course we can ask people who use these distros to just install a suitable JDK via their package manager, even if it means getting it from some PPA or third-party repository, but is that better? When we discussed this ~2 years (?) ago, the sentiment was: "No, requiring people to install system packages from third-party repos just to use Bazel is not acceptable".

I personally think that embedding a JDK in our official releases and distro packages is the right thing to do, as this is the only way to ensure that people use what we actually tested on CI and also use ourselves. We had breakages with JavaBuilder in the past even when people used the correct major version of OpenJDK, but a too new or too old patch version that we didn't test against.

For packages maintained by distributions themselves (this also applies to Homebrew), they would obviously feel different, but then they'd be responsible for picking matching versions of the JDK and Bazel in their overall version ecosystem and testing the results - just as they already successfully do with all other packages they maintain.

philwo commented 5 years ago

Actually I just realised that the APT source “component” we use right now is “jdk1.8” which would seem to imply that the Bazel version provided depends on JDK1.8 and already has this kind of differentiation.

This is just historical baggage - we had a special jdk1.7 build that used an older, fixed version of JavaBuilder for a while, but when Bazel finally broke compatibility and just could no longer build / run with that version, we dropped it. We just kept the remaining jdk1.8 component.

buchgr commented 5 years ago

Of course we can ask people who use these distros to just install a suitable JDK via their package manager, even if it means getting it from some PPA or third-party repository, but is that better? When we discussed this ~2 years (?) ago, the sentiment was: "No, requiring people to install system packages from third-party repos just to use Bazel is not acceptable".

That's what we are doing today. See https://docs.bazel.build/versions/master/install-ubuntu.html#using-bazel-custom-apt-repository

We had breakages with JavaBuilder in the past even when people used the correct major version of OpenJDK, but a too new or too old patch version that we didn't test against.

That's a problem and I think you are also aware that we have agreed that going forward we need to be able to supply the correct JavaBuilder for all supported major JDK versions. You additionally also know that we will soon hide the embedded JDK from the --host_javabase and --javabase and have the user supply their own JDK for these flags. I think it's out of question that we'll need to ensure that for these flags we are compatible with the distribution JDKs.

philwo commented 5 years ago

@buchgr Why do you think do we even have an embedded JDK, if asking people to simply install the correct one is fine? (No snark intended, honest question.)

cushon commented 5 years ago

For the target --javabase it's easy an necessary to support a range of JDKs because our toolchain supports cross compilation: users expect to be able to use Bazel to target deployment to a range of JDK versions (currently Java >= 6).

For --host_javabase the default toolchain is going to track six-monthly JDK releases going forward, so asking Java Bazel users to install the latest release to use as a host JDK would caused similar problems to the ones solved by using an embedded --server_javabase.

We've been discussing using a remote repository for the default --host_javabase, which ensures the toolchain works and is less onerous than manually installing a compatible host javabase.

gertvdijk commented 5 years ago

For everyone coming here using Google/search hoping to see a work-around posted, here is one for Debian/Ubuntu and derivatives:

Put in your bazelrc file these two startup options:

startup --host_jvm_args=-Djavax.net.ssl.trustStore=/etc/ssl/certs/java/cacerts \
        --host_jvm_args=-Djavax.net.ssl.trustStorePassword=changeit

From earlier comment:

related: https://groups.google.com/forum/#!topic/bazel-discuss/13uPDObyfQg

Thanks @cushon and @philwo!

calder commented 5 years ago

@gertvdijk: With Bazel 0.16.1 this fails with

INFO: Reading 'startup' options from /path/to/workspace/.bazelrc: --host_jvm_args=-Djavax.net.ssl.trustStore=/etc/ssl/certs/java/cac
erts --host_jvm_args=-Djavax.net.ssl.trustStorePassword=changeit
Server crashed during startup. Now printing /home/ccoalson/.cache/bazel/_bazel_ccoalson/f8087e59fd95af1ae29e8fcb7ff1a3
dc/server/jvm.out
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/home/ccoalson/.cache/bazel/_bazel_ccoalson/install/ca1b8b7c3e5200be14b7f27896826862/_embedded_binaries/A-server.jar) to field sun.nio.ch.SelectorImpl.selectedKeys
WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
gertvdijk commented 5 years ago

@calder 

Server crashed during startup.

This indicates something is hopelessly wrong. Some things to check:

emranbm commented 4 years ago

Same issue. Any update?

gertvdijk commented 4 years ago

Same issue. Any update?

Please be more specific on your system/environment. I've been using the approach I commented earlier for a year now. Are you able to apply a work-around like that? (probably needs adjustments to your system if not Debian/Ubuntu-like.)

emranbm commented 4 years ago

@gertvdijk Oh, why didn't I see that?! Solved my problem. Thanks a lot!

AndrewRayCode commented 3 years ago

On a Mac, I've installed my root cert to my jvm certs file, for me it's:

/Library/Java/JavaVirtualMachines/zulu-11.jdk/Contents/Home/lib/security/cacerts

I can verify my expected cert is in that file with:

keytool -list -v -keystore /Library/Java/JavaVirtualMachines/zulu-11.jdk/Contents/Home/lib/security/cacerts | grep "My Expected Cert"

In my ~/.bazelrc file, I have:

startup --host_jvm_args="-Djavax.net.ssl.trustStore=/Library/Java/JavaVirtualMachines/zulu-11.jdk/Contents/Home/lib/security/cacerts" \
    --host_jvm_args=-Djavax.net.ssl.trustStorePassword=changeit

When I run my go generate command, I see this at the start:

INFO: Reading 'startup' options from /Users/andy/.bazelrc: --host_jvm_args=-Djavax.net.ssl.trustStore=/Library/Java/JavaVirtualMachines/zulu-11.jdk/Contents/Home/lib/security/cacerts, --host_jvm_args=-Djavax.net.ssl.trustStorePassword=changeit

So It's parsing the file, and doesn't seem to complain about it.

However, it fails to install any packages while I'm on my company's corporate VPN (which is what inserts this cert):

PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

bazel --version
bazel 2.2.0

Any thoughts here?

philwo commented 3 years ago

Hi @AndrewRayCode,

unfortunately I don't know much about how Java handles SSL certificates, but I just had an idea. You could try to run Bazel with your system JDK and see if that makes it work:

# Run Bazel with the embedded JDK
$ bazel info
[...]
java-home: /private/var/tmp/_bazel_philwo/install/d07238c5957d0addf241b6be07f3c14b/embedded_tools/jdk
java-runtime: OpenJDK Runtime Environment (build 11.0.6+10-LTS) by Azul Systems, Inc.
java-vm: OpenJDK 64-Bit Server VM (build 11.0.6+10-LTS, mixed mode) by Azul Systems, Inc.
[...]

# Run Bazel with my system JDK
$ bazel --server_javabase=$JAVA_HOME info
[...]
java-home: /Library/Java/JavaVirtualMachines/zulu-11.jdk/Contents/Home
java-runtime: OpenJDK Runtime Environment (build 11.0.11+9-LTS) by Azul Systems, Inc.
java-vm: OpenJDK 64-Bit Server VM (build 11.0.11+9-LTS, mixed mode) by Azul Systems, Inc.
[...]
dom96 commented 1 year ago

I ran into this on macOS. After installing java via brew install java, then Java proper (using the .dmg from https://www.java.com/en/download/), it started to work.

fhanau commented 7 months ago

Dominik's response unfortunately did not work for me, but adding startup --host_jvm_args=-Djavax.net.ssl.trustStoreType=KeychainStore finally got this to work for me on macOS when using a private CA. Note that you need to have the relevant certificates installed to your keychain for this to work.