search

kubernetes-client / java

Official Java client library for kubernetes
http://kubernetes.io/
Apache License 2.0
3.55k stars 1.89k forks source link

IPv6 Service Address TLS Hostname Verification Fails #1289

Closed lyind closed 3 years ago

lyind commented 4 years ago

Kubernetes 1.19.1 (for example when setup using kubeadm) does not - by default - sign the IPv6 short-form of the api-server address.

Unfortunately kubernetes-client/java (version 10.0.0) uses the short-form string representation of the kubernetes service ClusterIp in the URL, which leads to the following exception:

2020-09-23 10:35:05.162 [] [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-0] ReflectorRunnable.defaultWatchErrorHandler() ERROR: class io.kubernetes.client.openapi.models.V1Pod#Reflector loop failed unexpectedly
io.kubernetes.client.openapi.ApiException: javax.net.ssl.SSLPeerUnverifiedException: Hostname 2a01:4f8:d0:1601::1 not verified:
    certificate: sha256/Xv8eDm3+ZqXDNTXy0friFPgGYo258qvYjPsc1XNSNbQ=
    DN: CN=kube-apiserver
    subjectAltNames: [2a01:4f8:d0:1601:0:0:0:1, 2a01:4f8:d0:1600:0:0:0:4, 2a01:4f8:d0:1600:ffff:ffff:fffe:0, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, node-d0-50-99-f1-8e-cb]
    at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:886)
    at io.kubernetes.client.informer.SharedInformerFactory$1.list(SharedInformerFactory.java:182)
    at io.kubernetes.client.informer.cache.ReflectorRunnable.run(ReflectorRunnable.java:74)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)
Caused by: javax.net.ssl.SSLPeerUnverifiedException: Hostname 2a01:4f8:d0:1601::1 not verified:
    certificate: sha256/Xv8eDm3+ZqXDNTXy0friFPgGYo258qvYjPsc1XNSNbQ=
    DN: CN=kube-apiserver
    subjectAltNames: [2a01:4f8:d0:1601:0:0:0:1, 2a01:4f8:d0:1600:0:0:0:4, 2a01:4f8:d0:1600:ffff:ffff:fffe:0, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, node-d0-50-99-f1-8e-cb]
    at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:350)
    at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:300)
    at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:185)
    at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
    at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
    at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
    at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
    at io.kubernetes.client.util.credentials.TokenFileAuthentication.intercept(TokenFileAuthentication.java:72)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
    at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229)
    at okhttp3.RealCall.execute(RealCall.java:81)
    at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:882)
    ... 8 common frames omitted

Valid alternatives could be to either use the full (un-shortened) IPv6 address (2a01:4f8:d0:1601:0:0:0:1 in my case) or just use the default DNS name kubernetes. People looking to avoid the service name lookup, ie. for performance/reliability reasons, could instantiate the client separately.

This issue is somewhat related to the (fixed): #839

lyind commented 4 years ago

I may share my workaround (many implementations possible, but this should be a drop-in):

// Workaround for https://github.com/kubernetes-client/java/issues/1289
// TODO when #1289 is fixed, replace with: return ClientBuilder.cluster().build();
return new ClientBuilder()
{
    public ClientBuilder withIpv6Workaround() throws IOException
    {
        String host = System.getenv(ENV_SERVICE_HOST);
        final String port = System.getenv(ENV_SERVICE_PORT);

        if (host != null && host.contains(":"))
        {
            // IPv6 address detected, ensure long format (as in api-server cert SANs)
            host = InetAddress.getByName(host).getHostAddress();
        }

        setBasePath(host, port);

        setCertificateAuthority(Files.readAllBytes(Paths.get(SERVICEACCOUNT_CA_PATH)));
        setAuthentication(new TokenFileAuthentication(SERVICEACCOUNT_TOKEN_PATH));

        return this;
    }
}
        .withIpv6Workaround().build();
brendandburns commented 4 years ago

Thanks for reporting. We'd be happy to take a PR to address this.

lyind commented 4 years ago

OkHttps HttpUrl.canonicalizeHost() method always "canonicalizes" IPv6 addresses in request URLs into the compressed/short representation (ie. 0:0:0:0:0:0:0:1 -> ::1).

OkHostnameVerifier.verifyIpAddress() checks for matching SANs using String.equalsIgnoreCase().

Unfortunately IPv6 addresses in TLS certificate SAN fields are stored in uncompressed format (ie. 0:0:0:0:0:0:0:1), which is correct:

When the subjectAltName extension contains an iPAddress, the address MUST be stored in the octet string in "network byte order", as specified in [RFC791]. The least significant bit (LSB) of each octet is the LSB of the corresponding byte in the network address. For IP version 4, as specified in [RFC791], the octet string MUST contain exactly four octets. For IP version 6, as specified in [RFC2460], the octet string MUST contain exactly sixteen octets.

https://tools.ietf.org/html/rfc5280#section-4.2.1.6 Kudos to user @aojea for pointing it out in https://github.com/kubernetes/kubernetes/issues/95007.

OkHostnameVerifier's deviation from the standard causes issues in the field.

Fortunately this is fixed starting from OkHttp 4.5.0. Issue leading to the fix: https://github.com/square/okhttp/issues/5885

It seems possible to workaround this by overriding the OkHttp dependencies in build.gradle (or pom.xml):

    compile 'com.squareup.okhttp3:logging-interceptor:4.9.0'
    compile 'com.squareup.okhttp3:okhttp:4.9.0'

Please update the OkHttp library.

brendandburns commented 4 years ago

We'll look into updating. 3.x to 4.x is a big jump in versioning, so we'll need to validate that it works and/or do it across a major version update. (which fortunately we're about to do)

lyind commented 4 years ago

@brendanburns I updated to 4.9.0 for testing purposes. The method to make the kubernetes CA certificate known to the HTTP client doesn't work anymore. The CA certificate from the crafted TrustManager is not accepted and one gets a "No certification path" exception:

2020-09-25 12:18:42.087 [] [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-0] ReflectorRunnable.defaultWatchErrorHandler() ERROR: class io.kubernetes.client.openapi.models.V1Pod#Reflector loop failed unexpectedly
io.kubernetes.client.openapi.ApiException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at io.kubernetes.client.util.Watch.createWatch(Watch.java:109)
    at io.kubernetes.client.informer.SharedInformerFactory$1.watch(SharedInformerFactory.java:190)
    at io.kubernetes.client.informer.SharedInformerFactory$1.watch(SharedInformerFactory.java:178)
    at io.kubernetes.client.informer.cache.ReflectorRunnable.run(ReflectorRunnable.java:103)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
    at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
    at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
    at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
    at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(Unknown Source)
    at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(Unknown Source)
    at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(Unknown Source)
    at java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source)
    at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
    at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
    at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
    at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
    at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)
    at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)
    at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
    at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
    at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.kt:379)
    at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.kt:337)
    at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:209)
    at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)
    at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)
    at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)
    at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
    at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
    at io.kubernetes.client.util.Watch.createWatch(Watch.java:93)
    ... 9 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at java.base/sun.security.validator.PKIXValidator.doBuild(Unknown Source)
    at java.base/sun.security.validator.PKIXValidator.engineValidate(Unknown Source)
    at java.base/sun.security.validator.Validator.validate(Unknown Source)
    at java.base/sun.security.ssl.X509TrustManagerImpl.validate(Unknown Source)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)
    ... 39 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(Unknown Source)
    at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source)
    at java.base/java.security.cert.CertPathBuilder.build(Unknown Source)
    ... 45 common frames omitted
lyind commented 3 years ago

After some debugging I found that my dependency injection framework injected a No-Arg constructor created ApiClient instance in my SharedInformerFactory instances.

Sorry for the inconvenience.

I wish the No-Arg constructors for ApiClient, CoreV1Api, SharedInformerFactory and similar could be replaced by static factory methods like ApiClient.defaultClient() or similar. All the No-Arg constructors for CoreV1Api, Exec and such are easy to miss. Explicit is better than implicit.

lyind commented 3 years ago

Seems like the original issue is still valid, though.

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

fejta-bot commented 3 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

fejta-bot commented 3 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community. /close

k8s-ci-robot commented 3 years ago

@fejta-bot: Closing this issue.

In response to [this](https://github.com/kubernetes-client/java/issues/1289#issuecomment-787425997): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
liyuntian commented 1 year ago

I use client-java:16.0.0. my kubernetes version is 1.21. I have the same problem,which version this bug is fixed?

njucjc commented 11 months ago

After some debugging I found that my dependency injection framework injected a No-Arg constructor created ApiClient instance in my SharedInformerFactory instances.

Sorry for the inconvenience.

I wish the No-Arg constructors for ApiClient, CoreV1Api, SharedInformerFactory and similar could be replaced by static factory methods like ApiClient.defaultClient() or similar. All the No-Arg constructors for CoreV1Api, Exec and such are easy to miss. Explicit is better than implicit.

@lyind How can I fix this issue

lyind commented 11 months ago

@njucjc It's long fixed (by OkHttp client upgrade and later switch to Vert.x).

Also make sure you setup a provider for a configured ApiClient via your dependency injection framework which returns the result of ClientBuilder.cluster().build(), for example.

You may want to do similar for CoreV1Api and SharedInformerFactory.