search

apache-spark-on-k8s / spark

Apache Spark enhanced with native Kubernetes scheduler back-end: NOTE this repository is being ARCHIVED as all new development for the kubernetes scheduler back-end is now on https://github.com/apache/spark/
https://spark.apache.org/
Apache License 2.0
612 stars 118 forks source link

Issues parsing service account certificates #622

Closed adelbertc closed 6 years ago

adelbertc commented 6 years ago

This is on the 0.5.0 release.

When you tell Spark to use the Kubernetess ServiceAccount credentials (e.g. the driver or the default behavior of the resource staging server), it will look for certificates in the automatically mounted file /var/run/secrets/kubernetes.io/serviceaccount/ca.crt. However, at least on my Kubernetes 1.6.x server, it always fails with this error:

Caused by: java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input
    at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
    at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
    at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:93)
    at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:71)
    at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:114)
    at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:93)
    at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:63)
    ... 3 more

HOWEVER if I change the certificate file to remove the trailing newline, Spark is happy with it. It seems perhaps this is part of the PEM format but curl happily accepts the unmodified cert. I am unsure if this is a fault of PEM or of how Java is choosing to parse PEM files.

adelbertc commented 6 years ago

OK I just tried this on a K8s 1.8.x cluster and it doesn't have the weird trailing newline. Unsure if this is a bug in K8s 1.6.x itself or just how we setup the cluster, but don't think this is an issue with Spark-on-K8s. Closing.

foxish commented 6 years ago

Thanks for investigating @adelbertc!

mrow4a commented 6 years ago

We encountered the same in Openstack with Spark @adelbertc @foxish. Maybe this solution (https://git.openstack.org/cgit/openstack/magnum/commit/?id=edee7030e4deee4e95e68daa1623ea305ce202e5) can be useful in investigation, as this solved the problem in Openstack - the problem was cloud related, not K8S.

Generally, the certificate has to be striped for "parser" spark uses.