How to load dependency packages using private nexus repository proxy

[PiePra]

We have to use a private maven repository proxy to download given packages under deps. The SparkApplication however stays in status SUBMITION_FAILED because it is unable to verify the certificate of the private repository proxy.

Server access error at url <path to dependency on nexus> (javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target)

we tried various settings to pass the ca certificate to driver and executor using a JKS file. Nevertheless the connection producing the error seems to happen before even a pod is scheduled on kubernetes, so the ca certificate must be provided at an earlier stage in the process. How can we get around this?

The manifest is as follows:

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: pyspark-pi
  namespace: spark-apps
spec:
  volumes:
    - name: ssl
      configMap:
        name: my-org.com
  sparkConf:
    spark.ssl.keyStore: /mnt/spark/ssl/bundle.jks
    spark.ssl.keyStorePassword: changeit
    spark.ssl.trustStore: /mnt/spark/ssl/bundle.jks
    spark.ssl.trustStorePassword: changeit
    spark.ssl.trustStoreType: JKS
  deps:
    repositories:
    -  <nexus_repository_url>
    packages:
    - "com.amazonaws:aws-java-sdk-bundle:1.12.629"
    - "org.apache.hadoop:hadoop-aws:3.3.6"
  hadoopConf:
    fs.s3a.impl: org.apache.hadoop.fs.s3a.S3AFileSystem
  type: Python
  pythonVersion: "3"
  mode: cluster
  image: "gcr.io/spark-operator/spark-py:v3.1.1"
  driver:
    javaOptions: "-Djavax.net.ssl.trustStorePassword=changeit -Djavax.net.ssl.trustStore=/mnt/spark/ssl/bundle.jks -Djavax.net.ssl.keyStorePassword=changeit -Djavax.net.ssl.keyStore=/mnt/spark/ssl/bundle.jks"
    volumeMounts:
      - name: ssl
        mountPath: /mnt/spark/ssl
  executor:
    javaOptions: "-Djavax.net.ssl.trustStorePassword=changeit -Djavax.net.ssl.trustStore=/mnt/spark/ssl/bundle.jks -Djavax.net.ssl.keyStorePassword=changeit -Djavax.net.ssl.keyStore=/mnt/spark/ssl/bundle.jks"
    volumeMounts:
      - name: ssl
        mountPath: /mnt/spark/ssl

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.