search

kubeflow / spark-operator

Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
Apache License 2.0
2.78k stars 1.38k forks source link

sparkConf creates incorrect path from `spark.jars.packages` and `spark.jars.repositories` #1039

Open arichardet opened 4 years ago

arichardet commented 4 years ago

Using spark-operator:v1beta2-1.2.0-3.0.0, and creating a SparkApplication as follows:

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: pyspark-pi
  namespace: spark-operator
spec:
  sparkConf:
    spark.jars.packages: "org.apache.hadoop:hadoop-aws:2.7.6,com.amazonaws:aws-java-sdk:aws-java-sdk-1.7.6"
    spark.jars.repositories: <redacted>
...

The url that is created is incorrect: <redacted>/com/amazonaws/aws-java-sdk/aws-java-sdk-1.7.6/aws-java-sdk-aws-java-sdk-1.7.6.jar Note the repeated aws-java-sdk-aws-java-sdk

github-actions[bot] commented 2 weeks ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.