Adding exposed ports on driver spec fails

[lboudard]

I'm trying to set some additional ports (trying to get a jupyter notebook as driver) as defined in driver specs.

Applying the manifest fails with the following error:

Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(SparkApplication.spec.driver): unknown field "ports" in io.k8s.sparkoperator.v1beta2.SparkApplication.spec.driver

Below my application definition:

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: spark-job
spec:
  type: Python
  pythonVersion: "3"
  mode: client
  image: ###
  imagePullPolicy: Always
  mainApplicationFile: local:///home/user/myapp/__main__.py
  sparkVersion: "3.2.0"
  sparkConfigMap: spark-conf
  monitoring:
    exposeDriverMetrics: true
    exposeExecutorMetrics: true
    prometheus:
      jmxExporterJar: "/opt/spark/jars/jmx_prometheus_javaagent-0.11.0.jar"
      port: 8090
  driver:
    ports:
    - name: jupyter-lab
      containerPort: 8080
      protocol: TCP
    - name: gateway
      containerPort: 8888
      protocol: TCP
    serviceAccount: spark
    cores: 2
    memory:  "512m"
    envFrom:
      - configMapRef:
          name: myapp
      - secretRef:
          name: myapp
    secrets:
      - name: spark
        path: /mnt/secrets
        secretType: GCPServiceAccount
  executor:
    serviceAccount: spark
    cores: 2
    memory: "512m"
    envFrom:
      - configMapRef:
          name: myapp
      - secretRef:
          name: myapp
    secrets:
      - name: spark
        path: /mnt/secrets
        secretType: GCPServiceAccount

I tried to customize the other fields defined in driver spec and it seems working.

Version info: chart-info: spark-operator-1.1.10 spark-operator version: v1beta2-1.2.3-3.1.1

[drod331]

I'm hitting the exact error as well on both DriverSpec or ExecutorSpec, same SparkOperator version v1beta2-1.2.3-3.1.1, chart spark-operator-1.1.9. I've looked through the docs and the code and it all seems like it should work. Any ideas, @luizm ?

[luizm]

Hey @drod331, Unfortunately I'm not using spark-operator anymore, taking a look quickly it seems right. Maybe try to use another image: https://console.cloud.google.com/gcr/images/spark-operator/GLOBAL/spark-operator?gcrImageListsize=30&gcrImageListsort=-uploaded&pli=1 or compile and use your own image.

[zobar]

I can confirm that ports is missing from both the driver and executor specs in the CRD (it would be here for the driver and here for the executor, between podSecurityContext and schedulerName). If you copy/paste a ports definition from somewhere else in the file, it does work.

ports:
  items:
    properties:
      containerPort:
        format: int32
        type: integer
      hostIP:
        type: string
      hostPort:
        format: int32
        type: integer
      name:
        type: string
      protocol:
        type: string
    required:
    - containerPort
    - protocol
    type: object
  type: array
  x-kubernetes-list-map-keys:
  - containerPort
  - protocol
  x-kubernetes-list-type: map

I assume that there's some kind of synchronization that someone forgot to do between the code, which does mention Ports, and the CRD, which doesn't.

[mattdornfeld]

Is there any estimate for when this might get done? If it's not too complicated I might be able to take it on. Currently working on a project that will need this feature.

[hiradha]

Can the maintainers please merge this? We would like to have this functionality.

[MohammadForouhesh]

please fix this known issue after more than 2 years.

[ChenYi015]

The ports definition of Spark pods is missed in CRD files, and have been updated in helm chart v1.4.5 with #2046 .