spring-cloud / spring-cloud-dataflow

A microservices-based Streaming and Batch data processing in Cloud Foundry and Kubernetes
https://dataflow.spring.io
Apache License 2.0
1.11k stars 583 forks source link

volumeMount as cmd arg fails to convert to List<VolumeMount> #5428

Closed Manu10744 closed 1 year ago

Manu10744 commented 1 year ago

Description: I want to configure SCDF Server in Kubernetes to mount a volume into the pods that are created when I start a Task. I did so using the following deployment spec:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-dataflow-deployment
  namespace: demo
spec:
  revisionHistoryLimit: 1
  selector:
    matchLabels:
      app: spring-cloud-dataflow
  template:
    metadata:
      labels:
        app: spring-cloud-dataflow
    spec:
      containers:
        - name: spring-cloud-dataflow
          image: <my_custom_image>
          imagePullPolicy: Always
          resources:
            requests:
              cpu: "5m"
              memory: "480M"
            limits:
              cpu: "4"
              memory: "2G"
          ports:
            - containerPort: 8080
          startupProbe:
            httpGet:
              path: /scdf/dashboard
              port: 8080
            failureThreshold: 30
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /scdf/dashboard
              port: 8080
            failureThreshold: 2
            periodSeconds: 10
          args:
            - --spring.cloud.dataflow.task.platform.kubernetes.accounts.default.volumes=[{name:'batch-volume',persistentVolumeClaim:{claimName:'kf-batch-pvc-demosuite-23-6-test'}}]
            - --spring.cloud.dataflow.task.platform.kubernetes.accounts.default.volumeMounts=[{name:'batch-volume',mountPath:'/tmp'}] 
            - --spring.cloud.dataflow.features.streams-enabled=false
            - --spring.cloud.dataflow.features.schedules-enabled=false
            - --spring.cloud.dataflow.features.tasks-enabled=true
            - --spring.cloud.dataflow.task.platform.kubernetes.accounts.default.limits.memory=1024Mi
            - --spring.cloud.dataflow.task.platform.kubernetes.accounts.default.imagePullSecret=harbor-secret
            - --spring.cloud.dataflow.task.platform.kubernetes.accounts.default.imagePullPolicy=always
            - --spring.sql.init.mode=always
            - --spring.sql.init.data-locations=classpath*:/org/springframework/cloud/dataflow/server/db/migration/h2/V1__INITIAL_SETUP.sql
            - --spring.sql.init.continue-on-error=true
            - --spring.flyway.enabled=false
            - --server.port=8080
            - --server.servlet.context-path=/scdf
          env:
            - name: SPRING_DATASOURCE_URL
              value: jdbc:h2:tcp://my-h2db-service:1521/scdf
            - name: KUBERNETES_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: "metadata.namespace"
            - name: SPRING_CLOUD_CONFIG_ENABLED
              value: 'false'
            - name: SPRING_CLOUD_DATAFLOW_SERVER_URI
              value: 'http://scdf-service:8080'
            - name: SPRING_CLOUD_DATAFLOW_TASK_COMPOSED_TASK_RUNNER_URI
              value: 'docker://springcloud/spring-cloud-dataflow-composed-task-runner:2.6.3'
            - name: SPRING_CLOUD_KUBERNETES_CONFIG_ENABLE_API
              value: 'false'
            - name: SPRING_CLOUD_KUBERNETES_SECRETS_ENABLE_API
              value: 'false'
            - name: JAVA_TOOL_OPTIONS
              value: '-Xmx1g'
      serviceAccountName: scdf-sa 
      imagePullSecrets:
        - name: my-registry-secret

After redeploying the SCDF server it crashes with the following error:

***************************
APPLICATION FAILED TO START
***************************

Description:

Failed to bind properties under 'spring.cloud.dataflow.task.platform.kubernetes.accounts.default.volume-mounts' to java.util.List<io.fabric8.kubernetes.api.model.VolumeMount>:

    Property: spring.cloud.dataflow.task.platform.kubernetes.accounts.default.volume-mounts
    Value: "[{name:'batch-volume',mountPath:'/tmp'}]"
    Origin: "spring.cloud.dataflow.task.platform.kubernetes.accounts.default.volumeMounts" from property source "commandLineArgs"
    Reason: org.springframework.core.convert.ConverterNotFoundException: No converter found capable of converting from type [java.lang.String] to type [java.util.List<io.fabric8.kubernetes.api.model.VolumeMount>]

Action:

Update your application's configuration

Release versions:

{
  "versions": {
    "implementation": {
      "name": "spring-cloud-dataflow-server",
      "version": "2.10.3"
    },
    "core": {
      "name": "Spring Cloud Data Flow Core",
      "version": "2.10.3"
    },
    "dashboard": {
      "name": "Spring Cloud Dataflow UI",
      "version": "3.3.3"
    },
    "shell": {
      "name": "Spring Cloud Data Flow Shell",
      "version": "2.10.3",
      "url": "https://repo.maven.apache.org/maven2/org/springframework/cloud/spring-cloud-dataflow-shell/2.10.3/spring-cloud-dataflow-shell-2.10.3.jar"
    }
  },
  "features": {
    "streams": false,
    "tasks": true,
    "schedules": false,
    "monitoringDashboardType": "NONE"
  },
  "runtimeEnvironment": {
    "appDeployer": {
      "deployerImplementationVersion": null,
      "deployerName": null,
      "deployerSpiVersion": null,
      "javaVersion": null,
      "platformApiVersion": null,
      "platformClientVersion": null,
      "platformHostVersion": null,
      "platformSpecificInfo": {},
      "platformType": null,
      "springBootVersion": null,
      "springVersion": null
    },
    "taskLaunchers": [
      {
        "deployerImplementationVersion": "2.8.3",
        "deployerName": "KubernetesTaskLauncher",
        "deployerSpiVersion": "2.8.3",
        "javaVersion": "17.0.7",
        "platformApiVersion": "v1",
        "platformClientVersion": "unknown",
        "platformHostVersion": "unknown",
        "platformSpecificInfo": {
          "namespace": "demosuite-23-6-test",
          "master-url": "https://10.96.0.1:443/"
        },
        "platformType": "Kubernetes",
        "springBootVersion": "2.7.11",
        "springVersion": "5.3.27"
      }
    ]
  },
  "monitoringDashboardInfo": {
    "url": "",
    "source": "default-scdf-source",
    "refreshInterval": 15
  },
  "security": {
    "isAuthentication": false,
    "isAuthenticated": false,
    "username": null,
    "roles": []
  }
}

Steps to reproduce: Configure SCDF server like above and it should reproduce

After moving the configuration into a ConfigMap everything worked. But it should be possible to do the same with command line arguments, shouldnt it?

onobc commented 1 year ago

Hi @Manu10744 , I am glad that the ConfigMap worked. Setting these Fabric8 object properties in the command line args is not currently supported as it requires a custom converter.

In the deployer, we manually convert from string->Fabric8 (example.

You may be able to use spring.cloud.deployer.kubernetes.volumes=... as a command line arg which would not resolve the properties until the manual path above.