Spring Cloud Data Flow attempting to use batch/v1beta1 apis instead of batch/v1 for scheduling cron jobs; Gives 404 error

Name and Version

bitnami/spring-cloud-dataflow 18.1.1

What architecture are you using?

amd64

What steps will reproduce the bug?

We have a spring cloud data flow cluster installed in kubernetes (Server Version: v1.25.5). We installed it using the helm chart bitnami/spring-cloud-dataflow 18.1.1

helm upgrade --install spring-cloud-data-flow -n dev01 bitnami/spring-cloud-dataflow ^
--set "externalDatabase.dataflow.username=scdf" ^
--set "externalDatabase.host=postgresql-stolon-proxy" ^
--set "externalDatabase.password=<snip>" ^
--set "externalDatabase.port=5432" ^
--set "externalDatabase.scheme=postgresql" ^
--set "externalDatabase.skipper.username=scdf" ^
--set "externalDatabase.username=scdf" ^
--set "externalKafka.brokers=kafka-headless:9092" ^
--set "externalKafka.enabled=true" ^
--set "externalKafka.zkNodes=kafka-zookeeper:2181" ^
--set "kafka.enabled=false" ^
--set "mariadb.enabled=false" ^
--set "rabbitmq.enabled=false"

Following is the deployment manifest for the scdf server:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-data-flow-spring-cloud-dataflow-server
  namespace: dev01
  labels:
    app.kubernetes.io/component: server
    app.kubernetes.io/instance: spring-cloud-data-flow
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: spring-cloud-dataflow
    helm.sh/chart: spring-cloud-dataflow-18.1.1
  annotations:
    deployment.kubernetes.io/revision: '7'
    meta.helm.sh/release-name: spring-cloud-data-flow
    meta.helm.sh/release-namespace: dev01
  managedFields:
    - manager: node-fetch
      operation: Update
      apiVersion: apps/v1
      time: '2023-04-28T15:50:30Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:spec:
          f:template:
            f:metadata:
              f:annotations:
                f:kubectl.kubernetes.io/restartedAt: {}

    - manager: kube-controller-manager
      operation: Update
      apiVersion: apps/v1
      time: '2023-06-02T13:37:16Z'
      fieldsType: FieldsV1
      subresource: status
  selfLink: >-
    /apis/apps/v1/namespaces/dev01/deployments/spring-cloud-data-flow-spring-cloud-dataflow-server
status:
  observedGeneration: 7
  replicas: 1
  updatedReplicas: 1
  readyReplicas: 1
  availableReplicas: 1
  conditions:
    - type: Available
      status: 'True'
      lastUpdateTime: '2023-06-02T01:00:58Z'
      lastTransitionTime: '2023-06-02T01:00:58Z'
      reason: MinimumReplicasAvailable
      message: Deployment has minimum availability.
    - type: Progressing
      status: 'True'
      lastUpdateTime: '2023-06-02T13:37:16Z'
      lastTransitionTime: '2023-06-02T13:37:16Z'
      reason: NewReplicaSetAvailable
      message: >-
        ReplicaSet
        "spring-cloud-data-flow-spring-cloud-dataflow-server-7c7d4b7b75" has
        successfully progressed.
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: server
      app.kubernetes.io/instance: spring-cloud-data-flow
      app.kubernetes.io/name: spring-cloud-dataflow
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: server
        app.kubernetes.io/instance: spring-cloud-data-flow
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: spring-cloud-dataflow
        helm.sh/chart: spring-cloud-dataflow-18.1.1
      annotations:
        checksum/configuration: e703fac261658f6ececf92b919f4b0fae16e345878e96197866286e630a1f34e
        kubectl.kubernetes.io/restartedAt: '2023-04-28T15:50:30Z'
    spec:
      volumes:
        - name: database
          secret:
            secretName: spring-cloud-data-flow-spring-cloud-dataflow-externaldb
            defaultMode: 420
        - name: config
          configMap:
            name: spring-cloud-data-flow-spring-cloud-dataflow-server
            items:
              - key: application.yaml
                path: application.yml
            defaultMode: 420
        - name: scripts
          configMap:
            name: spring-cloud-data-flow-spring-cloud-dataflow-scripts
            defaultMode: 493
      initContainers:
        - name: wait-for-backends
          image: docker.io/bitnami/kubectl:1.25.9-debian-11-r9
          command:
            - /scripts/wait-for-backends.sh
          resources: {}
          volumeMounts:
            - name: scripts
              mountPath: /scripts/wait-for-backends.sh
              subPath: wait-for-backends.sh
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      containers:
        - name: server
          image: docker.io/bitnami/spring-cloud-dataflow:2.10.3-debian-11-r4
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          env:
            - name: BITNAMI_DEBUG
              value: 'false'
            - name: SERVER_PORT
              value: '8080'
            - name: SPRING_CLOUD_CONFIG_ENABLED
              value: 'false'
            - name: SPRING_CLOUD_KUBERNETES_CONFIG_ENABLE_API
              value: 'false'
            - name: SPRING_CLOUD_KUBERNETES_SECRETS_ENABLE_API
              value: 'false'
            - name: SPRING_CLOUD_KUBERNETES_SECRETS_PATHS
              value: /etc/secrets
            - name: SPRING_CLOUD_DATAFLOW_SERVER_URI
              value: >-
                http://spring-cloud-data-flow-spring-cloud-dataflow-server.dev01.svc.cluster.local:8080
            - name: SPRING_CLOUD_DATAFLOW_FEATURES_STREAMS_ENABLED
              value: 'true'
            - name: SPRING_CLOUD_DATAFLOW_FEATURES_TASKS_ENABLED
              value: 'true'
            - name: SPRING_CLOUD_DATAFLOW_FEATURES_SCHEDULES_ENABLED
              value: 'true'
            - name: SPRING_CLOUD_SKIPPER_CLIENT_SERVER_URI
              value: >-
                http://spring-cloud-data-flow-spring-cloud-dataflow-skipper.dev01.svc.cluster.local/api
            - name: SPRING_APPLICATION_JSON
              value: >-
                { "maven": { "local-repository": null, "remote-repositories": {
                "repo1": { "url": "https://repo.spring.io/libs-snapshot"} } } }
            - name: KUBERNETES_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: KUBERNETES_TRUST_CERTIFICATES
              value: 'false'
            - name: SPRING_CLOUD_DATAFLOW_TASK_COMPOSEDTASKRUNNER_URI
              value: >-
                docker://docker.io/bitnami/spring-cloud-dataflow-composed-task-runner:2.10.3-debian-11-r1
          resources: {}
          volumeMounts:
            - name: database
              readOnly: true
              mountPath: /etc/secrets/database
            - name: config
              readOnly: true
              mountPath: /opt/bitnami/spring-cloud-dataflow/conf
          livenessProbe:
            httpGet:
              path: /management/health
              port: http
              scheme: HTTP
            initialDelaySeconds: 120
            timeoutSeconds: 1
            periodSeconds: 20
            successThreshold: 1
            failureThreshold: 6
          readinessProbe:
            httpGet:
              path: /management/health
              port: http
              scheme: HTTP
            initialDelaySeconds: 120
            timeoutSeconds: 1
            periodSeconds: 20
            successThreshold: 1
            failureThreshold: 6
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
          securityContext:
            runAsUser: 1001
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      serviceAccountName: spring-cloud-data-flow-spring-cloud-dataflow
      serviceAccount: spring-cloud-data-flow-spring-cloud-dataflow
      securityContext:
        fsGroup: 1001
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 1
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/component: server
                    app.kubernetes.io/instance: spring-cloud-data-flow
                    app.kubernetes.io/name: spring-cloud-dataflow
                topologyKey: kubernetes.io/hostname
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

Following is the output of the command kubectl api-versions

acme.cert-manager.io/v1
admissionregistration.k8s.io/v1
apiextensions.k8s.io/v1
apiregistration.k8s.io/v1
apps/v1
authentication.k8s.io/v1
authorization.k8s.io/v1
autoscaling/v1
autoscaling/v2
autoscaling/v2beta2
batch/v1
cert-manager.io/v1
certificates.k8s.io/v1
coordination.k8s.io/v1
crd.k8s.amazonaws.com/v1alpha1
discovery.k8s.io/v1
events.k8s.io/v1
flowcontrol.apiserver.k8s.io/v1beta1
flowcontrol.apiserver.k8s.io/v1beta2
metrics.k8s.io/v1beta1
monitoring.coreos.com/v1
monitoring.coreos.com/v1alpha1
networking.k8s.io/v1
node.k8s.io/v1
policy/v1
rbac.authorization.k8s.io/v1
scheduling.k8s.io/v1
storage.k8s.io/v1
storage.k8s.io/v1beta1
v1
velero.io/v1

We are getting the following error whenever we try to load the schedules page in the UI.

Failure executing: GET at: https://172.20.0.1/apis/batch/v1beta1/namespaces/dev01/cronjobs. Message: the server could not find the requested resource. Received status: Status(apiVersion=v1, code=404, details=StatusDetails(causes=[], group=null, kind=null, name=null, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=the server could not find the requested resource, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=NotFound, status=Failure, additionalProperties={}).
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
    at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
    at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:768)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:389)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:926)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.base/java.lang.Thread.run(Thread.java:833)
2023-06-06T15:32:48.541186169Z

What do you see instead?

Screenshot 2023-06-06 at 10 47 48 PM

Folks, any help troubleshooting why it is calling batch/v1beta1 instead of batch/v1 is highly appreciated.

spring-cloud / spring-cloud-skipper