Weird behavior when running Spring Cloud Task applications with Spring Boot Native support

[klopfdreh]

Description: Because of the startup time we are currenty evaluating to migrate our Spring Cloud Task applications to Spring Boot Native.

We managed to get the native application running and uploading files to the s3 bucket, but there is a weird behavior of the native application compared to the java application in the Spring Cloud Data Flow UI.

Setup

1. We created an application with type task and Spring Boot version Spring Boot 3.x and URI docker://...../<imagename>:<version> (the docker image is a native image)

2. We created a task which just points towards the application

3. We did the same for a Java docker image. So there are two applications s3-uploader-native-app and s3-uploader-app and two tasks s3-uploader-native-task and s3-uploader-task

4. We launched the task of the native application like the task of the java application

Outcome

1. Java Application: It runs like expected and if you click on the execution id everything is fine. (The task is listed at the side of the information)

2. Native Application: The application executed without any errors, but if you click on the execution id some information are missing (The task is NOT listed at the side of the information)

3. There is also a difference in the task executions view: The java application finished with the execution name s3-uploader-task. The native application finished with the execution name s3-uploader-native-app

4. During the runtime however, the native task is executing with the correct name s3-uploader-native-task - it is switched to s3-uploader-native-app when completed:

5. If you click on the name of the native task execution in the task executions view (not on the execution id) an error occurs and you are redirected to the task view

6. Also an error occurs if you click on the execution id and after this on the job execution id (I guess it is because the UI can't resolve the task execution for the job execution)

Release versions:

• Spring Boot 3.3.1 (tickets: https://github.com/spring-projects/spring-boot/issues/41562 / https://github.com/spring-projects/spring-boot/issues/41663)
• Spring Cloud Config 2023.0.2 / 4.1.3 (tickets: https://github.com/spring-cloud/spring-cloud-config/issues/2439)
• Spring Cloud AWS 3.1.1 (tickets: https://github.com/awspring/spring-cloud-aws/pull/856)
• AWS SDK V2 Java: (tickets: https://github.com/aws/aws-sdk-java-v2/issues/5127)
• AWS CRT Java: (tickets: https://github.com/awslabs/aws-crt-java/issues/780 / https://github.com/awslabs/aws-crt-java/pull/749)
• Spring Cloud Data Flow 2.11.2: version.json
• Snappy-Java: (tickets: https://github.com/xerial/snappy-java/issues/429) (This causes the error of prometheus rsocket client)

Custom apps: N/A

Steps to reproduce:

• Create a simple Spring Cloud Task application with one Spring Batch job configuration and one Spring Batch step
• Use native-image of Liberica NIK or spring-boot-maven-plugin with packeto buildpacks to create a native image of the Spring Cloud Task application
• Register the application like explained in the description

Screenshots: See description

Additional context: When I manually change the name of any task execution in the database from s3-uploader-native-app to s3-uploader-native-task everthing is like it should be.

[corneil]

@klopfdreh Do you have any pod logs from the task application or data flow server during the launch?

[klopfdreh]

I had to rerun it and to change the urls / package names: s3-uploader-native-task-0lllwn2nog-hrkbq-s3-uploader-native-task-q2225oz2ww.log

[klopfdreh]

Hey @corneil - here is the pod log with

logging:
  level:
    org.springframework.jdbc.core: TRACE
spring:
  jpa:
    show-sql: true
    properties:
      hibernate:
        format_sql: true

s3-uploader-native-task-g666gry1g7-mqg5k-s3-uploader-native-task-d222pqonz6.log

It seems that this is caused by Spring Cloud Task

See the following update statement in the log

{"timestamp":"2024-08-06T12:29:28.223+0200","level":"DEBUG","thread":"main","logger":"org.springframework.jdbc.core.JdbcTemplate","message":"Executing prepared SQL statement [UPDATE BOOT3_TASK_EXECUTION set START_TIME = ?, TASK_NAME = ?, LAST_UPDATED = ?, PARENT_EXECUTION_ID = ? where TASK_EXECUTION_ID = ?]","context":"default"}
{"timestamp":"2024-08-06T12:29:28.223+0200","level":"TRACE","thread":"main","logger":"org.springframework.jdbc.core.StatementCreatorUtils","message":"Setting SQL statement parameter value: column index 1, parameter value [2024-08-06 12:29:28.222716], value class [java.sql.Timestamp], SQL type 93","context":"default"}
{"timestamp":"2024-08-06T12:29:28.223+0200","level":"TRACE","thread":"main","logger":"org.springframework.jdbc.core.StatementCreatorUtils","message":"Setting SQL statement parameter value: column index 2, parameter value [s3-uploader-native-app], value class [java.lang.String], SQL type 12","context":"default"}
{"timestamp":"2024-08-06T12:29:28.223+0200","level":"TRACE","thread":"main","logger":"org.springframework.jdbc.core.StatementCreatorUtils","message":"Setting SQL statement parameter value: column index 3, parameter value [2024-08-06 12:29:28.223544], value class [java.sql.Timestamp], SQL type 93","context":"default"}
{"timestamp":"2024-08-06T12:29:28.223+0200","level":"TRACE","thread":"main","logger":"org.springframework.jdbc.core.StatementCreatorUtils","message":"Setting SQL statement parameter value: column index 4, parameter value [null], value class [null], SQL type -5","context":"default"}
{"timestamp":"2024-08-06T12:29:28.223+0200","level":"TRACE","thread":"main","logger":"org.springframework.jdbc.core.StatementCreatorUtils","message":"Setting SQL statement parameter value: column index 5, parameter value [57], value class [java.lang.Long], SQL type -5","context":"default"}

Application Properties:

management.metrics.export.prometheus.enabled: true
management.prometheus.metrics.export.rsocket.port: 7001
spring.datasource.driverClassName: oracle.jdbc.OracleDriver
management.prometheus.metrics.export.rsocket.host: prometheus-proxy.svc.cluster.local
management.metrics.tags.application: ${spring.cloud.task.name:unknown}-${spring.cloud.task.executionid:unknown}
spring.cloud.task.name: s3-uploader-native-task
spring.cloud.deployer.bootVersion: 3
management.metrics.tags.service: task-application
spring.datasource.url: ******
spring.cloud.task.initialize-enabled: false
management.metrics.export.prometheus.rsocket.port: 7001
management.metrics.export.prometheus.rsocket.enabled: true
spring.batch.jdbc.table-prefix: BOOT3_BATCH_
spring.cloud.task.schemaTarget: boot3
management.metrics.export.prometheus.rsocket.host: prometheus-proxy.svc.cluster.local
management.prometheus.metrics.export.rsocket.enabled: true
spring.cloud.task.tablePrefix: BOOT3_TASK_
management.prometheus.metrics.export.enabled: true

Platform properties:

deployer.s3-uploader-native-app.kubernetes.volumeMounts: [{name: 'efs-pvc', mountPath: '/persistent'}]
deployer.s3-uploader-native-app.kubernetes.volumes: [{name: 'efs-pvc', persistentVolumeClaim: { claimName: 'efs-pvc', readOnly: 'false' }}]
deployer.s3-uploader-native-app.kubernetes.secretKeyRefs: [{envVarName: 'AWS_IAM_USER_ACCESS_KEY_ID', secretName: 'aws-s3-credentials', dataKey: 'AWS_IAM_USER_ACCESS_KEY_ID'},{envVarName: 'AWS_IAM_USER_SECRET_KEY', secretName: 'aws-s3-credentials', dataKey: 'AWS_IAM_USER_SECRET_KEY'}]
app.s3-uploader-native-app.spring.cloud.task.tablePrefix: BOOT3_TASK_
deployer.s3-uploader-native-app.kubernetes.environmentVariables: JAVA_OPTS='-Xmx3616m -XX:TieredStopAtLevel=1 -noverify -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/persistent/',APPLICATION_NAME='s3-uploader-native-app',OTEL_SERVICE_NAME='s3-uploader-native-task',AWS_S3_ENDPOINT='$[AWS_BATCH_S3_ENDPOINT]',S3_UPLOADER_RESOURCES_TO_UPLOAD_0_INPUT_RESOURCES_PATTERN_URIS='file://native-image/*-exec.jar',S3_UPLOADER_RESOURCES_TO_UPLOAD_0_OUTPUT_FOLDER_RESOURCE_URI='s3://batch-s3-bucket-dev/',SPRING_CLOUD_CONFIG_NAME_VALUE='s3-uploader-native-task',SPRING_CLOUD_CONFIG_LABEL_VALUE='test',DRY_RUN='false',RUN_ID='test-s3-uploader-native_20240806_082020_s3uploader',TASK_IDENTIFIER='test-s3-uploader-native_20240806_082020'
deployer.s3-uploader-native-app.kubernetes.deploymentLabels: business-serviceid-of-local:Test-1587,alog_data-class:ConfidentialPseudo,application:test-s3-uploader-native_20240806_082020-s3-uploader-native-task,cloud-env:dev
app.s3-uploader-native-app.spring.cloud.task.schemaTarget: boot3
app.s3-uploader-native-app.spring.batch.jdbc.table-prefix: BOOT3_BATCH_
deployer.s3-uploader-native-app.kubernetes.limits.cpu: 1500m
deployer.s3-uploader-native-app.kubernetes.limits.memory: 4000Mi
app.s3-uploader-native-app.spring.cloud.task.initialize-enabled: false
app.s3-uploader-native-app.spring.cloud.deployer.bootVersion: 3

[klopfdreh]

It seems that the task name is not resolved correctly at the start of the task application.

I had a deeper look into this - the interface TaskNameResolver is implemented by SimpleTaskNameResolver which takes it out of ${spring.cloud.task.name:} - due to relax binding it should be spring.cloud.task.name: s3-uploader-native-task as of the application properties. I also had a look into the environment variables of the pod and there is a SPRING_CLOUD_TASK_NAME with the value s3-uploader-native-task

I have no idea why it is switched at start.

[klopfdreh]

I changed spring.application.name as I saw that SimpleTaskNameResolver uses the id of the context in case of spring.cloud.task.name is not set - see https://github.com/spring-cloud/spring-cloud-task/blob/main/spring-cloud-task-core/src/main/java/org/springframework/cloud/task/repository/support/SimpleTaskNameResolver.java#L59 - maybe @Value needs to be changed to a Properties class and should use the relax binding instead. The question here is if @Value also uses env vars?

[klopfdreh]

I found a workaround:

spring:
  application:
    name: ${spring.cloud.task.name}
  cloud:
    task:
      name: ${SPRING_CLOUD_TASK_NAME:s3-uploader-native-task}

Maybe this is an issue of Spring Cloud Task with Spring Cloud Config

[klopfdreh]

@corneil - did you had the chance to also test it, yet?

[klopfdreh]

Just a small shoutout: We managed to migrate 11 task applications to Spring Boot Native (with this workaround here) and a lot of commits to various projects.