Data Flow needs to support Batch 5.0 and Task 3.0

[cppwfs]

Currently releases of SCDF do not support running of Spring Batch 5.0 and Spring Cloud Task 3.0. This is because of some breaking changes introduced in the schemas of both projects SCDF will need to support both Spring Batch 4.x/Task 2.x as well as Spring Batch 5.x/Task 3.x schemas.

The standard buildpack for Spring Boot applications adds a label named org.springframework.boot it requires the use of Docker API to retrieve the image information.

[markpollack]

There are a few approaches to explore. Exploring the following options:

• Create a view in the DBs to expose the 5.0 schema as 4.x compatible
• Subclasses of the 4.x JobExplorer/DAO classes that can support reading both 4.x and 5.x tables

Other possible approaches considered are

• Separate process that is Spring Boot 3.0/Batch 5.0 based that acts as a proxy to the SCDF server process to read 5.0 batch job information.

Discarded approaches

shadow jar the 5.0 classes, but that will pull in all of spring 5 and be very brittle/confusing.

[anselmhou]

may I know what is the current status for this issue? If I want to work with spring batch but want to have an admin console, is there any workaround or I should source an alternative solution. Thanks

[onobc]

We are in progress w/ this now @anselmhou . We are planning on delivering it in 2.11.0 but we do not have a firm date for that established yet. We will reply here once we come up w/ a better timeframe for this.

[michoudu13820]

Hello @onobc ,

Do you have any updates on the release date ? Or an estimation when the compatibilty between spring cloud data flow and SpringBatch5 / Task3 will be available ?

Thanks :)

[onobc]

Hi @michoudu13820 batch5/task3 will be in the 2.11.0 release. It is most likely about 1 month from now but we will have a better defined date w/in the next week.

Thanks for your patience.

[rallepalle]

Hi @onobc, is there already news about an upgrade to release 2.11.0 ?

[onobc]

Hi @rallepalle , we have set the target date for 2023-05-17.

[michoudu13820]

By the way, just having a question, will this version of SCDF be retro-compatible with batch 4/task 2 ? In order to be able to launch two kind of tasks (one with batch 4/task 2 and one with batch 5/task 3 )

[onobc]

HI @michoudu13820 , yes this version will support running both.

[obiwan87]

Hi,

I installed the dockerized SCDF version 2.11.0-SNAPSHOT. My goal is to migrate all our tasks to Spring Boot 3 / Task 3 / Batch 5.

Our environment: Spring Boot 3.1.0 SCDF 2.11.0-SNAPSHOT (docker)

To keep things simple I created a minimal Task and configured it following the most current revision of the sample task timestamp-batch.

I was able to import the task into SCDF.

However, after launching it I get the following error:

java.lang.IllegalStateException: Failed to execute ApplicationRunner
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:764) [spring-boot-3.1.0.jar!/:3.1.0]
    at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:751) [spring-boot-3.1.0.jar!/:3.1.0]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:318) [spring-boot-3.1.0.jar!/:3.1.0]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1305) [spring-boot-3.1.0.jar!/:3.1.0]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1294) [spring-boot-3.1.0.jar!/:3.1.0]
    at media.ippen.cue.dataflow.delayedcounter.DelayedCounterApp.main(DelayedCounterApp.java:13) [classes!/:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
    at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) [delayed-counter-1.0.0.jar:?]
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:95) [delayed-counter-1.0.0.jar:?]
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) [delayed-counter-1.0.0.jar:?]
    at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65) [delayed-counter-1.0.0.jar:?]
Caused by: org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar [SELECT JOB_EXECUTION_ID, PARAMETER_NAME, PARAMETER_TYPE, PARAMETER_VALUE, IDENTIFYING
FROM BATCH_JOB_EXECUTION_PARAMS
WHERE JOB_EXECUTION_ID = ?

(FYI: I'm using mysql for persisting the SCDF's data.)

Seems like there is columns missing in the BATCH_JOB_EXECUTION_PARAMS table. I double-checked and confirmed it in my MySQL docker container.

mysql> SHOW COLUMNS IN BATCH_JOB_EXECUTION_PARAMS;
+------------------+--------------+------+-----+---------+-------+
| Field            | Type         | Null | Key | Default | Extra |
+------------------+--------------+------+-----+---------+-------+
| JOB_EXECUTION_ID | bigint(20)   | NO   | MUL | NULL    |       |
| TYPE_CD          | varchar(6)   | NO   |     | NULL    |       |
| KEY_NAME         | varchar(100) | NO   |     | NULL    |       |
| STRING_VAL       | varchar(250) | YES  |     | NULL    |       |
| DATE_VAL         | datetime     | YES  |     | NULL    |       |
| LONG_VAL         | bigint(20)   | YES  |     | NULL    |       |
| DOUBLE_VAL       | double       | YES  |     | NULL    |       |
| IDENTIFYING      | char(1)      | NO   |     | NULL    |       |
+------------------+--------------+------+-----+---------+-------+
8 rows in set (0.00 sec)

Why do I get this error? Is this due to Spring Boot 3.1.0? Let me know if you need more information about how my project is configured.

Thank you

[cppwfs]

@obiwan87 We are adding support for running Batch 4.0 for Boot 2.7 and Batch 5.0 for Boot 3.0/3.1 to SCDF 2.11.0.

[klopfdreh]

Will SCDF server still be available for Java lower 17? As we run the rest api on a system that only has Java 1.8 available and it includes spring-cloud-dataflow-server-core.

This would be very important for us.

[onobc]

Indeed @klopfdreh - the server is still running Spring Boot 2.x. We are only adding the ability to launch the SB3 version of the apps / tasks.

[klopfdreh]

This helps us a lot! Thanks!

[priyansms]

Hi @onobc @markpollack When can we expect the SCDF 2.11.0 released and generally available for use ? do we have a date ?

[umutpiri]

Hi, is there any planned date for SCDF 2.11 release? Or is there any workaround to be able to use SCDF with spring batch 5?

[vauvenal5]

Hi!

Is supporting Batch5/Task3 still planed for the 2.11.0 release end of July? If not, do you have a rough estimate when to expect it?

Thanks.

[cppwfs]

Yes, we are still working on adding this feature to the 2.11.0 release.

[onobc]

Main work has been done here. There are other fine-grained issues to cover the remaining work.

[klopfdreh]

Great to hear! Hope you consider to also include 1-2 of my PRs 😁

[onobc]

@klopfdreh I am not sure if we will be able to include any of your PRs in 2.11.0 strictly because they are big enough changes. We ❤️ all of your contributions and will make it a priority in getting these in as quick as possible after 2.11.0 - thanks again and we appreciate your patience.

[priyansms]

Hi @onobc when can we expect a release for 2.11.0 ?

[onobc]

Hi @priyansms , thank you for your patience. I believe we are going to cut and RC1 this week and the follow w/ a GA the following week. Getting ready for Spring One plus some other high priority Spring wide initiatives have caused the date to slip. Trust me, we want this out as bad as you and are working on getting it out ASAP. Thank you.

[hi-rullah]

Hi,

I installed the dockerized SCDF version 2.11.0-SNAPSHOT. My goal is to migrate all our tasks to Spring Boot 3 / Task 3 / Batch 5.

Our environment: Spring Boot 3.1.0 SCDF 2.11.0-SNAPSHOT (docker)

To keep things simple I created a minimal Task and configured it following the most current revision of the sample task timestamp-batch.

I was able to import the task into SCDF.

However, after launching it I get the following error:

java.lang.IllegalStateException: Failed to execute ApplicationRunner
  at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:764) [spring-boot-3.1.0.jar!/:3.1.0]
  at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:751) [spring-boot-3.1.0.jar!/:3.1.0]
  at org.springframework.boot.SpringApplication.run(SpringApplication.java:318) [spring-boot-3.1.0.jar!/:3.1.0]
  at org.springframework.boot.SpringApplication.run(SpringApplication.java:1305) [spring-boot-3.1.0.jar!/:3.1.0]
  at org.springframework.boot.SpringApplication.run(SpringApplication.java:1294) [spring-boot-3.1.0.jar!/:3.1.0]
  at media.ippen.cue.dataflow.delayedcounter.DelayedCounterApp.main(DelayedCounterApp.java:13) [classes!/:?]
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
  at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
  at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) [delayed-counter-1.0.0.jar:?]
  at org.springframework.boot.loader.Launcher.launch(Launcher.java:95) [delayed-counter-1.0.0.jar:?]
  at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) [delayed-counter-1.0.0.jar:?]
  at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65) [delayed-counter-1.0.0.jar:?]
Caused by: org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar [SELECT JOB_EXECUTION_ID, PARAMETER_NAME, PARAMETER_TYPE, PARAMETER_VALUE, IDENTIFYING
FROM BATCH_JOB_EXECUTION_PARAMS
WHERE JOB_EXECUTION_ID = ?

(FYI: I'm using mysql for persisting the SCDF's data.)

Seems like there is columns missing in the BATCH_JOB_EXECUTION_PARAMS table. I double-checked and confirmed it in my MySQL docker container.

mysql> SHOW COLUMNS IN BATCH_JOB_EXECUTION_PARAMS;
+------------------+--------------+------+-----+---------+-------+
| Field            | Type         | Null | Key | Default | Extra |
+------------------+--------------+------+-----+---------+-------+
| JOB_EXECUTION_ID | bigint(20)   | NO   | MUL | NULL    |       |
| TYPE_CD          | varchar(6)   | NO   |     | NULL    |       |
| KEY_NAME         | varchar(100) | NO   |     | NULL    |       |
| STRING_VAL       | varchar(250) | YES  |     | NULL    |       |
| DATE_VAL         | datetime     | YES  |     | NULL    |       |
| LONG_VAL         | bigint(20)   | YES  |     | NULL    |       |
| DOUBLE_VAL       | double       | YES  |     | NULL    |       |
| IDENTIFYING      | char(1)      | NO   |     | NULL    |       |
+------------------+--------------+------+-----+---------+-------+
8 rows in set (0.00 sec)

Why do I get this error? Is this due to Spring Boot 3.1.0? Let me know if you need more information about how my project is configured.

Thank you

See the guide for Spring Batch v5 migration. The table BATCH_JOB_EXECUTION_PARAMS columns are different for v4 and v5 https://github.com/spring-projects/spring-batch/wiki/Spring-Batch-5.0-Migration-Guide#systemcommandtasklet-updates

[ragovi]

Hi gents, SCDF 2.11 is not yet compatible with spring batch 5.X. I missed something, why is this issue closed?

[michael-wirth]

Hi gents, SCDF 2.11 is not yet compatible with spring batch 3.X. I missed something, why is this issue closed?

Do you mean Spring Boot 3.X? I think this is another story.

Spring Batch 5.X is working fine in my project.

[ragovi]

Hi Michael, Sorry I meant spring batch 5. I tested SCDF 2.11 yesterday and it's using the SQL schemas for spring batch 4, so it fails when trying to query the tables generated by spring batch. If I let data flow generate the schema, it generates v4 version and spring batch jobs v5 fail. Which version is working for you Michael?

[michael-wirth]

Which DB and SCDF patch version are you using?

I remember an issue with Oracle DB Flyway scripts in SCDF 2 11.1. I didn't test any other DBs.

[ragovi]

I tested with Oracle DB and SCDF 2.11.3

[cppwfs]

Hello @ragovi, Please open a new issue. And if you could provide the following information it would be helpful.

Are you using 2.11.3-SNAPSHOT? The last release of SCDF was 2.11.2. Can you provide a stack trace of what you are seeing? How was the schema created? Did you migrate from a previous release of SCDF or starting fresh?
Are you using a schema created by a Spring Batch application?