search

spring-cloud / spring-cloud-dataflow

A microservices-based Streaming and Batch data processing in Cloud Foundry and Kubernetes
https://dataflow.spring.io
Apache License 2.0
1.11k stars 579 forks source link

CTR always times out waiting for boot3 tasks to start #5918

Open jnt0r opened 3 weeks ago

jnt0r commented 3 weeks ago

Description: When starting boot3 tasks as composed tasks the CTR fails waiting for the tasks to start because the boot3 tasks only write their status into the BOOT3TASK tables but the CTR checks the TASK_ tables. As I found in documentation and other comments it should be possible to run boot3 tasks with CTR. Am I right with this?

I found that CTR adds following parameter to the launched tasks "SPRING_CLOUD_TASK_PARENT-SCHEMA-TARGET". But I do not find this property documented in spring cloud task documentation. What is this about? Is there some additional work necessary on the custom task side? Is it possible to configure the CTR or the launched tasks to write/watch in both schemas?

Release versions: SCDF 2.11.3 and CTR 2.11.3

corneil commented 2 weeks ago

CTR should have no issue launch boot 3 tasks, we have acceptance tests where we orchestrate multiple boot 2.x and boot 3.x tasks without issue.

The property --spring.cloud.task.parent-schema-target ensure that task parents can properly be identified across the boot 2.x and boot 3.x task executions which have separate tables that we cannot mess with from data flow side. It has no specific meaning to the child task.

Do you have an more information on if the boot 3.x task is actually running? What platform are you using? Do you see any errors in the task logs?

jnt0r commented 2 weeks ago

I will try to deploy a composed task using the boot2 and boot3 samples (timestamp) and try to verify the behavior. For my understanding, we are using spring batch and spring cloud task as described in the documentation.

I was just wondering that I could not find the Property --spring.cloud.task.parent-schema-target described anywhere in the documentation of spring cloud task.

I am running SCDF on Kubernetes. The tasks complete successfully and I therefore see no errors. I also see the results of the actual task and batch job in SCDF, but the composed task runner fails to get the status. It just seems that my tasks ignore the parent-schema-target property.

jnt0r commented 2 weeks ago

Is there any specialty or important point to configure in my tasks to make them use the parent-schema-target, or should it work out of the box? Is there any possibility I deactivated that functionality by mistake?

jnt0r commented 2 weeks ago

I just tested it with the timestamp-batch-task in Version 3.0.0 and the CTR also failed to check if the task started correctly. Again, the timestamp task completed correctly and I found no obvious log with an error. The task did not log anything related with "parent", "target schema" or so on.

I had to build the docker image by myself as I had to add another database driver but did not change the configuration of the task.

corneil commented 2 weeks ago

Is there any specialty or important point to configure in my tasks to make them use the parent-schema-target, or should it work out of the box? Is there any possibility I deactivated that functionality by mistake?

The task doesn't need to do anything about the property in question. Since the arguments are logged we can find the combination of parent id and parent schema to find all of the ctr process children.

corneil commented 2 weeks ago

I just tested it with the timestamp-batch-task in Version 3.0.0 and the CTR also failed to check if the task started correctly. Again, the timestamp task completed correctly and I found no obvious log with an error. The task did not log anything related with "parent", "target schema" or so on.

I had to build the docker image by myself as I had to add another database driver but did not change the configuration of the task.

You can follow these instructions for adding a database driver to an existing jar and then build the container using Paketo pack.

corneil commented 2 weeks ago

I will try to deploy a composed task using the boot2 and boot3 samples (timestamp) and try to verify the behavior. For my understanding, we are using spring batch and spring cloud task as described in the documentation.

I was just wondering that I could not find the Property --spring.cloud.task.parent-schema-target described anywhere in the documentation of spring cloud task.

I am running SCDF on Kubernetes. The tasks complete successfully and I therefore see no errors. I also see the results of the actual task and batch job in SCDF, but the composed task runner fails to get the status. It just seems that my tasks ignore the parent-schema-target property.

Can you use kail to retrieve logs from all pods using kail -n <scdf-namespace> | tee scdf.log from just before you deploy the task?

jnt0r commented 2 weeks ago

I just tested it with the timestamp-batch-task in Version 3.0.0 and the CTR also failed to check if the task started correctly. Again, the timestamp task completed correctly and I found no obvious log with an error. The task did not log anything related with "parent", "target schema" or so on. I had to build the docker image by myself as I had to add another database driver but did not change the configuration of the task.

You can follow these instructions for adding a database driver to an existing jar and then build the container using Paketo pack.

Is there any special configuration when building with Paketo buildpacks that I am missing? Our build process depends on Dockerfiles for now, and I did not test if I am able to build using pack.

corneil commented 2 weeks ago

I just tested it with the timestamp-batch-task in Version 3.0.0 and the CTR also failed to check if the task started correctly. Again, the timestamp task completed correctly and I found no obvious log with an error. The task did not log anything related with "parent", "target schema" or so on. I had to build the docker image by myself as I had to add another database driver but did not change the configuration of the task.

You can follow these instructions for adding a database driver to an existing jar and then build the container using Paketo pack.

Is there any special configuration when building with Paketo buildpacks that I am missing? Our build process depends on Dockerfiles for now, and I did not test if I am able to build using pack.

Install pack using these instructions https://buildpacks.io/docs/for-platform-operators/how-to/integrate-ci/pack/#pack-cli and follow the guide in the docs.