search

spring-cloud / spring-cloud-deployer-kubernetes

The Spring Cloud Deployer implementation for Kubernetes
Apache License 2.0
157 stars 98 forks source link

Pods keep running forever #292

Closed jsa4000 closed 5 years ago

jsa4000 commented 5 years ago

Hello,

I am experiencing an issue running jobs using Spring Batch processing with partitions (master/slave) using spring-cloud-deployer-kubernetes.

To put some context, following is the environment I have used:

When I launch the task, the job is processed correctly, however the Pods still running forever. Both master and slave, hangs on o.s.c.t.r.support.SimpleTaskRepository : Updating: TaskExecution with executionId=9 with the following.... However, when I run the same job using the local deployer, all instanciated java applications (master and slaves) exit successfully. I have tested tasklets and run correctly in k8s, showing the Completed (0/1) status, however using partitions is not the same expected behaviour.

The steps are very straightforward:

app register --type task --name batch-uploader-app --uri docker:jsa4000/dataflow-batch-uploader-k8s:0.0.1-SNAPSHOT
task create batch-uploader-task --definition "batch-uploader-app"
task launch batch-uploader-task --arguments "--spring.profiles.active=k8s,master"

Following is the output using the shell.


dataflow:>job execution display --id 3
╔═══════════════════════════════════════════╤════════════════════════════╗
║                    Key                    │           Value            ║
╠═══════════════════════════════════════════╪════════════════════════════╣
║Job Execution Id                           │3                           ║
║Task Execution Id                          │5                           ║
║Task Instance Id                           │3                           ║
║Job Name                                   │job                         ║
║Create Time                                │Fri Apr 05 09:37:17 GMT 2019║
║Start Time                                 │Fri Apr 05 09:37:17 GMT 2019║
║End Time                                   │Fri Apr 05 09:38:03 GMT 2019║
║Running                                    │false                       ║
║Stopping                                   │false                       ║
║Step Execution Count                       │7                           ║
║Execution Status                           │COMPLETED                   ║
║Exit Status                                │COMPLETED                   ║
║Exit Message                               │                            ║
║Definition Status                          │Created                     ║
║Job Parameters                             │                            ║
║-spring.cloud.task.executionid(STRING)     │5                           ║
║run.id(LONG)                               │3                           ║
║-spring.datasource.username(STRING)        │******                      ║
║-spring.profiles.active(STRING)            │k8s,master                  ║
║-spring.cloud.task.name(STRING)            │batch-uploader-task         ║
║-spring.datasource.password(STRING)        │******                      ║
║-spring.datasource.driverClassName(STRING) │org.postgresql.Driver       ║
║-spring.datasource.url(STRING)             │******                      ║
╚═══════════════════════════════════════════╧════════════════════════════╝

However, when I print out the pods currently running (here, I have run the same job more that once):

kubectl get pods

NAME                             READY     STATUS      RESTARTS   AGE
batch-uploader-task-1kege0xx6r   1/1       Running     0          8m
batch-uploader-task-1kege3dmz5   1/1       Running     0          44m
batch-uploader-task-2k1y1gv69z   1/1       Running     0          28m
batch-uploader-task-ke7d70mrny   1/1       Running     0          12m
batch-uploader-task-qneje9z30r   1/1       Running     0          41m
db-migration-xwb7h               0/1       Completed   0          1h
minio-7d8869cc85-p5k5c           1/1       Running     0          1h
notifier-task-3lvmv15l70         0/1       Completed   0          48m
notifier-task-8nmwm9vg1o         0/1       Completed   0          54m
postgres-7ddd5ffdf8-rh72p        1/1       Running     0          1h
rest-service-6d98576859-pd89l    1/1       Running     0          1h
scdf-server-58cb976466-9gqrv     1/1       Running     0          1h
uploaderjobtask-3k7m7vdv4m       1/1       Running     0          7m
uploaderjobtask-5keye0gkww       1/1       Running     0          28m
uploaderjobtask-5keyepzvmx       1/1       Running     0          7m
uploaderjobtask-6kg9gyvm3r       1/1       Running     0          41m
uploaderjobtask-8k3534mn6e       1/1       Running     0          28m
uploaderjobtask-947w75n6qg       1/1       Running     0          41m
uploaderjobtask-eq2w25r6x6       1/1       Running     0          41m
uploaderjobtask-gxz0ze35ej       1/1       Running     0          41m
uploaderjobtask-jq7j7v0y5y       1/1       Running     0          7m
uploaderjobtask-jq7j7y72z5       1/1       Running     0          28m
uploaderjobtask-kgjrj9jv8z       1/1       Running     0          28m
uploaderjobtask-n8575wnk7p       1/1       Running     0          28m
uploaderjobtask-n8575wnknq       1/1       Running     0          28m
uploaderjobtask-wd3y3zr98k       1/1       Running     0          41m
uploaderjobtask-xjk9kw2d3p       1/1       Running     0          7m
uploaderjobtask-ygk3kwqyvz       1/1       Running     0          41m
...

And those are the logs:

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.1.3.RELEASE)

2019-04-05 07:36:02.977  INFO 1 --- [           main] com.example.uploader.BatchApplication    : Starting BatchApplication on batch-uploader-task-qneje9z30r with PID 1 (/app/classes started by root in /)
2019-04-05 07:36:02.982 DEBUG 1 --- [           main] com.example.uploader.BatchApplication    : Running with Spring Boot v2.1.3.RELEASE, Spring v5.1.5.RELEASE
2019-04-05 07:36:02.983  INFO 1 --- [           main] com.example.uploader.BatchApplication    : The following profiles are active: k8s,master
2019-04-05 07:36:04.420  INFO 1 --- [           main] faultConfiguringBeanFactoryPostProcessor : No bean named 'errorChannel' has been explicitly defined. Therefore, a default PublishSubscribeChannel will be created.
2019-04-05 07:36:04.432  INFO 1 --- [           main] faultConfiguringBeanFactoryPostProcessor : No bean named 'taskScheduler' has been explicitly defined. Therefore, a default ThreadPoolTaskScheduler will be created.
2019-04-05 07:36:04.440  INFO 1 --- [           main] faultConfiguringBeanFactoryPostProcessor : No bean named 'integrationHeaderChannelRegistry' has been explicitly defined. Therefore, a default DefaultHeaderChannelRegistry will be created.
2019-04-05 07:36:04.592  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.retry.annotation.RetryConfiguration' of type [org.springframework.retry.annotation.RetryConfiguration$$EnhancerBySpringCGLIB$$8f7fe9d4] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-04-05 07:36:04.644  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration$$EnhancerBySpringCGLIB$$99bc51af] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-04-05 07:36:04.689  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.integration.config.IntegrationManagementConfiguration' of type [org.springframework.integration.config.IntegrationManagementConfiguration$$EnhancerBySpringCGLIB$$4ae0cde] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-04-05 07:36:04.720  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'integrationDisposableAutoCreatedBeans' of type [org.springframework.integration.config.annotation.Disposables] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-04-05 07:36:04.749  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.task.batch.configuration.TaskBatchAutoConfiguration' of type [org.springframework.cloud.task.batch.configuration.TaskBatchAutoConfiguration$$EnhancerBySpringCGLIB$$1f2bfd3b] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-04-05 07:36:04.756  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.task.batch.listener.BatchEventAutoConfiguration' of type [org.springframework.cloud.task.batch.listener.BatchEventAutoConfiguration$$EnhancerBySpringCGLIB$$6782073e] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-04-05 07:36:05.480  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2019-04-05 07:36:05.534  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2019-04-05 07:36:05.534  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.16]
2019-04-05 07:36:05.554  INFO 1 --- [           main] o.a.catalina.core.AprLifecycleListener   : The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib]
2019-04-05 07:36:05.767  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2019-04-05 07:36:05.768  INFO 1 --- [           main] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 2706 ms
2019-04-05 07:36:06.477  INFO 1 --- [           main] c.e.uploader.config.MasterConfiguration  : Worker spring profile: k8s,worker
2019-04-05 07:36:06.581  INFO 1 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting...
2019-04-05 07:36:06.741  INFO 1 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Start completed.
2019-04-05 07:36:06.766 DEBUG 1 --- [           main] o.s.c.t.c.SimpleTaskAutoConfiguration    : Using org.springframework.cloud.task.configuration.DefaultTaskConfigurer TaskConfigurer
2019-04-05 07:36:06.767 DEBUG 1 --- [           main] o.s.c.t.c.DefaultTaskConfigurer          : No EntityManager was found, using DataSourceTransactionManager
2019-04-05 07:36:07.503  INFO 1 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
2019-04-05 07:36:07.791  INFO 1 --- [           main] o.s.b.c.r.s.JobRepositoryFactoryBean     : No database type set, using meta data indicating: POSTGRES
2019-04-05 07:36:07.826  INFO 1 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : No TaskExecutor has been set, defaulting to synchronous executor.
2019-04-05 07:36:08.060 DEBUG 1 --- [           main] o.s.c.t.r.s.TaskRepositoryInitializer    : Initializing task schema for postgresql database
2019-04-05 07:36:08.203  INFO 1 --- [           main] o.s.s.c.ThreadPoolTaskScheduler          : Initializing ExecutorService 'taskScheduler'
2019-04-05 07:36:08.286  INFO 1 --- [           main] o.s.i.endpoint.EventDrivenConsumer       : Adding {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2019-04-05 07:36:08.286  INFO 1 --- [           main] o.s.i.channel.PublishSubscribeChannel    : Channel 'UploaderJobTask.errorChannel' has 1 subscriber(s).
2019-04-05 07:36:08.286  INFO 1 --- [           main] o.s.i.endpoint.EventDrivenConsumer       : started _org.springframework.integration.errorLogger
2019-04-05 07:36:08.432 DEBUG 1 --- [           main] o.s.c.t.r.support.SimpleTaskRepository   : Starting: TaskExecution{executionId=4, parentExecutionId=null, exitCode=null, taskName='batch-uploader-task', startTime=Fri Apr 05 07:36:08 GMT 2019, endTime=null, exitMessage='null', externalExecutionId='null', errorMessage='null', arguments=[--spring.datasource.username=postgres, --spring.cloud.task.name=batch-uploader-task, --spring.datasource.url=jdbc:postgresql://postgres.default.svc.cluster.local:5432/dataflow, --spring.datasource.driverClassName=org.postgresql.Driver, --spring.datasource.password=password, --spring.profiles.active=k8s,master, --spring.cloud.task.executionid=4]}
2019-04-05 07:36:08.559  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2019-04-05 07:36:08.564  INFO 1 --- [           main] com.example.uploader.BatchApplication    : Started BatchApplication in 6.427 seconds (JVM running for 7.166)
2019-04-05 07:36:08.566  INFO 1 --- [           main] o.s.b.a.b.JobLauncherCommandLineRunner   : Running default command line with: [--spring.datasource.username=postgres, --spring.cloud.task.name=batch-uploader-task, --spring.datasource.url=jdbc:postgresql://postgres.default.svc.cluster.local:5432/dataflow, --spring.datasource.driverClassName=org.postgresql.Driver, --spring.datasource.password=password, --spring.profiles.active=k8s,master, --spring.cloud.task.executionid=4]
2019-04-05 07:36:08.715  INFO 1 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : Job: [SimpleJob: [name=job]] launched with the following parameters: [{-spring.cloud.task.executionid=4, run.id=2, -spring.datasource.username=postgres, -spring.profiles.active=k8s,master, -spring.cloud.task.name=batch-uploader-task, -spring.datasource.password=password, -spring.datasource.driverClassName=org.postgresql.Driver, -spring.datasource.url=jdbc:postgresql://postgres.default.svc.cluster.local:5432/dataflow}]
2019-04-05 07:36:08.725  INFO 1 --- [           main] o.s.c.t.b.l.TaskBatchExecutionListener   : The job execution id 2 was run within the task execution 4
2019-04-05 07:36:08.737  INFO 1 --- [           main] o.s.batch.core.job.SimpleStepHandler     : Executing step: [masterStep]
2019-04-05 07:36:08.769  INFO 1 --- [           main] c.e.u.batch.MultiResourcePartitioner     : Creating the partitions. gridSize=1
2019-04-05 07:36:08.770  INFO 1 --- [           main] c.e.u.batch.MultiResourcePartitioner     : Adding dataflow-bucket:first_resource resource to partition
2019-04-05 07:36:08.770  INFO 1 --- [           main] c.e.u.batch.MultiResourcePartitioner     : Adding dataflow-bucket:second_resource resource to partition
2019-04-05 07:36:08.770  INFO 1 --- [           main] c.e.u.batch.MultiResourcePartitioner     : Adding dataflow-bucket:third_resource resource to partition
2019-04-05 07:36:08.770  INFO 1 --- [           main] c.e.u.batch.MultiResourcePartitioner     : Adding dataflow-bucket:fourth_resource resource to partition
2019-04-05 07:36:08.770  INFO 1 --- [           main] c.e.u.batch.MultiResourcePartitioner     : Adding dataflow-bucket:fifth_resource resource to partition
2019-04-05 07:36:08.770  INFO 1 --- [           main] c.e.u.batch.MultiResourcePartitioner     : Adding dataflow-bucket:sixth_resource resource to partition
2019-04-05 07:36:08.797 DEBUG 1 --- [           main] o.s.c.t.b.p.DeployerPartitionHandler     : 6 partitions were returned
2019-04-05 07:36:09.511  INFO 1 --- [           main] o.s.c.d.s.k.DefaultContainerFactory      : Using Docker image: jsa4000/dataflow-batch-uploader-k8s:0.0.1-SNAPSHOT
2019-04-05 07:36:09.511  INFO 1 --- [           main] o.s.c.d.s.k.DefaultContainerFactory      : Using Docker entry point style: exec
2019-04-05 07:36:10.425  INFO 1 --- [           main] o.s.c.d.s.k.DefaultContainerFactory      : Using Docker image: jsa4000/dataflow-batch-uploader-k8s:0.0.1-SNAPSHOT
2019-04-05 07:36:10.425  INFO 1 --- [           main] o.s.c.d.s.k.DefaultContainerFactory      : Using Docker entry point style: exec
2019-04-05 07:36:21.005  INFO 1 --- [           main] o.s.c.d.s.k.DefaultContainerFactory      : Using Docker image: jsa4000/dataflow-batch-uploader-k8s:0.0.1-SNAPSHOT
2019-04-05 07:36:21.006  INFO 1 --- [           main] o.s.c.d.s.k.DefaultContainerFactory      : Using Docker entry point style: exec
2019-04-05 07:36:21.558  INFO 1 --- [           main] o.s.c.d.s.k.DefaultContainerFactory      : Using Docker image: jsa4000/dataflow-batch-uploader-k8s:0.0.1-SNAPSHOT
2019-04-05 07:36:21.558  INFO 1 --- [           main] o.s.c.d.s.k.DefaultContainerFactory      : Using Docker entry point style: exec
2019-04-05 07:36:32.124  INFO 1 --- [           main] o.s.c.d.s.k.DefaultContainerFactory      : Using Docker image: jsa4000/dataflow-batch-uploader-k8s:0.0.1-SNAPSHOT
2019-04-05 07:36:32.124  INFO 1 --- [           main] o.s.c.d.s.k.DefaultContainerFactory      : Using Docker entry point style: exec
2019-04-05 07:36:32.556  INFO 1 --- [           main] o.s.c.d.s.k.DefaultContainerFactory      : Using Docker image: jsa4000/dataflow-batch-uploader-k8s:0.0.1-SNAPSHOT
2019-04-05 07:36:32.556  INFO 1 --- [           main] o.s.c.d.s.k.DefaultContainerFactory      : Using Docker entry point style: exec
2019-04-05 07:36:52.940  INFO 1 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : Job: [SimpleJob: [name=job]] completed with the following parameters: [{-spring.cloud.task.executionid=4, run.id=2, -spring.datasource.username=postgres, -spring.profiles.active=k8s,master, -spring.cloud.task.name=batch-uploader-task, -spring.datasource.password=password, -spring.datasource.driverClassName=org.postgresql.Driver, -spring.datasource.url=jdbc:postgresql://postgres.default.svc.cluster.local:5432/dataflow}] and the following status: [COMPLETED]
2019-04-05 07:36:52.943 DEBUG 1 --- [           main] o.s.c.t.r.support.SimpleTaskRepository   : Updating: TaskExecution with executionId=4 with the following {exitCode=0, endTime=Fri Apr 05 07:36:52 GMT 2019, exitMessage='null', errorMessage='null'}
  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.1.3.RELEASE)

2019-04-05 07:36:35.740  INFO 1 --- [           main] com.example.uploader.BatchApplication    : Starting BatchApplication on uploaderjobtask-ygk3kwqyvz with PID 1 (/app/classes started by root in /)
2019-04-05 07:36:35.745 DEBUG 1 --- [           main] com.example.uploader.BatchApplication    : Running with Spring Boot v2.1.3.RELEASE, Spring v5.1.5.RELEASE
2019-04-05 07:36:35.746  INFO 1 --- [           main] com.example.uploader.BatchApplication    : The following profiles are active: k8s,worker
2019-04-05 07:36:37.685  INFO 1 --- [           main] faultConfiguringBeanFactoryPostProcessor : No bean named 'errorChannel' has been explicitly defined. Therefore, a default PublishSubscribeChannel will be created.
2019-04-05 07:36:37.700  INFO 1 --- [           main] faultConfiguringBeanFactoryPostProcessor : No bean named 'taskScheduler' has been explicitly defined. Therefore, a default ThreadPoolTaskScheduler will be created.
2019-04-05 07:36:37.709  INFO 1 --- [           main] faultConfiguringBeanFactoryPostProcessor : No bean named 'integrationHeaderChannelRegistry' has been explicitly defined. Therefore, a default DefaultHeaderChannelRegistry will be created.
2019-04-05 07:36:37.934  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.retry.annotation.RetryConfiguration' of type [org.springframework.retry.annotation.RetryConfiguration$$EnhancerBySpringCGLIB$$4e4faabe] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-04-05 07:36:38.003  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration$$EnhancerBySpringCGLIB$$588c1299] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-04-05 07:36:38.050  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.integration.config.IntegrationManagementConfiguration' of type [org.springframework.integration.config.IntegrationManagementConfiguration$$EnhancerBySpringCGLIB$$c37dcdc8] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-04-05 07:36:38.071  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'integrationDisposableAutoCreatedBeans' of type [org.springframework.integration.config.annotation.Disposables] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2019-04-05 07:36:38.829  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2019-04-05 07:36:38.981  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2019-04-05 07:36:38.984  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.16]
2019-04-05 07:36:39.022  INFO 1 --- [           main] o.a.catalina.core.AprLifecycleListener   : The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib]
2019-04-05 07:36:39.282  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2019-04-05 07:36:39.283  INFO 1 --- [           main] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 3443 ms
2019-04-05 07:36:40.141  INFO 1 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting...
2019-04-05 07:36:40.346  INFO 1 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Start completed.
2019-04-05 07:36:40.374 DEBUG 1 --- [           main] o.s.c.t.c.SimpleTaskAutoConfiguration    : Using org.springframework.cloud.task.configuration.DefaultTaskConfigurer TaskConfigurer
2019-04-05 07:36:40.374 DEBUG 1 --- [           main] o.s.c.t.c.DefaultTaskConfigurer          : No EntityManager was found, using DataSourceTransactionManager
2019-04-05 07:36:41.195  INFO 1 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
2019-04-05 07:36:41.507  INFO 1 --- [           main] o.s.b.c.r.s.JobRepositoryFactoryBean     : No database type set, using meta data indicating: POSTGRES
2019-04-05 07:36:41.551  INFO 1 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : No TaskExecutor has been set, defaulting to synchronous executor.
2019-04-05 07:36:41.990  INFO 1 --- [           main] o.s.s.c.ThreadPoolTaskScheduler          : Initializing ExecutorService 'taskScheduler'
2019-04-05 07:36:42.073  INFO 1 --- [           main] o.s.i.endpoint.EventDrivenConsumer       : Adding {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2019-04-05 07:36:42.073  INFO 1 --- [           main] o.s.i.channel.PublishSubscribeChannel    : Channel 'UploaderJobTask.errorChannel' has 1 subscriber(s).
2019-04-05 07:36:42.074  INFO 1 --- [           main] o.s.i.endpoint.EventDrivenConsumer       : started _org.springframework.integration.errorLogger
2019-04-05 07:36:42.237 DEBUG 1 --- [           main] o.s.c.t.r.support.SimpleTaskRepository   : Creating: TaskExecution{executionId=0, parentExecutionId=4, exitCode=null, taskName='batch-uploader-task_job_slaveStep:partition1', startTime=Fri Apr 05 07:36:42 GMT 2019, endTime=null, exitMessage='null', externalExecutionId='null', errorMessage='null', arguments=[--spring.profiles.active=k8s,worker, --spring.cloud.task.initialize.enable=false, --spring.batch.initializer.enabled=false, --spring.cloud.task.job-execution-id=2, --spring.cloud.task.step-execution-id=12, --spring.cloud.task.step-name=slaveStep, --spring.cloud.task.name=batch-uploader-task_job_slaveStep:partition1, --spring.cloud.task.parentExecutionId=4]}
2019-04-05 07:36:42.339  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2019-04-05 07:36:42.345  INFO 1 --- [           main] com.example.uploader.BatchApplication    : Started BatchApplication in 7.561 seconds (JVM running for 8.344)
2019-04-05 07:36:42.347  INFO 1 --- [           main] o.s.b.a.b.JobLauncherCommandLineRunner   : Running default command line with: [--spring.profiles.active=k8s,worker, --spring.cloud.task.initialize.enable=false, --spring.batch.initializer.enabled=false, --spring.cloud.task.job-execution-id=2, --spring.cloud.task.step-execution-id=12, --spring.cloud.task.step-name=slaveStep, --spring.cloud.task.name=batch-uploader-task_job_slaveStep:partition1, --spring.cloud.task.parentExecutionId=4]
2019-04-05 07:36:42.481 DEBUG 1 --- [           main] o.s.c.t.b.p.DeployerStepExecutionHandler : Executing step slaveStep:partition1 with step execution id 12 and job execution id 2
This tasklet ran partition: dataflow-bucket:second_resource
2019-04-05 07:36:42.574 DEBUG 1 --- [           main] o.s.c.t.r.support.SimpleTaskRepository   : Updating: TaskExecution with executionId=9 with the following {exitCode=0, endTime=Fri Apr 05 07:36:42 GMT 2019, exitMessage='null', errorMessage='null'}

The code can be seen in my github repo.

package com.example.uploader;

import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.task.configuration.EnableTask;

@EnableTask
@EnableBatchProcessing
@SpringBootApplication
public class BatchApplication {
    public static void main(String[] args) {
        SpringApplication.run(BatchApplication.class, args);
    }

}

package com.example.uploader.config;

import com.example.uploader.batch.ResourceLoaderResolver;
import lombok.extern.slf4j.Slf4j;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.explore.JobExplorer;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.core.partition.PartitionHandler;
import org.springframework.batch.core.partition.support.Partitioner;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.cloud.deployer.spi.task.TaskLauncher;
import org.springframework.cloud.task.batch.partition.DeployerPartitionHandler;
import org.springframework.cloud.task.batch.partition.NoOpEnvironmentVariablesProvider;
import org.springframework.cloud.task.batch.partition.PassThroughCommandLineArgsProvider;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
import org.springframework.core.io.Resource;
import org.springframework.core.io.ResourceLoader;

import java.util.ArrayList;
import java.util.List;

@Slf4j
@Profile("master")
@Configuration
public class MasterConfiguration {

    @Value("${batch.max-workers:1}")
    private int maxWorkers;

    @Bean
    public Step masterStep(StepBuilderFactory stepBuilderFactory,
                           Partitioner partitioner,
                           PartitionHandler partitionHandler) {
        return stepBuilderFactory.get("masterStep")
                .partitioner("slaveStep", partitioner)
                .partitionHandler(partitionHandler)
                .build();
    }

    @Bean
    public DeployerPartitionHandler partitionHandler(@Value("${batch.worker-app}") String resourceLocation,
                                                     @Value("${spring.application.name}") String applicationName,
                                                     @Value("${spring.profiles.active}") String activeProfile,
                                                     ApplicationContext context,
                                                     TaskLauncher taskLauncher,
                                                     JobExplorer jobExplorer,
                                                     ResourceLoaderResolver resolver) {
        ResourceLoader resourceLoader = resolver.get(resourceLocation);
        Resource resource = resourceLoader.getResource(resourceLocation);
        DeployerPartitionHandler partitionHandler = new DeployerPartitionHandler(taskLauncher,
                jobExplorer,resource,"slaveStep");

        log.info("Worker spring profile: " + activeProfile.replace("master","worker"));

        List<String> commandLineArgs = new ArrayList<>(3);
        commandLineArgs.add("--spring.profiles.active=" + activeProfile.replace("master","worker"));
        commandLineArgs.add("--spring.cloud.task.initialize.enable=false");
        commandLineArgs.add("--spring.batch.initializer.enabled=false");

        partitionHandler.setCommandLineArgsProvider(new PassThroughCommandLineArgsProvider(commandLineArgs));
        partitionHandler.setEnvironmentVariablesProvider(new NoOpEnvironmentVariablesProvider());
        partitionHandler.setMaxWorkers(maxWorkers);
        partitionHandler.setApplicationName(applicationName);

        return partitionHandler;
    }

    @Bean
    public Job job(JobBuilderFactory jobBuilderFactory) {
        return jobBuilderFactory.get("job")
                .incrementer(new RunIdIncrementer())
                .start(masterStep( null, null, null))
                .build();
    }
}

package com.example.uploader.config;

import lombok.extern.slf4j.Slf4j;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepScope;
import org.springframework.batch.core.explore.JobExplorer;
import org.springframework.batch.core.repository.JobRepository;
import org.springframework.batch.core.step.tasklet.Tasklet;
import org.springframework.batch.repeat.RepeatStatus;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.cloud.task.batch.partition.DeployerStepExecutionHandler;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;

@Slf4j
@Profile("worker")
@Configuration
public class SlaveConfiguration {

    @Bean
    public DeployerStepExecutionHandler stepExecutionHandler(ApplicationContext context,JobExplorer jobExplorer,
                                                             JobRepository jobRepository ) {
        return new DeployerStepExecutionHandler(context, jobExplorer, jobRepository);
    }

    @Bean
    @StepScope
    public Tasklet workerTasklet(final @Value("#{stepExecutionContext['fileName']}")String fileName) {
        return (contribution, chunkContext) -> {
            System.out.println("This tasklet ran partition: " + fileName);
            return RepeatStatus.FINISHED;
        };
    }

    @Bean
    public Step slaveStep(StepBuilderFactory stepBuilderFactory) throws Exception {
        return stepBuilderFactory.get("slaveStep")
                .tasklet(workerTasklet(null))
                .build();
    }
}

spring:
  profiles:
    active: local,master
  application:
    name: UploaderJobTask
  batch:
    initialize-schema: always

batch:
  max-threads: 1
  max-workers: 1
  resourcesPath: dataflow-bucket
  tempPath: /tmp/data
  filePattern: "*.csv"

logging:
  level:
    root: INFO
    org.springframework.cloud.task: DEBUG
    org.springframework.web: INFO
    com.example: DEBUG

management:
  endpoints:
    web.exposure.include: health,beans,env,info

---
spring:
  profiles: local
  datasource:
    url: jdbc:postgresql://dockerhost:5432/dataflow
    username: postgres
    password: password
    driver-class-name: org.postgresql.Driver

batch:
  max-threads: 2
  max-workers: 2
  worker-app: maven://com.example:batch-uploader-k8s:0.0.1-SNAPSHOT
  storage:
    url: http://dockerhost:9000
    accessKey: minio
    secretKey: password

---
spring:
  profiles: docker
  datasource:
    url: jdbc:postgresql://postgres:5432/dataflow
    username: postgres
    password: password
    driver-class-name: org.postgresql.Driver

batch:
  max-threads: 4
  max-workers: 1
  worker-app: docker:jsa4000/dataflow-batch-uploader-k8s:0.0.1-SNAPSHOT
  storage:
    url: http://minio:9000
    accessKey: minio
    secretKey: password

---
spring:
  profiles: k8s
  datasource:
    url: jdbc:postgresql://postgres.default.svc.cluster.local:5432/dataflow
    username: postgres
    password: password
    driver-class-name: org.postgresql.Driver

batch:
  max-threads: 2
  max-workers: 2
  worker-app: docker:jsa4000/dataflow-batch-uploader-k8s:0.0.1-SNAPSHOT
  storage:
    url: http://minio.default.svc.cluster.local:9000
    accessKey: minio
    secretKey: password
plugins {
    id 'org.springframework.boot' version '2.1.3.RELEASE'
    id 'com.google.cloud.tools.jib' version '1.0.2'
    id 'java'
}

apply plugin: 'io.spring.dependency-management'
apply plugin: 'maven'

group = 'com.example'
version = '0.0.1-SNAPSHOT'
sourceCompatibility = '1.8'

configurations {
    compileOnly {
        extendsFrom annotationProcessor
    }
}

repositories {
    mavenCentral()
    mavenLocal()
}

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-batch'
    implementation 'org.springframework.boot:spring-boot-starter-aop'
    implementation 'org.springframework.batch:spring-batch-integration'
    implementation 'org.springframework.cloud:spring-cloud-starter-task:2.1.1.RELEASE'
    implementation 'org.springframework.cloud:spring-cloud-task-dependencies:2.1.1.RELEASE'
    //implementation 'org.springframework.cloud:spring-cloud-deployer-local:2.0.0.RELEASE'
    implementation 'org.springframework.cloud:spring-cloud-deployer-kubernetes:2.0.1.RELEASE'
    runtimeOnly 'org.postgresql:postgresql'
    compileOnly 'org.projectlombok:lombok'
    annotationProcessor 'org.projectlombok:lombok'
    testImplementation 'org.springframework.boot:spring-boot-starter-test'
    testImplementation 'org.springframework.batch:spring-batch-test'
    testRuntimeOnly 'com.h2database:h2'
}

jib {
    from {
        image = 'openjdk:8u201-jre-alpine'
    }
    to {
        image = "registry.hub.docker.com/jsa4000/dataflow-${project.name}"
        credHelper = 'osxkeychain'
        tags = [version, 'latest']
    }
    container {
        jvmFlags = ['-XX:+UnlockExperimentalVMOptions',
                    '-XX:+UseCGroupMemoryLimitForHeap',
                    '-XX:+HeapDumpOnOutOfMemoryError',
                    '-XX:MaxRAMFraction=2']
        useCurrentTimestamp = true
        labels = [ownerName: 'jsa4000',
                  projectName: project.name,
                  creationDate: System.nanoTime().toString()]
        //format = 'OCI'
    }
}

Thanks in advance

Javier

chrisjs commented 5 years ago

@jsa4000 in your application properties, can you try setting spring.cloud.task.closecontext_enabled to true? (https://docs.spring.io/spring-cloud-task/docs/current-SNAPSHOT/reference/html/features-lifecycle.html)

jsa4000 commented 5 years ago

@chrisjs Thanks for your response.

That is exactly what I did, however I dunno the drawbacks of doing that since it is not enabled by default. It seems more like a workaround rathen a solution.

If it is not the case then this flag surely works for me. I would put into the documentation for everyone else. :)

chrisjs commented 5 years ago

@jsa4000 https://spring.io/blog/2018/02/14/spring-cloud-task-2-0-0-m3-is-now-available in the "Update To An Important Default" section provides a little more info, where true used to be the default. as its a spring cloud task property, its located in the task docs link above. for additional details you may want to ping on the task gitter channel: https://gitter.im/spring-cloud/spring-cloud-task

jsa4000 commented 5 years ago

Thanks for the links and for the support. Really appreciated.