search

spring-cloud / spring-cloud-dataflow

A microservices-based Streaming and Batch data processing in Cloud Foundry and Kubernetes
https://dataflow.spring.io
Apache License 2.0
1.11k stars 583 forks source link

CI/CD for Scheduled Tasks #4996

Open mileshbhana opened 2 years ago

mileshbhana commented 2 years ago

Hi,

Currently using SCDF to co-ordinate an Integration Engine in which our tasks are connectors between our system and external parties. This is deployed to a EKS environment.

We have these running as Scheduled Jobs.

I know that CI/CD is not currently support for Scheduled Jobs.

We are using another script that deletes all the scheduled jobs that match the updated application and reinstates them with the new version.

What are your recommended alternatives if the underlying task application changes?

cppwfs commented 2 years ago

SCDF needs ability to update schedules when a user updates a task's version. Currently SCDF expects that once a schedule has been created it will be managed fully by the scheduler and thus does not store metadata about the schedule (except for what is returned by the scheduling system Cronjobs).

markpollack commented 2 years ago

The workaround is to delete task or the schedule. https://dataflow.spring.io/docs/batch-developer-guides/continuous-deployment/cd-basics/#scheduling-on-kubernetes

A 'task upgrade' feature would be nice, but we don't have plans for that at the moment. I'll leave this issue open as a feature request.

markpollack commented 2 years ago

Also mentioned here - https://github.com/spring-cloud/spring-cloud-dataflow/issues/4845