spring-cloud / spring-cloud-dataflow

A microservices-based Streaming and Batch data processing in Cloud Foundry and Kubernetes
https://dataflow.spring.io
Apache License 2.0
1.11k stars 581 forks source link

feat: Add `suspend` API to safely delete scheduled tasks after execution is complete #5948

Open injae-kim opened 2 days ago

injae-kim commented 2 days ago

Problem description:

Hi! currently we migrate batch system to spring-cloud-dataflow.

SCDF doesn't support to update cronjob, workaround is to delete&re-create schedule (issue)

We use many scheduled tasks on spring-cloud-dataflow and I found that scheduled task doesn't support update. And I found that update cronJob is not recommended by k8s side too (issue, issue2)

But, we need some way to update (or, at least suspend) scheduled tasks on below case.

  1. Register scheduled task with app v1
  2. ✅ After we deploy new app v2, need to suspend(not delete immediately, wait until complete) scheduled tasks v1 and re-create with v2

We want to safe-delete scheduled task v1 after complete to run cronJob that currently running, so need suspend instead of delete (which immediately delete cronJob)

Solution description:

// SCDF SchdulerService
public void suspend(..) {
  ..
}       

Add suspend API on DefaultSchedulerService and KubernetesScheduler.

// CronJobSpec
    public void setSuspend(Boolean suspend) {
        this.suspend = suspend;
    }

Then we can suspend v1 scheduled task by setSuspend on CronJobSpec.

Description of alternatives:

We want to safe-delete(delete after complete to run cronJob that currently running), so we can't use delete and need suspend or update.

But update is not recommended on k8s side so suspend API is proper solution I think!

Additional context:

X

injae-kim commented 2 days ago

We need this suspend API now for our service! PTAL 🙇

If you think this issue is not resolved soon, please tell me~ I'll create PR!