search

spring-cloud / spring-cloud-dataflow

A microservices-based Streaming and Batch data processing in Cloud Foundry and Kubernetes
https://dataflow.spring.io
Apache License 2.0
1.1k stars 578 forks source link

Add container auto-cleanup option as a param on SCDF server config #4844

Open nightswimmings opened 2 years ago

nightswimmings commented 2 years ago

After the implementation of the shell cleanup commands, it would be nice if there was a way to set up SCDF with a param in such a way that it handles the expiration/cleanup of finished pods history in the background. Perhaps by setting defaultly things like .spec.successfulJobsHistoryLimit and .spec.failedJobsHistoryLimit (kubernetes). Would be nice it has the shape of a param like:

Would be even nicer if this could be overridden task-wise

What do you think?

markpollack commented 2 years ago

Hi. This seems quite reasonable. BTW, this features in k8s is related to this - https://kubernetes.io/docs/concepts/workloads/controllers/ttlafterfinished/ I've added this to our general backlog.

gertwillems commented 7 months ago

any idea if/when this would be implemented? Would be usefull for us to downsize the etcd pressure from running alot of jobs/cronjobs.

cppwfs commented 7 months ago

We are currently working on SCDF 3.x, and we will be resolving the TTL issue for K8's jobs. We will be announcing a release date soon, so stay tuned!