numaproj / numaflow

Kubernetes-native platform to run massively parallel data/streaming jobs
https://numaflow.numaproj.io
Apache License 2.0
1.01k stars 98 forks source link

Avoid race conditions generated by frequent pipeline deletion/creation #1770

Open whynowy opened 1 week ago

whynowy commented 1 week ago

Summary

When a pipeline is created, a job is generated to create buffers in ISBService, similarly, when a pipeline is deleted, another job is generated to clean up buffers from ISBService. However, the job/pod takes time to start and finish, for example, pod needs to wait for available resources to start, if someone does frequent pipeline deletion and recreation, it is possible that the creation job/pod starts and finishes earlier than the clean up job/pod, and that will cause problems.

Figure out if there's a way to prevent this from happening.


Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.