e2e test for Pause Feature

[juliev0]

Summary

We will want to confirm that:

• updating a Pipeline causes Pipeline to follow this sequence of events: pause, update, run
• updating an ISBService causes all Pipelines using it to follow this sequence of events: pause all pipelines, wait for them to be paused, ISBService updated, ISBService completely reconciled (i.e. StatefulSet has been updated with all new Pods), pipelines running
• updating a Numaflow Controller causes all Pipelines on namespace to follow this sequence of events: pause all pipelines, wait for them to be paused, Numaflow Controller Deployment updated and reconciled (created a new ReplicaSet which has created new Pods which are running), pipelines running

We need to see if it is a challenge to check Pipeline Phase for "Paused" quickly enough before it becomes running. That could happen if we are simply doing a "Get" periodically on the Pipeline - we could miss it. Best approach is probably to do a Watch and receive an update that we can check every time. If that approach doesn't work for some reason, we can use "kubernetes events" to check.

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[juliev0]

I'm torn between having a separate test for this vs just taking our existing test and adding a few if ${DATA_LOSS_PREVENTION} env var set type steps. I don't want to have a bunch of redundant code in 2 places after all.

It kind of sounds like in the future we may not even support the case of basic CRUD, so perhaps we do for now add if ${DATA_LOSS_PREVENTION} env var set steps to check for pausing to the existing test?