Switching a Pipeline's ISBService results in Pipeline Containers stuck in Init stage

[juliev0]

Describe the bug

I'm not sure whether or not this is a bug or expected behavior. Basically, the Pipeline Pods can't start after changing the interStepBufferServiceName.

This occurs both when I first attempt to pause the pipeline, and also when I don't.

To Reproduce Steps to reproduce the behavior:

1. Create 2 basic Jetstream ISBServices (in my case I was using jetstream 2.10.3) with different names on the same namespace
2. Create simple pipeline setting the interstepBufferServiceName to one of them
3. Optional step: Pause pipeline by setting lifecycle.desiredPhase=Paused and wait for status.Phase=Paused. Note just Daemon Pod is running.
4. Update pipeline's interStepBufferServiceName to the other one
5. Any Pods that are running restart and have error in "init" container (in the case of paused pipeline, it's just the Daemon Pod):

   {"level":"info","ts":"2024-07-31T04:33:27.481863765Z","logger":"numaflow.isbsvc-validate","caller":"commands/isbsvc_validate.go:81","msg":"Buffers, buckets and side inputs store might have not been created yet, will retry if the limit is not reached","pipeline":"simple-pipeline","error":"failed to query information of stream \"example-namespace-simple-pipeline-cat-0\", nats: stream not found"}
   {"level":"info","ts":"2024-07-31T04:33:32.545809559Z","logger":"numaflow.isbsvc-validate","caller":"commands/isbsvc_validate.go:81","msg":"Buffers, buckets and side inputs store might have not been created yet, will retry if the limit is not reached","pipeline":"simple-pipeline","error":"failed to query information of stream \"example-namespace-simple-pipeline-cat-0\", nats: stream not found"}
   {"level":"info","ts":"2024-07-31T04:33:43.258374939Z","logger":"numaflow.isbsvc-validate","caller":"commands/isbsvc_validate.go:81","msg":"Buffers, buckets and side inputs store might have not been created yet, will retry if the limit is not reached","pipeline":"simple-pipeline","error":"failed to query information of stream \"example-namespace-simple-pipeline-cat-0\", nats: stream not found"}

Expected behavior Pipeline is able to resume running on the new ISBService

Screenshots If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• Kubernetes: Client Version: v1.29.1 Server Version: v1.28.8+k3s1
• Numaflow: quay.io/numaio/numaflow-rc:v0.01 (released within the last week)

Additional context Add any other context about the problem here.

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We often sort issues this way to know what to prioritize.

For quick help and support, join our slack channel.

[whynowy]

This operation should not be allowed.

[juliev0]

I see. I will close this then. Thanks.