search

numaproj / numaflow

Kubernetes-native platform to run massively parallel data/streaming jobs
https://numaflow.numaproj.io
Apache License 2.0
1.01k stars 98 forks source link

clean up pod hangs after user fixes the ISB specification #1751

Closed KeranYang closed 4 weeks ago

KeranYang commented 1 month ago

Describe the bug

Numaflow clean up job pod stays in CreateContainerConfigError with error Warning Failed 2s (x7 over 53s) kubelet Error: secret "isbsvc-default-js-client-auth" not found.

If a cleanup job finds certain resources already not found, it should skip instead of error.

To Reproduce Steps to reproduce the behavior:

  1. Ensure that we don't have the default ISB service.
  2. Create a new ISB service with a new name "test-isb".
  3. Create a pipeline without specifying the ISB name.
  4. Expect the pipeline creation to fail with the error: Error: secret "isbsvc-default-js-client-auth" not found.
  5. Update the pipeline spec to use the new ISB service created in step 2 and re-apply.
  6. Expect the pipeline being successfully created but the cleanup pod hanging with error Warning Failed 2s (x7 over 53s) kubelet Error: secret "isbsvc-default-js-client-auth" not found

Expected behavior If a cleanup job finds certain resources already not found, it should skip instead of error.

Message from the maintainers:

Impacted by this bug? Give it a 👍. We often sort issues this way to know what to prioritize.

For quick help and support, join our slack channel.

whynowy commented 1 month ago

Questions:

  1. Somehow I don't see step 4, but pipeline complains ISB Service not found.
  2. I don't see step 6 either, pipeline is successfully created.
KeranYang commented 4 weeks ago

@whynowy I just asked Harsh to see if I missed anything, pending a response.

KeranYang commented 4 weeks ago

@whynowy , Harsh was not able to reproduce any more. I am closing this issue.