nats: no keys found error in numa containers

[joelcomp1]

Describe the bug

We have a pipeline that is constantly throwing this error:

2023-08-31T19:57:45-04:00 {"level":"error","ts":"2023-08-31T23:57:45.094318472Z","logger":"numaflow.MapUDF-processor","caller":"jetstream/kv_watch.go:199","msg":"Failed to get keys","vertex":"test-tx-flow-wfgen-process","pipeline":"test-tx-flow","kvBucketName":"test-test-ingested-wfgen-process_OT","watcher":"test-test-flow-ingested-wfgen-process_OT","error":"nats: no keys found","stacktrace":"github.com/numaproj/numaflow/pkg/watermark/store/jetstream.(jetStreamWatch).lastUpdateKVTime\n\t/home/runner/work/numaflow/numaflow/pkg/watermark/store/jetstream/kv_watch.go:199\ngithub.com/numaproj/numaflow/pkg/watermark/store/jetstream.(jetStreamWatch).Watch.func1\n\t/home/runner/work/numaflow/numaflow/pkg/watermark/store/jetstream/kv_watch.go:161"}

for all the numa containers in each pod.

I tired an example pipeline and it worked, I then started to wonder if some how jetstream didn't get configured correctly for the pipeline above, so I took the exact same pipeline but re-named it and it went away. This time I also noticed the job run to setup the stream. Is it possible my initial job failed for my pipeline to setup jetstream and now its in a bad state?

To Reproduce Steps to reproduce the behavior:

Not 100% sure at this point, I suspect maybe the job initially failed?

Expected behavior Should not constantly see the error in the logs

Screenshots See Above Log Environment (please complete the following information):

• Kubernetes: 1.27
• Numaflow: 0.9.0

Additional context Running on ARM64

Message from the maintainers:

Impacted by this bug? Give it a 👍. We often sort issues this way to know what to prioritize.

[vigith]

could you please try numaflow 0.9.2 release?

[joelcomp1]

That seems to have fixed it, should have tried that to start. Thanks!

[syayi]

Hi @joelcomp1 ! It's great to hear that you're utilizing Numaflow. My name is Sri and I'm a member of the Numaproj team. We're genuinely interested in hearing about your experience with Numaflow. Would you be comfortable having a conversation about it?

[joelcomp1]

@syayi yes that would be great we would love that.

[syayi]

@joelcomp1 Awesome! Let me know what would be the best way to connect?

You can join the community here and we can co-ordinate there if that works too.