Watermark is not moving for FCI Pipeline

[shashank10456]

Describe the bug A clear and concise description of what the bug is.

Started seeing the following logs and the watermark is -1 after that for inference pod in FCI

{"level":"error","ts":"2024-03-01T23:42:04.855078746Z","logger":"numaflow.MapUDF-processor","caller":"jetstream/kv_store.go:299","msg":"Failed to get value","pipeline":"numalogic-osam-fci","vertex":"inference","kvName":"oss-analytics-numalogicosamfci-usw2-prd-numalogic-osam-fci-preprocess-inference_OT","watcher":"oss-analytics-numalogicosamfci-usw2-prd-numalogic-osam-fci-preprocess-inference_OT","error":"nats: key not found","stacktrace":"github.com/numaproj/numaflow/pkg/shared/kvs/jetstream.(*jetStreamStore).lastUpdateKVTime\n\t/home/runner/work/numaflow/numaflow/pkg/shared/kvs/jetstream/kv_store.go:299\ngithub.com/numaproj/numaflow/pkg/shared/kvs/jetstream.(*jetStreamStore).Watch.func1\n\t/home/runner/work/numaflow/numaflow/pkg/shared/kvs/jetstream/kv_store.go:216"}

To Reproduce Steps to reproduce the behavior:

1. ....
2. ....
3. ....

Expected behavior A clear and concise description of what you expected to happen.

The watermark should not have been older than 10 min.

Screenshots If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• Kubernetes: [e.g. v1.18.6]
• Numaflow: [e.g. v0.5.1]

Additional context Add any other context about the problem here.

---

For quick help and support, join our slack channel.

Message from the maintainers:

Impacted by this bug? Give it a 👍. We often sort issues this way to know what to prioritize.

ISBSvc Logs: [Uploading ISBS.json…]()

[vigith]

please update to the latest numaflow-go SDK

[vigith]

please reopen if you are seeing the same problem after moving to the latest SDK.