Add additional operational excellence metrics in NumaRollout

[chandankumar4]

Summary

What change needs making?

Add the following metrics at different layers.

PipelineRollout

• [ ] Number of pipeline synced
• [ ] How long the pipeline rollout reconciliation takes.
  • [ ] Phase: Update, Create, Delete
  • [ ] TODO: Phase: Pause metrics
• [ ] Error while syncing pipeline rollout

PipelineRollout queue

• [ ] Queue length

ISBSVC Rollout

• [ ] Number of ISBSVC Rollout synced
• [ ] How long the ISBSVC Rollout reconciliation takes.
  • [ ] Phase: Update, Create, Delete
• [ ] Error while syncing ISBSVC Rollout

NumaflowController Rollout

• [ ] Number of NumaflowController Rollout synced
• [ ] How long the NumaflowController Rollout reconciliation takes.
  • [ ] Phase: Update, Create, Delete
• [ ] Error while syncing NumaflowController Rollout
• Sub task:
  • (K8s API calls (during syncing/cache))
    • [ ] Number of kubernetes requests executed during reconciliation, (Only for Numaflow Controller)
    • [ ] Number of kubectl executions
    • [ ] Number of pending kubectl executions
  • Cache
    • [ ] Number of monitored kubernetes API resources
    • [ ] Number of k8s resource objects in the cache
    • [ ] Error on fetching cluster cache

Overall Indicator

• [ ] Healthiness/Liveness
• [ ] Alert on Pod restarts
• [ ] Alert on memory consumption if reach 70% memory limit
• [ ] Alert on error rate on Pipeline reconciliation
• [ ] Alert on error rate on ISBSVC reconciliation
• [ ] Alert on error rate on Numaflow Controller reconciliation

Use Cases

When would you use this?

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[juliev0]

consider adding queue length (PipelineRollout queue)