search

Azure / fleet

Multi-cluster core
MIT License
76 stars 22 forks source link

[BUG] Resync triggers unreal Work metrics #867

Open sschepens opened 2 months ago

sschepens commented 2 months ago

Describe the bug

Resyncs trigger reprocessing of Work objects that haven't changed and this ends up generating unrealistic work_apply_time_seconds because this metric is based on a timestamp set on an annotation whenever the Work object was created/updated, so we end up with metrics that always hit the maximum bucket limits and are not really useful.

ryanzhang-oss commented 2 months ago

@sschepens, thank you for the issue. You are right, that specific metrics will increase even when the work is not changed. I can think of a few ways to improve it. For example, we can only increase this if there is change. I think most of the functionality is there, I wonder if you want to give it try?