feat(syncer/metrics): add metrics to syncer

[vgonkivs]

Overview

Add metrics to syncer

Checklist

• [ ] New and updated code has appropriate documentation
• [ ] New and updated code has new and/or updated testing
• [ ] Required CI checks are passing
• [ ] Visual proof for any user facing features like CLI or documentation updates
• [ ] Linked issues closed with keywords

[codecov-commenter]

Codecov Report

Attention: 75 lines in your changes are missing coverage. Please review.

Comparison is base (28ff21c) 63.60% compared to head (0070afe) 63.03%. Report is 13 commits behind head on main.

Files	Patch %	Lines
sync/metrics.go	11.90%	69 Missing and 5 partials :warning:
sync/sync_head.go	83.33%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #136 +/- ## ========================================== - Coverage 63.60% 63.03% -0.58% ========================================== Files 39 39 Lines 3366 3476 +110 ========================================== + Hits 2141 2191 +50 - Misses 1060 1113 +53 - Partials 165 172 +7 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[vgonkivs]

Please note, that peer info is not shown on the second screenshot because only one node was running

[Wondertan]

@vgonkivs, this is a feature, not misc

[Wondertan]

Tracking requestHeaders time feels like a job for tracing rather than metrics. Mainly because traces would provide the same data as metrics with additional insights on sub operations/spans. Although, this two can co-exist with metrics for monitoring anomalies and traces specifically for debugging.

[Wondertan]

For completely synced metric. I think we should add it, as syncLoopStop or similar.

[walldiss]

Tracing is easier for debugging, while metrics are handy to have general overview of app. Having request time histogram will help us to quickly spot that there are some issues and then proceed to traces to investigate with more details.

Completed sync it could be a isSyncedGauge, that does simple async if check on internal syncer state. For request time it would be great to have syncRequestTimeHist, which could track requests finished with their execution time.

[Wondertan]

From our quick sync with @walldiss, we agreed to add a few more metrics:

• syncLoopTimeHist - histogram that tracks the time syncing took and when it finished.
  • @walldiss, please advice on bucketing(might need opentelemetry bump)
• syncLoopGauge - gauge to show in runtime whether the syncing is active
• requestRangeTimeHist - histogram to track range requests time(in addition to traces in exchange)

[vgonkivs]

syncLoopGauge -> why can't we have a simple atomic with values 1/0? It will be enough I believe.

[vgonkivs]

please advice on bucketing(might need opentelemetry bump) -> I have tried buckets in the node repo but it hasn't worked for me.