feat: add anchor metrics

[smrz2001]

This PR adds a few more anchor polling related metrics and implements a simple pacing for anchor polling with some jitter.

@stbrody, I've not added anything besides the jitter for generally preventing status requests from lining up, whether during normal operation or after a restart.

I was thinking about adding a sleep between queries in the first iteration like you suggested but wouldn't that also cause requests to line back up in the next iteration? You might have to elaborate the approach a bit more, and I can implement it.

EDIT: Removed all polling updates in favor of https://github.com/ceramicnetwork/js-ceramic/pull/3218

[smrz2001]

the idea I had with the sleep was to add a very small sleep (like 1ms or less) after polling each request (possibly with some jitter), during the first pass through the store after a restart. Then on subsequent loops through the store you won't need that anymore since everything will have gotten into the cache with different expiration times due to those sleeps.

I don't totally love the idea of having logic that runs on only the first loop and not subsequent loops though, feels kind of messy, but I haven't been able to think of anything else that will help the problem of the first loop hammering the CAS as fast as possible.

Yeah, I see what you're saying 😢

What if we don't send any requests during the first iteration of the loop and just set up the next polling times? Kind of like the "trailing edge" throttling in the lodash.throttle function that Samika was looking at.

What do you think of this version? Requests for updated streams will also go out at the trailing edge of the pacing interval but that's ok? Or we could also add a check to only do this the very first time through the loop.

  checkPollTime(streamId: string): boolean {
    // Add some jitter to prevent all streams from being polled at exactly the same time after a restart
    const jitter = Math.floor(Math.random() * 2 * ANCHOR_POLL_JITTER_MS) - ANCHOR_POLL_JITTER_MS
    const nextPollTime = this.#pollCache.get(streamId)
    if (nextPollTime && (Date.now() < nextPollTime)) {
      return false
    }
    this.#pollCache.set(streamId, ANCHOR_POLL_PACING_MS + jitter)
    return !!nextPollTime;
  }

[smrz2001]

Removed all polling updates in favor of finalizing https://github.com/ceramicnetwork/js-ceramic/pull/3218.