Perf observability: Add blob encoding latency at client side

[jianoaix]

Why are these changes needed?

This adds blob encoding latency metric on the Batcher (i.e. client) side.

Currently we only have encoding latency on Encoder (i.e. server) side, which has the following problems:

• It doesn't track the TCP connection creation latency
• It doesn't track the request/response data transmission latency between Batcher and Encoder
• It doesn't track the queuing delay (in case of high traffic)

Having this Batcher-side latency metric addresses all of these problems, and provides a more accurate picture of encoding subsystem performance.

Checks

• [x] I've made sure the lint is passing in this PR.
• [x] I've made sure the tests are passing. Note that there might be a few flaky tests, in that case, please comment that they are not relevant.
• Testing Strategy
  • [ ] Unit tests
  • [ ] Integration tests
  • [ ] This PR is not tested :(

[jianoaix]

Nice, so is the idea that if we subtract server side latency with the client side latency the diff will be the transmission time?

Yep. I think it anyway makes more sense to watch the overall latency from client's perspective, that's what will affect the E2E latency eventually.