feat: expose metrics from the kubenurse httpclient

[ghouscht]

The following new metrics were added:

• kubenurse_httpclient_requests_total - Total issued requests by kubenurse, partitioned by http code/method.
• kubenurse_httpclient_trace_request_duration_seconds - Latency histogram for requests from the kubenurse httpclient, partitioned by event.
• httpclient_request_duration_seconds - Latency histogram of request latencies from the kubenurse httpclient.

To discuss: Maybe we should make the httpclient_trace_request_duration_seconds metrics an opt-in feature, because it adds quite a lot of new metric series:

• 2 additional series (_sum, _count) * 8 event labels = 16 series
• 10 buckets (including +Inf) * 8 events labels = 80

= 96 time series added!

Another solution would be to split the httpclient_trace_request_duration_seconds histogram into multiple histograms to be able to use more granular buckets. For example the dns_start/done part should be very fast, so there is no need for the .25, .5, 1 buckets.

[coveralls]

Pull Request Test Coverage Report for Build 1676578814

• 74 of 118 (62.71%) changed or added relevant lines in 4 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-1.9%) to 79.048%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
internal/servicecheck/transport.go	8	29	27.59%
internal/servicecheck/httptrace.go	47	70	67.14%
<!--	Total:	74	118	62.71%	-->

Totals
Change from base Build 1662046263:	-1.9%
Covered Lines:	415
Relevant Lines:	525

---

💛 - Coveralls