Add internal Prometheus metrics

[Noah-Huppert]

Add an internal Prometheus metrics endpoint.

Collect the following metrics:

• Total requests sum (Put in a middleware handler)
• Request time (Put in middleware handler)
• Total jobs sum
• Job time

[Noah-Huppert]

• Total panics (Put in panic handler middleware)

[Noah-Huppert]

• api_error_responses_total
• jobs_failures_total

[Noah-Huppert]

Settled on:

• serverless_registry_api
  • api
  • [x] requests_total
  • [x] requests_duration_milliseconds
  • [x] handlers_panics_total
  • [x] error_responses_total
  • jobs
  • [x] submitted_total
  • [x] run_duration_milliseconds
  • [x] failures_total

[Noah-Huppert]

Unsure how to record serverless_registry_api_api_error_responses_total.

Todo:

• [x] Look for a place where we can run code after a response has been sent.

[Noah-Huppert]

The serverless_registry_api_api_error_responses_total metric is the only metric with a status_code label for the response's status code.

• [x] Determine what other api metrics could benefit from a status code label
• [x] Re-instrument these metrics with metrics.MetricsResponseWriter

[Noah-Huppert]

• [x] Make unified metrics handler

[Noah-Huppert]

Metrics spec:

Format

• Namespace: <namespace>
  • Subsystem: <subsystem>
    • [ ] <metric>
      • <label>

...

• Namespace: serverless_registry_api
  • Subsystem: api
  • [x] response_durations_milliseconds
    • path
    • method
    • status_code
  • [x] handler_panics_total
    • path
    • method
  • Subsystem: jobs
  • [x] submitted_total
    • job_type
  • [x] run_durations_milliseconds
    • job_type
    • successful

[Noah-Huppert]

• [x] Remove the serverless_registry_api_jobs_submitted_total metric since a count on serverless_registry_api_jobs_run_durations_milliseconds could tell us the same thing.