Add Allocation Service Metrics

[abbasahmed]

Problem: We currently do not have a lot of metrics around Allocations API service in Thundernetes. Reference: Issue #384

Solution:

This PR adds in 6 metrics:

• AllocationsTimeTakenDuration The time taken for a successful allocation to complete.
• AllocationsRetriesCounter The number of retries taken for an allocation to complete.
• Allocations429ErrorsCounter The number of 429 (too many requests) errors during allocation
• Allocations404ErrorsCounter The number of 404 (not found) errors during allocation
• Allocations500ErrorsCounter The number of 500 (internal) errors during allocation
• Allocation409ErrorsCounter The number of 409 (request conflict) errors during allocation

These new metrics can help us monitor the allocation service in ways such as monitoring the performance of allocation service in terms of speed and reliability. The metrics allow us to quickly monitor the errors of allocation service helping us to make quicker inferences/decisions.

Along with the metrics, we have also added a couple of panels in the Grafana dashboard to visualize these events.

[ghost]

All CLA requirements met.

[dgkanatsios]

Great work! Can you add a small description on the PR? Thanks!

[abbasahmed]

Hi @dgkanatsios, I've added a description of the PR. Also a heads up, we are currently still adding some code changes so currently the PR is in draft status.

[dgkanatsios]

hey @abbasahmed, let me know how I can help to land this PR! Appreciate all the hard work, thanks!

[dgkanatsios]

@abbasahmed @ghov kind ping, we want to release 0.6 next week and it would be great to include your changes!

[abbasahmed]

@dgkanatsios will resolve the conflicts and publish the PR today. Sorry for the hold up on this PR!

[dgkanatsios]

thanks @abbasahmed, let me know if you need any help!