Use metrics to track delete request performance

[mniewrzal]

As a team OKR we define that we need to keep eye on metainfo endpoint stability and performance. This is initial task to use metrics to verify metainfo.BeginDeleteObject performance over time. So general idea is to define redash query that will give us information about general method time execution.

• we need to use function_times metric
• take average value from a week
• compare weeks (or months) between to see if we are not seeing decreased performance

Described steps are just draft idea and its possible that there is a better way to track metainfo method performance.

This is mainly a recon ticket for https://github.com/storj/team-metainfo/issues/98 but it will be also valuable to monitor metainfo.BeginDeleteObject performance after deploying server-side copy where we made huge changes how we are deleting objects and segments from DB.

Acceptance criteria:

• we should be notified by email (optionally slack channel #team-metainfo)
• daily report about difference in metainfo.BeginDeleteObject average execution time
• report should also contain min and max value for execution time

[Erikvv]

Should we add this to the copy milestone?

[mniewrzal]

Should we add this to the copy milestone?

I was thinking about it but decided to keep it out of copy milestone as its helpful but not mandatory from a feature perspective.

[mniewrzal]

Good starting point to see objects related method times https://chronograf.datasci.storj.io/sources/3/dashboards/95?tempVars%5Bsatellite%5D=mars&lower=now%28%29%20-%202d

[Fadila82]

• redash dashboard: https://redash.datasci.storj.io/dashboard/begindeleteobject_1?p_interval=1w&p_since=4w&p_status=success&p_w1272_satellite=us-central1
• Parametrized query for execution time of BeginDeleteObject: https://redash.datasci.storj.io/queries/2806/source?p_interval=1w&p_satellite=qa-us-central1&p_since=4w&p_status=success#4489
• One query per satellite that gives average execution time for the last 4 weeks (updated every day):
  • https://redash.datasci.storj.io/queries/2810 (europe-north-1)
  • https://redash.datasci.storj.io/queries/2811 (asia-east1)
  • https://redash.datasci.storj.io/queries/2812 (europe-west1)
  • https://redash.datasci.storj.io/queries/2813 (us2)
  • https://redash.datasci.storj.io/queries/2814 (QA)
  • https://redash.datasci.storj.io/queries/2815 (saltlake)
  • https://redash.datasci.storj.io/queries/2818 (us-central1)
• Query to get last week average time on previous 3 weeks per satellite (every day): https://redash.datasci.storj.io/queries/2816
• Alert: https://redash.datasci.storj.io/alerts/113 that is triggered if the ratio last week/3 previous weeks is equal to or greater than 1.10

[mniewrzal]

I believe we can mark it as done. Good job!