cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.11k stars 3.81k forks source link

roachprod-microbench: update weekly script to support export to prometheus #104717

Closed herkolategan closed 1 month ago

herkolategan commented 1 year ago

With the addition of https://github.com/cockroachdb/cockroach/pull/104300, roachprod-microbench will be able to export microbenchmark results to open metrics format. The output is written to stdout, using the export sub-command.

Thus the weekly microbenchmark script [1] will need to be updated to direct the output to a file. This file needs to be placed in a gcs bucket [2] which is monitored by another process. This process will take the uploaded file and load it into bigtable via promtable.

The file is moved to a processed folder within the bucket, so we should ensure the naming is unique. Since these files are small it would be good to keep a history for a while at least.

The command can take labels as well which it will pass along in the metrics (this is useful to categorise the metrics for later use when querying). It also takes a timestamp. The implementer should supply both of these. The timestamp from the git revision and branch the microbenchmarks ran for might be a good consideration (others could be cloud & machine type). The labels could include on what machine type the microbenchmarks ran and other related info (possibly the branch etc.).

Also Note: We might not want this to happen if someone is running the job ad-hoc for their own purposes. There might be logic required to prevent sending the results for ad-hoc (non weekly) runs.

[1] https://github.com/cockroachdb/cockroach/blob/master/build/teamcity/cockroach/nightlies/microbenchmark_weekly.sh [2] gs://cockroach-testeng-metrics/omloader/incoming

Jira issue: CRDB-28690

blathers-crl[bot] commented 1 year ago

cc @cockroachdb/test-eng

herkolategan commented 1 month ago

Switched to using InfluxDB