storage: categorized write metrics are inflated

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.

https://www.cockroachlabs.com

Other

29.95k stars 3.79k forks source link

Closed nicktrav closed 2 months ago

nicktrav commented 3 months ago

In a multi-store configuration, the categorized write metrics can report inflated numbers.

For example, the following was pulled from a single node that has four stores:

Note also that the metrics per store are ~identical, which is sus.

Digging into the code, we have a single DiskWriteStatsCollector for all engines, and the collector doesn't internally separate based on store.

Jira issue: CRDB-39906

blathers-crl[bot] commented 3 months ago

Hi @nicktrav, please add branch-* labels to identify which branch(es) this C-bug affects.

_{:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

aadityasondhi commented 2 months ago

aadityasondhi commented 2 months ago

The commit that introduced this bug is not present in v24.1, but is in v24.2.

Commit: 91f53eae420edbb40cddfd2559bcb4406f7a4c93

> git tag --contains 91f53eae420edbb40cddfd2559bcb4406f7a4c93
custombuild-v24.1.0-alpha.5-1783-g6cde73f5565
v24.2.0-alpha.1
v24.2.0-alpha.2