search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.89k stars 3.78k forks source link

contention: contention events on secondary tenant keys are not tenant prefixed #83518

Open adityamaru opened 2 years ago

adityamaru commented 2 years ago

In a recent support escalation, we received a debug zip taken by the system tenant in a serverless cluster. In the crdb_internal.cluster_contention_events.txt file we observed entries of the form:

table_id        index_id        num_contention_events   cumulative_contention_time      key     txn_id  count
54      2       587     04:20:39.581918 \xbe\x8a\x12foo\x00\x01\x88    c7d4c999-3a21-4901-8947-a6f10229bd3b    583

This translates to:

root@localhost:26257/defaultdb> select crdb_internal.pretty_key(b'\xbe\x8a\x12foo\x00\x01\x88', 0);
  crdb_internal.pretty_key
----------------------------
  /54/2/"foo"/0
(1 row)

When we dug into the logs of the debug zip we did indeed see contention on this key, but in a secondary tenants' keyspace i.e. /Tenant/123/54/2/"foo"/0. This seems to point to a bug in the interaction of contention events on keys in the secondary tenants keyspace when viewing from the system tenant. The cluster contention event keys should probably be tenant prefixed when looking at it from a point of view of the system tenant so as to make debugging easier.

Jira issue: CRDB-17121

Epic CRDB-20388

stevendanna commented 2 months ago

db-server to confirm this is still an issue.