search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.89k stars 3.77k forks source link

kv/kvserver/concurrency: TestLockTableConcurrentRequests failed #122089

Closed cockroach-teamcity closed 4 months ago

cockroach-teamcity commented 5 months ago

kv/kvserver/concurrency.TestLockTableConcurrentRequests failed on master @ 911f1ce389342459592d89fe7b78bcba3a2d265a:

Fatal error:

panic: test timed out after 24m57s
running tests:
    TestLockTableConcurrentRequests (24m57s)
    TestLockTableConcurrentRequests/concurrency_2 (24m57s)

Stack: 

goroutine 261024 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error

``` === RUN TestLockTableConcurrentRequests test_log_scope.go:170: test logs captured to: outputs.zip/logTestLockTableConcurrentRequests4202612793 test_log_scope.go:81: use -show-logs to present logs inline === RUN TestLockTableConcurrentRequests/concurrency_2 ```

Parameters:

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-37695

arulajmani commented 5 months ago

I'm going to see if I can reproduce this with https://github.com/cockroachdb/cockroach/pull/122056.

arulajmani commented 5 months ago

This is reproducing 1/10,000 failures. Interestingly, it's not hitting a validation failure even with lock promotion validation in there. I'll switch debugging tacts here.

arulajmani commented 5 months ago

It's readily reproducible under deadlock. Debugging using https://github.com/cockroachdb/cockroach/pull/122009.

arulajmani commented 5 months ago

It's readily reproducible under deadlock.

This turned out to be a red herring. Turns out, this test is extremely susceptible to overload. If I don't constrain the number of parallel runs (using --cpu), we see failures on release-23.2 as well.

I haven't been able to piece the entire story together with the logging I've added. Most failures take the form where all requests are queued behind a single lock holder. For example:

Lock Table ``` num=10 lock: "a" holder: txn: 00000000-0000-0000-0000-000000000194 epoch: 0, iso: Serializable, ts: 0.000000002,0, info: unrepl [(str: Exclusive seq: 0)] queued locking requests: active: true req: 388, strength: Exclusive, txn: 00000000-0000-0000-0000-0000000000fd lock: "b" queued locking requests: active: false req: 389, strength: Exclusive, txn: 00000000-0000-0000-0000-0000000000fe active: true req: 402, strength: Intent, txn: 00000000-0000-0000-0000-000000000107 active: true req: 403, strength: Intent, txn: 00000000-0000-0000-0000-000000000108 active: true req: 407, strength: Intent, txn: 00000000-0000-0000-0000-00000000010b active: true req: 418, strength: Intent, txn: 00000000-0000-0000-0000-000000000112 active: true req: 436, strength: Intent, txn: 00000000-0000-0000-0000-00000000011b active: true req: 441, strength: Intent, txn: 00000000-0000-0000-0000-000000000115 active: true req: 456, strength: Intent, txn: 00000000-0000-0000-0000-000000000122 active: true req: 469, strength: Intent, txn: 00000000-0000-0000-0000-000000000130 active: true req: 476, strength: Intent, txn: 00000000-0000-0000-0000-000000000125 active: true req: 499, strength: Intent, txn: 00000000-0000-0000-0000-000000000142 active: true req: 506, strength: Intent, txn: 00000000-0000-0000-0000-000000000146 active: true req: 514, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000014b active: true req: 520, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000014e active: true req: 554, strength: Intent, txn: 00000000-0000-0000-0000-000000000167 active: true req: 558, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000169 active: true req: 560, strength: Intent, txn: 00000000-0000-0000-0000-000000000168 active: true req: 561, strength: Intent, txn: 00000000-0000-0000-0000-000000000168 active: true req: 563, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000016b active: true req: 620, strength: Intent, txn: 00000000-0000-0000-0000-00000000018f active: true req: 630, strength: Intent, txn: 00000000-0000-0000-0000-000000000190 lock: "c" queued locking requests: active: false req: 417, strength: Intent, txn: 00000000-0000-0000-0000-000000000111 active: true req: 428, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000119 active: true req: 431, strength: Shared, txn: 00000000-0000-0000-0000-00000000011a active: true req: 449, strength: Intent, txn: 00000000-0000-0000-0000-000000000123 active: true req: 477, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000133 active: true req: 486, strength: Intent, txn: 00000000-0000-0000-0000-00000000012c active: true req: 511, strength: Intent, txn: 00000000-0000-0000-0000-000000000141 active: true req: 513, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000014a active: true req: 556, strength: Intent, txn: 00000000-0000-0000-0000-000000000167 active: true req: 562, strength: Intent, txn: 00000000-0000-0000-0000-000000000167 active: true req: 565, strength: Intent, txn: 00000000-0000-0000-0000-00000000016c active: true req: 570, strength: Intent, txn: 00000000-0000-0000-0000-000000000171 active: true req: 575, strength: Intent, txn: 00000000-0000-0000-0000-000000000174 active: true req: 588, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000017d active: true req: 607, strength: Intent, txn: 00000000-0000-0000-0000-00000000017f active: true req: 619, strength: Intent, txn: 00000000-0000-0000-0000-000000000183 active: true req: 626, strength: Intent, txn: 00000000-0000-0000-0000-000000000184 lock: "d" queued locking requests: active: false req: 389, strength: Exclusive, txn: 00000000-0000-0000-0000-0000000000fe active: true req: 417, strength: Intent, txn: 00000000-0000-0000-0000-000000000111 active: false req: 429, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000106 active: false req: 431, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000011a active: true req: 432, strength: Intent, txn: 00000000-0000-0000-0000-000000000106 active: true req: 443, strength: Shared, txn: 00000000-0000-0000-0000-00000000011f active: true req: 452, strength: Intent, txn: 00000000-0000-0000-0000-000000000125 active: true req: 471, strength: Shared, txn: 00000000-0000-0000-0000-00000000012c active: true req: 478, strength: Intent, txn: 00000000-0000-0000-0000-000000000134 active: true req: 484, strength: Intent, txn: 00000000-0000-0000-0000-000000000139 active: true req: 497, strength: Intent, txn: 00000000-0000-0000-0000-00000000013a active: true req: 517, strength: Intent, txn: 00000000-0000-0000-0000-00000000014c active: true req: 521, strength: Intent, txn: 00000000-0000-0000-0000-000000000149 active: true req: 523, strength: Intent, txn: 00000000-0000-0000-0000-000000000150 active: true req: 537, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000015a active: true req: 541, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000015d active: true req: 568, strength: Intent, txn: 00000000-0000-0000-0000-00000000016f active: true req: 573, strength: Shared, txn: 00000000-0000-0000-0000-000000000168 active: true req: 582, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000178 active: true req: 618, strength: Intent, txn: 00000000-0000-0000-0000-00000000018e active: true req: 633, strength: Intent, txn: 00000000-0000-0000-0000-000000000197 lock: "e" queued locking requests: active: false req: 389, strength: Shared, txn: 00000000-0000-0000-0000-0000000000fe active: false req: 396, strength: Intent, txn: 00000000-0000-0000-0000-000000000103 active: true req: 401, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000106 active: true req: 406, strength: Intent, txn: 00000000-0000-0000-0000-00000000010a active: true req: 425, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000117 active: true req: 429, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000106 active: true req: 435, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000011c active: true req: 444, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000120 active: true req: 474, strength: Intent, txn: 00000000-0000-0000-0000-000000000131 active: true req: 501, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000141 active: true req: 505, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000145 active: true req: 539, strength: Intent, txn: 00000000-0000-0000-0000-00000000015b active: true req: 549, strength: Intent, txn: 00000000-0000-0000-0000-000000000162 active: true req: 567, strength: Intent, txn: 00000000-0000-0000-0000-00000000016e active: true req: 569, strength: Intent, txn: 00000000-0000-0000-0000-000000000170 active: true req: 589, strength: Intent, txn: 00000000-0000-0000-0000-00000000017e active: true req: 616, strength: Intent, txn: 00000000-0000-0000-0000-00000000018d active: true req: 623, strength: Intent, txn: 00000000-0000-0000-0000-000000000189 active: true req: 629, strength: Intent, txn: 00000000-0000-0000-0000-000000000193 lock: "f" queued locking requests: active: false req: 388, strength: Exclusive, txn: 00000000-0000-0000-0000-0000000000fd active: true req: 389, strength: Shared, txn: 00000000-0000-0000-0000-0000000000fe active: true req: 395, strength: Intent, txn: 00000000-0000-0000-0000-000000000101 active: true req: 396, strength: Shared, txn: 00000000-0000-0000-0000-000000000103 active: true req: 399, strength: Intent, txn: 00000000-0000-0000-0000-000000000104 active: true req: 410, strength: Intent, txn: 00000000-0000-0000-0000-00000000010c active: true req: 411, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000106 active: true req: 419, strength: Intent, txn: 00000000-0000-0000-0000-00000000010d active: true req: 420, strength: Intent, txn: 00000000-0000-0000-0000-000000000112 active: true req: 433, strength: Intent, txn: 00000000-0000-0000-0000-000000000119 active: true req: 437, strength: Intent, txn: 00000000-0000-0000-0000-000000000115 active: true req: 439, strength: Intent, txn: 00000000-0000-0000-0000-00000000011d active: true req: 447, strength: Intent, txn: 00000000-0000-0000-0000-000000000122 active: true req: 450, strength: Intent, txn: 00000000-0000-0000-0000-000000000124 active: true req: 455, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000126 active: true req: 458, strength: Intent, txn: 00000000-0000-0000-0000-000000000125 active: true req: 462, strength: Intent, txn: 00000000-0000-0000-0000-00000000012c active: true req: 468, strength: Shared, txn: 00000000-0000-0000-0000-00000000012f active: true req: 480, strength: Intent, txn: 00000000-0000-0000-0000-000000000135 active: true req: 489, strength: Intent, txn: 00000000-0000-0000-0000-000000000139 active: true req: 496, strength: Shared, txn: 00000000-0000-0000-0000-000000000140 active: true req: 507, strength: Intent, txn: 00000000-0000-0000-0000-000000000144 active: true req: 508, strength: Intent, txn: 00000000-0000-0000-0000-000000000147 active: true req: 510, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000149 active: true req: 518, strength: Shared, txn: 00000000-0000-0000-0000-00000000014c active: true req: 522, strength: Intent, txn: 00000000-0000-0000-0000-00000000014f active: true req: 527, strength: Intent, txn: 00000000-0000-0000-0000-000000000152 active: true req: 529, strength: Intent, txn: 00000000-0000-0000-0000-000000000153 active: true req: 530, strength: Shared, txn: 00000000-0000-0000-0000-000000000154 active: true req: 534, strength: Intent, txn: 00000000-0000-0000-0000-000000000157 active: true req: 536, strength: Intent, txn: 00000000-0000-0000-0000-000000000159 active: true req: 543, strength: Intent, txn: 00000000-0000-0000-0000-00000000015f active: true req: 547, strength: Intent, txn: 00000000-0000-0000-0000-000000000160 active: true req: 550, strength: Intent, txn: 00000000-0000-0000-0000-000000000163 active: true req: 551, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000164 active: true req: 552, strength: Intent, txn: 00000000-0000-0000-0000-000000000165 active: true req: 572, strength: Intent, txn: 00000000-0000-0000-0000-000000000168 active: true req: 583, strength: Intent, txn: 00000000-0000-0000-0000-000000000179 active: true req: 597, strength: Shared, txn: 00000000-0000-0000-0000-000000000176 lock: "g" queued locking requests: active: false req: 389, strength: Exclusive, txn: 00000000-0000-0000-0000-0000000000fe active: true req: 393, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000100 active: true req: 421, strength: Intent, txn: 00000000-0000-0000-0000-000000000113 active: true req: 442, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000011e active: true req: 445, strength: Shared, txn: 00000000-0000-0000-0000-000000000121 active: true req: 457, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000127 active: true req: 466, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000012d active: true req: 467, strength: Intent, txn: 00000000-0000-0000-0000-00000000012e active: true req: 473, strength: Shared, txn: 00000000-0000-0000-0000-00000000012c active: true req: 481, strength: Shared, txn: 00000000-0000-0000-0000-000000000136 active: true req: 483, strength: Intent, txn: 00000000-0000-0000-0000-000000000137 active: true req: 503, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000144 active: true req: 512, strength: Intent, txn: 00000000-0000-0000-0000-000000000142 active: true req: 553, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000166 active: true req: 555, strength: Intent, txn: 00000000-0000-0000-0000-000000000168 active: true req: 578, strength: Intent, txn: 00000000-0000-0000-0000-000000000175 active: true req: 584, strength: Intent, txn: 00000000-0000-0000-0000-00000000017a active: true req: 596, strength: Intent, txn: 00000000-0000-0000-0000-000000000183 active: true req: 599, strength: Intent, txn: 00000000-0000-0000-0000-000000000184 active: true req: 603, strength: Intent, txn: 00000000-0000-0000-0000-000000000176 active: true req: 612, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000018a active: true req: 622, strength: Shared, txn: 00000000-0000-0000-0000-000000000190 active: true req: 632, strength: Intent, txn: 00000000-0000-0000-0000-000000000196 active: true req: 634, strength: Intent, txn: 00000000-0000-0000-0000-000000000198 lock: "h" queued locking requests: active: false req: 389, strength: Shared, txn: 00000000-0000-0000-0000-0000000000fe active: false req: 417, strength: Intent, txn: 00000000-0000-0000-0000-000000000111 active: false req: 429, strength: Intent, txn: 00000000-0000-0000-0000-000000000106 active: true req: 438, strength: Intent, txn: 00000000-0000-0000-0000-000000000115 active: true req: 448, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000121 active: true req: 460, strength: Intent, txn: 00000000-0000-0000-0000-000000000129 active: true req: 461, strength: Intent, txn: 00000000-0000-0000-0000-00000000012a active: true req: 463, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000012b active: true req: 487, strength: Intent, txn: 00000000-0000-0000-0000-00000000013a active: true req: 524, strength: Shared, txn: 00000000-0000-0000-0000-00000000014c active: true req: 526, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000152 active: true req: 574, strength: Intent, txn: 00000000-0000-0000-0000-000000000173 active: true req: 585, strength: Intent, txn: 00000000-0000-0000-0000-000000000172 active: true req: 587, strength: Intent, txn: 00000000-0000-0000-0000-00000000017c active: true req: 594, strength: Intent, txn: 00000000-0000-0000-0000-00000000016a active: true req: 604, strength: Intent, txn: 00000000-0000-0000-0000-00000000017f active: true req: 627, strength: Intent, txn: 00000000-0000-0000-0000-00000000018b active: true req: 631, strength: Intent, txn: 00000000-0000-0000-0000-000000000195 lock: "i" queued locking requests: active: false req: 388, strength: Shared, txn: 00000000-0000-0000-0000-0000000000fd active: true req: 392, strength: Intent, txn: 00000000-0000-0000-0000-0000000000fe active: true req: 405, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000109 active: true req: 415, strength: Intent, txn: 00000000-0000-0000-0000-00000000010f active: true req: 423, strength: Intent, txn: 00000000-0000-0000-0000-000000000115 active: true req: 426, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000118 active: true req: 446, strength: Intent, txn: 00000000-0000-0000-0000-000000000122 active: true req: 470, strength: Intent, txn: 00000000-0000-0000-0000-00000000012f active: true req: 472, strength: Intent, txn: 00000000-0000-0000-0000-00000000012f active: true req: 475, strength: Intent, txn: 00000000-0000-0000-0000-000000000132 active: true req: 485, strength: Intent, txn: 00000000-0000-0000-0000-000000000131 active: true req: 488, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000012c active: true req: 490, strength: Intent, txn: 00000000-0000-0000-0000-00000000013b active: true req: 494, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000013e active: true req: 500, strength: Intent, txn: 00000000-0000-0000-0000-000000000120 active: true req: 502, strength: Intent, txn: 00000000-0000-0000-0000-000000000143 active: true req: 509, strength: Intent, txn: 00000000-0000-0000-0000-000000000148 active: true req: 519, strength: Intent, txn: 00000000-0000-0000-0000-00000000014d active: true req: 533, strength: Intent, txn: 00000000-0000-0000-0000-000000000156 active: true req: 535, strength: Intent, txn: 00000000-0000-0000-0000-000000000158 active: true req: 540, strength: Intent, txn: 00000000-0000-0000-0000-00000000015c active: true req: 542, strength: Intent, txn: 00000000-0000-0000-0000-00000000015e active: true req: 544, strength: Intent, txn: 00000000-0000-0000-0000-000000000160 active: true req: 545, strength: Intent, txn: 00000000-0000-0000-0000-000000000161 active: true req: 546, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000161 active: true req: 548, strength: Intent, txn: 00000000-0000-0000-0000-00000000015f active: true req: 559, strength: Intent, txn: 00000000-0000-0000-0000-00000000016a active: true req: 566, strength: Intent, txn: 00000000-0000-0000-0000-00000000016d active: true req: 586, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000017b active: true req: 592, strength: Intent, txn: 00000000-0000-0000-0000-000000000180 active: true req: 593, strength: Shared, txn: 00000000-0000-0000-0000-000000000182 active: true req: 609, strength: Intent, txn: 00000000-0000-0000-0000-000000000187 active: true req: 611, strength: Intent, txn: 00000000-0000-0000-0000-000000000189 active: true req: 614, strength: Intent, txn: 00000000-0000-0000-0000-00000000018a active: true req: 617, strength: Intent, txn: 00000000-0000-0000-0000-00000000018d active: true req: 624, strength: Intent, txn: 00000000-0000-0000-0000-000000000191 active: true req: 625, strength: Intent, txn: 00000000-0000-0000-0000-000000000192 lock: "j" queued locking requests: active: false req: 389, strength: Intent, txn: 00000000-0000-0000-0000-0000000000fe active: true req: 391, strength: Intent, txn: 00000000-0000-0000-0000-0000000000ff active: true req: 413, strength: Intent, txn: 00000000-0000-0000-0000-00000000010d active: true req: 414, strength: Intent, txn: 00000000-0000-0000-0000-00000000010e active: true req: 416, strength: Intent, txn: 00000000-0000-0000-0000-000000000110 active: true req: 424, strength: Intent, txn: 00000000-0000-0000-0000-000000000116 active: true req: 493, strength: Shared, txn: 00000000-0000-0000-0000-00000000013c active: true req: 495, strength: Intent, txn: 00000000-0000-0000-0000-00000000013f active: true req: 498, strength: Shared, txn: 00000000-0000-0000-0000-000000000141 active: true req: 531, strength: Intent, txn: 00000000-0000-0000-0000-000000000155 active: true req: 557, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000160 active: true req: 571, strength: Intent, txn: 00000000-0000-0000-0000-000000000172 active: true req: 577, strength: Intent, txn: 00000000-0000-0000-0000-000000000174 active: true req: 581, strength: Intent, txn: 00000000-0000-0000-0000-000000000177 active: true req: 590, strength: Shared, txn: 00000000-0000-0000-0000-00000000017f active: true req: 591, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000181 active: true req: 610, strength: Intent, txn: 00000000-0000-0000-0000-000000000188 active: true req: 615, strength: Intent, txn: 00000000-0000-0000-0000-00000000018b ```

We see that all requests are blocked behind req 388, which itself is waiting on the lock held by txn 000000000194. In all my reproductions, the transaction in question that holds the lock has been finalized previously. Moreover, its locks have also been resolved, and resolved correctly, as judging by the lock table's state. How is this lock appearing in the lock table again?

One possibility would be if the test is slow and there's a "lagging" lock acquisition that lands after the locks have been resolved. But that doesn't seem plausible given we won't kick off intent resolution if there are in-flight requests still pending. See:

https://github.com/cockroachdb/cockroach/blob/cead8be66c30a7377dacbf1c678911d2cb17194d/pkg/kv/kvserver/concurrency/lock_table_test.go#L1363-L1368

The test also uses monotonic transaction IDs, so it's not possible for us to be creating a new transaction with the same ID.

Right now, I'm simply failing the test if I find a request is waiting for > 4 minutes. One thread to pull on would be to check introspect the waitingState on the request. I also want to add some sanity checks around known-to-be-finalized transactions.

All this being said, it seems like we're ending up in a state where the lock is held by a transaction that has been abandoned. If we were to end up in such a situation in reality, we'd detect and handle it correctly when the lock holder is pushed. Given this, and how rare[*] this is (outside of an overloaded test), I'm inclined to downgrade this to a GA-blocker. I still want to understand this a bit better before we're fully confident here. (cc @nvanbenschoten)

[*] For context, I've given this over 150K runs on my GCE worker today, without running under --deadlock, and I've only seen it fail once.

cockroach-teamcity commented 5 months ago

kv/kvserver/concurrency.TestLockTableConcurrentRequests failed on master @ f117eea22dd7be380c7141cdf6cd7aba92dd9c70:

Fatal error:

panic: test timed out after 24m57s
running tests:
    TestLockTableConcurrentRequests (24m57s)
    TestLockTableConcurrentRequests/concurrency_8 (24m54s)

Stack: 

goroutine 833581 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error

``` === RUN TestLockTableConcurrentRequests test_log_scope.go:170: test logs captured to: outputs.zip/logTestLockTableConcurrentRequests2351147515 test_log_scope.go:81: use -show-logs to present logs inline === RUN TestLockTableConcurrentRequests/concurrency_2 items: 1692, aborted: 0, concurrency violations: 758, lock table: num=0 === RUN TestLockTableConcurrentRequests/concurrency_8 ```

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 months ago

kv/kvserver/concurrency.TestLockTableConcurrentRequests failed on master @ 67547d7724f8a52646e2e8ecb3ca48b923957d90:

Fatal error:

panic: test timed out after 24m57s
running tests:
    TestLockTableConcurrentRequests (24m57s)
    TestLockTableConcurrentRequests/concurrency_8 (24m55s)

Stack: 

goroutine 1033150 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error

``` === RUN TestLockTableConcurrentRequests test_log_scope.go:170: test logs captured to: outputs.zip/logTestLockTableConcurrentRequests3274046890 test_log_scope.go:81: use -show-logs to present logs inline === RUN TestLockTableConcurrentRequests/concurrency_2 items: 1671, aborted: 0, concurrency violations: 661, lock table: num=0 === RUN TestLockTableConcurrentRequests/concurrency_8 ```

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 months ago

kv/kvserver/concurrency.TestLockTableConcurrentRequests failed on release-24.1 @ 6e3e9e6012dc04dabc64333a87e77a80d9f9d46b:

Fatal error:

panic: test timed out after 24m57s
running tests:
    TestLockTableConcurrentRequests (24m57s)
    TestLockTableConcurrentRequests/concurrency_16 (24m39s)

Stack: 

goroutine 2588996 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error

``` === RUN TestLockTableConcurrentRequests test_log_scope.go:170: test logs captured to: outputs.zip/logTestLockTableConcurrentRequests1983242438 test_log_scope.go:81: use -show-logs to present logs inline === RUN TestLockTableConcurrentRequests/concurrency_16 ```

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 months ago

kv/kvserver/concurrency.TestLockTableConcurrentRequests failed on master @ 8daa6933836c8ce1108fb22bf6663117fc71599b:

Fatal error:

panic: test timed out after 24m57s
running tests:
    TestLockTableConcurrentRequests (24m57s)
    TestLockTableConcurrentRequests/concurrency_16 (24m46s)

Stack: 

goroutine 2996049 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error

``` === RUN TestLockTableConcurrentRequests test_log_scope.go:170: test logs captured to: outputs.zip/logTestLockTableConcurrentRequests4126696952 test_log_scope.go:81: use -show-logs to present logs inline === RUN TestLockTableConcurrentRequests/concurrency_2 items: 1677, aborted: 0, concurrency violations: 861, lock table: num=0 === RUN TestLockTableConcurrentRequests/concurrency_8 items: 1677, aborted: 0, concurrency violations: 857, lock table: num=0 === RUN TestLockTableConcurrentRequests/concurrency_16 ```

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 months ago

kv/kvserver/concurrency.TestLockTableConcurrentRequests failed on master @ 737df7fe75c5698d9ba384ad322f30963582cc42:

Fatal error:

panic: test timed out after 24m57s
running tests:
    TestLockTableConcurrentRequests (24m57s)
    TestLockTableConcurrentRequests/concurrency_2 (24m57s)

Stack: 

goroutine 196958 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error

``` === RUN TestLockTableConcurrentRequests test_log_scope.go:170: test logs captured to: outputs.zip/logTestLockTableConcurrentRequests3562098866 test_log_scope.go:81: use -show-logs to present logs inline === RUN TestLockTableConcurrentRequests/concurrency_2 ```

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 months ago

kv/kvserver/concurrency.TestLockTableConcurrentRequests failed on master @ 6e94cb3b7992de4dec7688750cfd13971dd2930c:

Fatal error:

panic: test timed out after 24m57s
running tests:
    TestLockTableConcurrentRequests (24m57s)
    TestLockTableConcurrentRequests/concurrency_4 (9m32s)

Stack: 

goroutine 265982376 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error

``` === RUN TestLockTableConcurrentRequests test_log_scope.go:170: test logs captured to: outputs.zip/logTestLockTableConcurrentRequests127189903 test_log_scope.go:81: use -show-logs to present logs inline lock_table_test.go:1609: numRequests: 10000; numActiveTxns: 4; probability(txn-al reqs): 0.75; probability(new txns): 0.25; probability(duplicate access): 0.50 lock_table_test.go:1734: txns creted: 1874 === RUN TestLockTableConcurrentRequests/concurrency_1 items: 11874, aborted: 0, concurrency violations: 11131, lock table: num=0 === RUN TestLockTableConcurrentRequests/concurrency_4 ```

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 months ago

kv/kvserver/concurrency.TestLockTableConcurrentRequests failed on release-24.1 @ 3111a51bdb2cbf889c92f4440bf88b0eb682cb96:

Fatal error:

panic: test timed out after 24m57s
running tests:
    TestLockTableConcurrentRequests (24m57s)
    TestLockTableConcurrentRequests/concurrency_16 (24m44s)

Stack: 

goroutine 2930059 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error

``` === RUN TestLockTableConcurrentRequests test_log_scope.go:170: test logs captured to: outputs.zip/logTestLockTableConcurrentRequests2180491673 test_log_scope.go:81: use -show-logs to present logs inline === RUN TestLockTableConcurrentRequests/concurrency_16 ```

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 months ago

kv/kvserver/concurrency.TestLockTableConcurrentRequests failed on master @ e0068814dfcb4f975a53b79b5546b5fb85c0f927:

Fatal error:

panic: test timed out after 24m57s
running tests:
    TestLockTableConcurrentRequests (24m57s)
    TestLockTableConcurrentRequests/concurrency_32 (21s)

Stack: 

goroutine 315646842 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error

``` === RUN TestLockTableConcurrentRequests test_log_scope.go:170: test logs captured to: outputs.zip/logTestLockTableConcurrentRequests103809738 test_log_scope.go:81: use -show-logs to present logs inline lock_table_test.go:1609: numRequests: 10000; numActiveTxns: 16; probability(txn-al reqs): 0.75; probability(new txns): 0.25; probability(duplicate access): 0.50 lock_table_test.go:1734: txns creted: 1837 === RUN TestLockTableConcurrentRequests/concurrency_16 items: 11837, aborted: 0, concurrency violations: 11485, lock table: num=0 === RUN TestLockTableConcurrentRequests/concurrency_32 ```

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

kv/kvserver/concurrency.TestLockTableConcurrentRequests failed on master @ 347cdc76d4c5abb2e872f325e944337a46b5883f:

Fatal error:

panic: test timed out after 24m57s
running tests:
    TestLockTableConcurrentRequests (24m57s)
    TestLockTableConcurrentRequests/concurrency_8 (12m22s)

Stack: 

goroutine 350542742 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error

``` === RUN TestLockTableConcurrentRequests test_log_scope.go:170: test logs captured to: outputs.zip/logTestLockTableConcurrentRequests4178933035 test_log_scope.go:81: use -show-logs to present logs inline lock_table_test.go:1609: numRequests: 10000; numActiveTxns: 8; probability(txn-al reqs): 0.90; probability(new txns): 0.10; probability(duplicate access): 0.50 lock_table_test.go:1734: txns creted: 917 === RUN TestLockTableConcurrentRequests/concurrency_2 items: 10917, aborted: 0, concurrency violations: 10537, lock table: num=0 === RUN TestLockTableConcurrentRequests/concurrency_8 ```

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

kv/kvserver/concurrency.TestLockTableConcurrentRequests failed on master @ e215cf435851cb9490fe4494fe73d02041c4d2c0:

Fatal error:

panic: test timed out after 24m57s
running tests:
    TestLockTableConcurrentRequests (24m57s)
    TestLockTableConcurrentRequests/concurrency_4 (12m9s)

Stack: 

goroutine 368018592 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error

``` === RUN TestLockTableConcurrentRequests test_log_scope.go:170: test logs captured to: outputs.zip/logTestLockTableConcurrentRequests613030006 test_log_scope.go:81: use -show-logs to present logs inline lock_table_test.go:1609: numRequests: 10000; numActiveTxns: 4; probability(txn-al reqs): 0.75; probability(new txns): 0.10; probability(duplicate access): 0.50 lock_table_test.go:1734: txns creted: 753 === RUN TestLockTableConcurrentRequests/concurrency_1 items: 10753, aborted: 0, concurrency violations: 10204, lock table: num=0 === RUN TestLockTableConcurrentRequests/concurrency_4 ```

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

kv/kvserver/concurrency.TestLockTableConcurrentRequests failed on master @ 0fbef53e48c2a3784631a09a42236e1d0636a5a7:

Fatal error:

panic: test timed out after 24m57s
running tests:
    TestLockTableConcurrentRequests (24m57s)
    TestLockTableConcurrentRequests/concurrency_4 (11m8s)

Stack: 

goroutine 358365045 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error

``` === RUN TestLockTableConcurrentRequests test_log_scope.go:170: test logs captured to: outputs.zip/logTestLockTableConcurrentRequests2386894273 test_log_scope.go:81: use -show-logs to present logs inline lock_table_test.go:1609: numRequests: 10000; numActiveTxns: 2; probability(txn-al reqs): 0.75; probability(new txns): 0.25; probability(duplicate access): 0.50 lock_table_test.go:1734: txns creted: 1888 === RUN TestLockTableConcurrentRequests/concurrency_2 items: 11888, aborted: 0, concurrency violations: 11115, lock table: num=0 === RUN TestLockTableConcurrentRequests/concurrency_4 ```

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

kv/kvserver/concurrency.TestLockTableConcurrentRequests failed on master @ fe6496ccabf2d4b2234cf8418004e5863847c0bd:

Fatal error:

panic: test timed out after 24m57s
running tests:
    TestLockTableConcurrentRequests (24m57s)
    TestLockTableConcurrentRequests/concurrency_4 (10m55s)

Stack: 

goroutine 329327927 [running]:
testing.(*M).startAlarm.func1()
    GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
    GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error

``` === RUN TestLockTableConcurrentRequests test_log_scope.go:170: test logs captured to: outputs.zip/logTestLockTableConcurrentRequests829779615 test_log_scope.go:81: use -show-logs to present logs inline lock_table_test.go:1609: numRequests: 10000; numActiveTxns: 2; probability(txn-al reqs): 0.75; probability(new txns): 0.25; probability(duplicate access): 0.50 lock_table_test.go:1734: txns creted: 1845 === RUN TestLockTableConcurrentRequests/concurrency_2 items: 11845, aborted: 0, concurrency violations: 10743, lock table: num=0 === RUN TestLockTableConcurrentRequests/concurrency_4 ```

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!