roachtest: kv50/rangelookups/relocate/nodes=8 failed

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.

https://www.cockroachlabs.com

Other

30.1k stars 3.81k forks source link

roachtest: kv50/rangelookups/relocate/nodes=8 failed #82460

Closed cockroach-teamcity closed 1 year ago

cockroach-teamcity commented 2 years ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ 380f15c04826b432273cbd7ffe96b335ffc90396:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    ts_util.go:170,kv.go:787,monitor.go:105,errgroup.go:57: Found minute interval with 52.702778 lookup/sec above target of 50.000000 lookup/sec

    monitor.go:127,kv.go:832,kv.go:860,test_runner.go:883: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:832
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:860
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:883
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #81814 roachtest: kv50/rangelookups/relocate/nodes=8 failed [C-test-failure O-roachtest O-robot T-kv branch-release-22.1.0] - #79887 roachtest: kv50/rangelookups/relocate/nodes=8 failed [waiting for backport] [C-test-failure O-roachtest O-robot S-1 T-kv branch-master]

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-16405

Epic CRDB-18656

cockroach-teamcity commented 2 years ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ af4263069438f09c68f3eaa661ba80323fc22e74:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    ts_util.go:170,kv.go:787,monitor.go:105,errgroup.go:57: Found minute interval with 63.466667 lookup/sec above target of 50.000000 lookup/sec

    monitor.go:127,kv.go:832,kv.go:860,test_runner.go:883: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:832
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:860
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:883
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ e73379784b9f8921d116dc1ad0d2fd8a6256ff9f:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    ts_util.go:170,kv.go:787,monitor.go:105,errgroup.go:57: Found minute interval with 53.888889 lookup/sec above target of 50.000000 lookup/sec

    monitor.go:127,kv.go:832,kv.go:860,test_runner.go:883: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:832
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:860
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:883
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ 3181b7faa2f3b41d6a15ab4b74d2c60bcfe5132d:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    ts_util.go:170,kv.go:787,monitor.go:105,errgroup.go:57: Found minute interval with 56.091667 lookup/sec above target of 50.000000 lookup/sec

    monitor.go:127,kv.go:832,kv.go:860,test_runner.go:883: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:832
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:860
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:883
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ fc456a26830067b6dfbb8cd87e093a28d9b833d1:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    ts_util.go:170,kv.go:787,monitor.go:105,errgroup.go:57: Found minute interval with 52.505556 lookup/sec above target of 50.000000 lookup/sec

    monitor.go:127,kv.go:832,kv.go:860,test_runner.go:883: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:832
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:860
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:883
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

This test on roachdash | Improve this report!

aliher1911 commented 2 years ago

This is a known test flakiness which has a yet unmerged fix on master.

aayushshah15 commented 2 years ago

This actually looks like a new failure mode to what's on master.

cockroach-teamcity commented 2 years ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ 5e58d0ff8a596cd3ea9482a9b5a160fca82168a8:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    ts_util.go:170,kv.go:787,monitor.go:105,errgroup.go:57: Found minute interval with 55.972222 lookup/sec above target of 50.000000 lookup/sec

    monitor.go:127,kv.go:832,kv.go:860,test_runner.go:883: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:832
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:860
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:883
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ 31c6f8ccfdc8eed04ddeb3e2d8e59646d17365ee:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    ts_util.go:170,kv.go:787,monitor.go:105,errgroup.go:57: Found minute interval with 57.269444 lookup/sec above target of 50.000000 lookup/sec

    monitor.go:127,kv.go:832,kv.go:860,test_runner.go:883: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:832
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:860
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:883
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ 3de0153d6b39a9637bd4720bd1bc8f648c8475b4:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    monitor.go:127,kv.go:832,kv.go:860,test_runner.go:883: monitor failure: monitor task failed: pq: while carrying out changes [{ADD_VOTER n3,s3} {REMOVE_VOTER n6,s6}]: unable to find store 3 in range r506:/Table/106/1/68{64547619836971008-82975935594921984} [(n6,s6):8, (n1,s1):6, (n5,s5):3, next=9, gen=80, sticky=9223372036.854775807,2147483647]
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:832
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:860
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) pq: while carrying out changes [{ADD_VOTER n3,s3} {REMOVE_VOTER n6,s6}]: unable to find store 3 in range r506:/Table/106/1/68{64547619836971008-82975935594921984} [(n6,s6):8, (n1,s1):6, (n5,s5):3, next=9, gen=80, sticky=9223372036.854775807,2147483647]
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79887 roachtest: kv50/rangelookups/relocate/nodes=8 failed [waiting for backport] [C-test-failure O-roachtest O-robot S-1 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ 7d74b25b21bdabc3b0cc87ced447acc9967eb4d5:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    monitor.go:127,kv.go:832,kv.go:860,test_runner.go:883: monitor failure: monitor task failed: pq: while carrying out changes [{ADD_VOTER n1,s1} {REMOVE_VOTER n3,s3}]: unable to find store 1 in range r906:/Table/106/1/18{52045733674135552-70474049432086528} [(n6,s6):10, (n8,s8):11, (n3,s3):13, next=14, gen=78, sticky=9223372036.854775807,2147483647]
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:832
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:860
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) pq: while carrying out changes [{ADD_VOTER n1,s1} {REMOVE_VOTER n3,s3}]: unable to find store 1 in range r906:/Table/106/1/18{52045733674135552-70474049432086528} [(n6,s6):10, (n8,s8):11, (n3,s3):13, next=14, gen=78, sticky=9223372036.854775807,2147483647]
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79887 roachtest: kv50/rangelookups/relocate/nodes=8 failed [waiting for backport] [C-test-failure O-roachtest O-robot S-1 T-kv branch-master release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ 6f5197c818619f616a5c8e3ddfedc923f1f6bd26:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    monitor.go:127,kv.go:832,kv.go:860,test_runner.go:883: monitor failure: monitor task failed: pq: while carrying out changes [{ADD_VOTER n3,s3} {REMOVE_VOTER n5,s5}]: unable to find store 3 in range r338:/Table/106/1/68{09262672563116032-27690988321067008} [(n4,s4):9, (n1,s1):11, (n7,s7):8, next=12, gen=68, sticky=9223372036.854775807,2147483647]
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:832
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:860
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) pq: while carrying out changes [{ADD_VOTER n3,s3} {REMOVE_VOTER n5,s5}]: unable to find store 3 in range r338:/Table/106/1/68{09262672563116032-27690988321067008} [(n4,s4):9, (n1,s1):11, (n7,s7):8, next=12, gen=68, sticky=9223372036.854775807,2147483647]
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79887 roachtest: kv50/rangelookups/relocate/nodes=8 failed [waiting for backport] [C-test-failure O-roachtest O-robot S-1 T-kv branch-master release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ e8a6797dd6b1482b740c7c9ec681a5e84dd8a8d8:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    ts_util.go:170,kv.go:790,monitor.go:105,errgroup.go:57: Found minute interval with 52.191667 lookup/sec above target of 50.000000 lookup/sec

    monitor.go:127,kv.go:835,kv.go:863,test_runner.go:883: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:835
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:863
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:883
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79887 roachtest: kv50/rangelookups/relocate/nodes=8 failed [waiting for backport] [C-test-failure O-roachtest O-robot S-1 T-kv branch-master release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ 4da42a4fad47ca6646a77f81bd4daddadbeb2b27:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    ts_util.go:170,kv.go:790,monitor.go:105,errgroup.go:57: Found minute interval with 55.036111 lookup/sec above target of 50.000000 lookup/sec

    monitor.go:127,kv.go:835,kv.go:863,test_runner.go:883: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:835
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:863
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:883
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79887 roachtest: kv50/rangelookups/relocate/nodes=8 failed [waiting for backport] [C-test-failure O-roachtest O-robot S-1 T-kv branch-master release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ 1985775e222793428fbadf4dd3e2e4aba89ad6e7:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    ts_util.go:170,kv.go:790,monitor.go:105,errgroup.go:57: Found minute interval with 59.802778 lookup/sec above target of 50.000000 lookup/sec

    monitor.go:127,kv.go:835,kv.go:863,test_runner.go:883: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:835
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:863
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:883
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79887 roachtest: kv50/rangelookups/relocate/nodes=8 failed [waiting for backport] [C-test-failure O-roachtest O-robot S-1 T-kv branch-master no-test-failure-activity release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ fd04c2f3d7675d5a4a5a6c7d0c53cc502b019529:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    ts_util.go:170,kv.go:790,monitor.go:105,errgroup.go:57: Found minute interval with 51.061111 lookup/sec above target of 50.000000 lookup/sec

    monitor.go:127,kv.go:835,kv.go:863,test_runner.go:883: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:835
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:863
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:883
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ 443dd822332bb6487c949efbd9a4b700693850d4:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    ts_util.go:170,kv.go:790,monitor.go:105,errgroup.go:57: Found minute interval with 58.094444 lookup/sec above target of 50.000000 lookup/sec

    monitor.go:127,kv.go:835,kv.go:863,test_runner.go:883: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:835
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:863
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:883
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ 443dd822332bb6487c949efbd9a4b700693850d4:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    ts_util.go:170,kv.go:790,monitor.go:105,errgroup.go:57: Found minute interval with 58.686111 lookup/sec above target of 50.000000 lookup/sec

    monitor.go:127,kv.go:835,kv.go:863,test_runner.go:883: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:835
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:863
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:883
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ ad4b21c4707741e4b53fdc77a1fe01b1a8ac6fa5:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    ts_util.go:170,kv.go:790,monitor.go:105,errgroup.go:57: Found minute interval with 54.894444 lookup/sec above target of 50.000000 lookup/sec

    monitor.go:127,kv.go:835,kv.go:863,test_runner.go:883: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:835
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:863
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:883
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ 0d34ebfbe8bdc756532719b8d1fe79ced07f978b:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    ts_util.go:170,kv.go:790,monitor.go:105,errgroup.go:57: Found minute interval with 51.397222 lookup/sec above target of 50.000000 lookup/sec

    monitor.go:127,kv.go:835,kv.go:863,test_runner.go:883: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:835
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:863
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:883
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ 0ae8b84678937188c66180005a3099e552a4b896:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    ts_util.go:170,kv.go:790,monitor.go:105,errgroup.go:57: Found minute interval with 58.005556 lookup/sec above target of 50.000000 lookup/sec

    monitor.go:127,kv.go:835,kv.go:863,test_runner.go:883: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:835
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:863
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:883
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ 8c89608e52d71f4cb369b1fea2f407186637c19f:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    ts_util.go:170,kv.go:790,monitor.go:105,errgroup.go:57: Found minute interval with 59.316667 lookup/sec above target of 50.000000 lookup/sec

    monitor.go:127,kv.go:835,kv.go:863,test_runner.go:883: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:835
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:863
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:883
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ cfb3d385aa86893b262adbf797b27f011a6c6944:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    ts_util.go:170,kv.go:790,monitor.go:105,errgroup.go:57: Found minute interval with 50.800000 lookup/sec above target of 50.000000 lookup/sec

    monitor.go:127,kv.go:835,kv.go:863,test_runner.go:883: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:835
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:863
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:883
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ 42dba045cb6e5c9242a28045028738d005961a84:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    ts_util.go:170,kv.go:790,monitor.go:105,errgroup.go:57: Found minute interval with 52.002778 lookup/sec above target of 50.000000 lookup/sec

    monitor.go:127,kv.go:835,kv.go:863,test_runner.go:883: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:835
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:863
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:883
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ 98f4c58f1eda61ebdd4cf6b8f7fa2e5fdb6cede2:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    ts_util.go:170,kv.go:790,monitor.go:105,errgroup.go:57: Found minute interval with 59.433333 lookup/sec above target of 50.000000 lookup/sec

    monitor.go:127,kv.go:835,kv.go:863,test_runner.go:883: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:835
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:863
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:883
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ 6fdb8f55c6e3224d1d4b8bb2b5f7e757d57d29df:

          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:835
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:863
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:883
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1968
          | [...repeated from below...]
        Wraps: (6) output in run_105831.237419303_n9_workload_run_kv
        Wraps: (7) ./workload run kv --tolerate-errors --concurrency=512 --duration=10m --read-percent=50 {pgurl:1-8} returned
          | stderr:
          |
          | stdout:
        Wraps: (8) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2202
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2083
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Run
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:661
          | github.com/cockroachdb/cockroach/pkg/roachprod.Run
          |     github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:384
          | main.execCmdEx
          |     main/pkg/cmd/roachtest/cluster.go:341
          | main.execCmd
          |     main/pkg/cmd/roachtest/cluster.go:229
          | main.(*clusterImpl).RunE
          |     main/pkg/cmd/roachtest/cluster.go:1954
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1.2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:786
          | main.(*monitorImpl).Go.func1
          |     main/pkg/cmd/roachtest/monitor.go:105
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (9) one or more parallel execution failure
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *cluster.WithCommandDetails (8) *withstack.withStack (9) *errutil.leafError

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ d3b519c2b635a6e82be32c44f96df085ee9c7044:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    ts_util.go:170,kv.go:790,monitor.go:105,errgroup.go:57: Found minute interval with 53.258333 lookup/sec above target of 50.000000 lookup/sec

    monitor.go:127,kv.go:835,kv.go:863,test_runner.go:883: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:835
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:863
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:883
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ 5a3f4c12ed87d8161d5065d7a6991c2a7065cdd0:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    ts_util.go:170,kv.go:790,monitor.go:105,errgroup.go:57: Found minute interval with 60.380556 lookup/sec above target of 50.000000 lookup/sec

    monitor.go:127,kv.go:835,kv.go:863,test_runner.go:883: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:835
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:863
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:883
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.kv50/rangelookups/relocate/nodes=8 failed with artifacts on release-22.1 @ 9b2d7611f98d6f63e52a996f35b855e2ce1aa4e5:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    ts_util.go:170,kv.go:790,monitor.go:105,errgroup.go:57: Found minute interval with 51.119444 lookup/sec above target of 50.000000 lookup/sec

    monitor.go:127,kv.go:835,kv.go:863,test_runner.go:883: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:835
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVRangeLookups.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:863
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:883
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

github-actions[bot] commented 1 year ago

We have marked this test failure issue as stale because it has been inactive for 1 month. If this failure is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 5 days to keep the test failure queue tidy.