search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30k stars 3.79k forks source link

roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [self-delegated snaps] #72083

Closed cockroach-teamcity closed 2 years ago

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ d91fead28392841a943251842fbd43a0affb2eca:

          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1071
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:905
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1371
        Wraps: (2) monitor failure
        Wraps: (3) unexpected node event: 11: dead (exit status 137)
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

    cluster.go:1300,context.go:91,cluster.go:1288,test_runner.go:866: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3647465-1635401487-34-n12cpu4-geo --oneshot --ignore-empty-nodes: exit status 1 4: skipped
        1: 13290
        3: 12804
        2: 13472
        8: skipped
        6: 11783
        12: skipped
        7: 11853
        11: dead (exit status 137)
        9: 11501
        5: 12334
        10: 11563
        Error: UNCLASSIFIED_PROBLEM: 11: dead (exit status 137)
        (1) UNCLASSIFIED_PROBLEM
        Wraps: (2) attached stack trace
          -- stack trace:
          | main.glob..func14
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1175
          | main.wrap.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:281
          | github.com/spf13/cobra.(*Command).execute
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
          | github.com/spf13/cobra.(*Command).ExecuteC
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
          | github.com/spf13/cobra.(*Command).Execute
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
          | main.main
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2104
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:225
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1371
        Wraps: (3) 11: dead (exit status 137)
        Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) | See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #71802 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot branch-release-21.2]

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-10940

tbg commented 2 years ago

Repro'ed right away. But if I invoke the workload, it works:

$ roachprod run teamcity-5169311-1652696716-01-n12cpu4-geo:4 -- ./cockroach workload run tpcc --warehouses=3000 --workers=3000 --max-rate=490 --wait=false --ramp=15m0s --duration=45m0s --scatter --tolerate-errors '{pgurl:1-3,5-7,9-11}'
I220516 15:13:04.237342 1 workload/cli/run.go:414  [-] 1  creating load generator...
Initializing 3000 connections...
Initializing 0 idle connections...
Initializing 3000 workers and preparing statements...
I220516 15:13:13.858889 1 workload/cli/run.go:445  [-] 2  creating load generator... done (took 9.621546227s)
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
    1.0s        0            0.1            1.0     75.5     75.5     75.5     75.5 delivery
    1.0s        0            0.8            9.0     30.4     62.9     62.9     62.9 newOrder
    1.0s        0            0.1            1.0     96.5     96.5     96.5     96.5 orderStatus
    1.0s        0            0.7            7.0     16.8    285.2    285.2    285.2 payment
    1.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
    2.0s        0            0.0            0.5      0.0      0.0      0.0      0.0 delivery
    2.0s        0            0.0            4.5      0.0      0.0      0.0      0.0 newOrder
    2.0s        0            0.0            0.5      0.0      0.0      0.0      0.0 orderStatus
    2.0s        0            3.0            5.0    209.7    285.2    285.2    285.2 payment
    2.0s        0            1.0            0.5     48.2     48.2     48.2     48.2 stockLevel

It couldn't be something as silly as having shell expansion mess with the {}, I think, as we'd see that right around everywhere...

tbg commented 2 years ago

I think RunWithDetailsSingleNode is hiding the output from us. Running again at https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestStress/5170885?showRootCauses=true&expandBuildChangesSection=true with https://github.com/cockroachdb/cockroach/pull/81299.

tbg commented 2 years ago

BTW the raft receive queue oom once observed in this test can be seen/reproduced in https://github.com/cockroachdb/cockroach/pull/81289#issuecomment-1127991116

tbg commented 2 years ago

With #81299 the "reason" (🙄) reveals itself:

run_162811.027151198_n4_cockroach_workload_run_tpcc: 16:28:11 cluster.go:2012: running ./cockroach workload run tpcc --warehouses=3000 --workers=3000 --max-rate=490 --wait=false --ramp=15m0s --duration=45m0s --scatter --tolerate-errors {pgurl:1-3,5-7,9-11} on nodes: :4
teamcity-5170885-1652716498-01-n12cpu4-geo: ./cockroach workload run tp...
I220516 16:28:13.688025 1 workload/cli/run.go:414  [-] 1  creating load generator...
Initializing 3000 connections...
E220516 18:05:47.199375 1 workload/cli/run.go:437  [-] 2  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: context deadline exceeded
Error: failed to initialize the load generator: context deadline exceeded
LAST EXIT STATUS: 1run_162811.027151198_n4_cockroach_workload_run_tpcc: 18:05:47 cluster.go:2027: > Error for Node 4: Non-zero exit code: 1
tbg commented 2 years ago

From the first run:

teamcity-5169311-1652696716-01-n12cpu4-geo-0003> W220516 12:15:00.655602 78721 kv/kvclient/kvcoord/dist_sender.go:1615 ⋮ [n3,client=10.142.0.2:41040,user=root] 12904 slow range RPC: have been waiting 4593.01s (1 attempts) for RPC AdminScatter [‹/Table/108/1/2886/2/1077›,‹/Table/108/1/2916/6/352›) to r1075:‹/Table/108/1/2{886/2/1077-916/6/352}› [(n7,s7):5, (n2,s2):2, (n6,s6):4, next=6, gen=103, sticky=1652698458.115962514,0]; resp: ‹(err: <nil>), *roachpb.AdminScatterResponse›

12:15:00 is when workload gives up:

10:58:23 cluster.go:2014: > ./cockroach workload run tpcc --warehouses=3000 --workers=3000 --max-rate=490 --wait=false --ramp=15m0s --duration=45m0s --scatter --tolerate-errors {pgurl:1-3,5-7,9-11}
12:15:00 test_impl.go:318: test failure:        monitor.go:127,tpcc.go:1096,tpcc.go:931,test_runner.go:876: monitor failure: monitor task failed: Non-zero exit code: 1

so I think the problem here is that the workload's 60 minute timeout applies to the --scatter. The timeout is here

https://github.com/cockroachdb/cockroach/blob/ab6dc85f664da1344a5aa59dd49aa7c74d20ccef/pkg/workload/cli/run.go#L418-L441

and the scattering happens in Ops:

https://github.com/cockroachdb/cockroach/blob/c9b6b80c52d9970caac90c5933f03d117bb59c73/pkg/workload/tpcc/tpcc.go#L728-L740

So why did we get slower at scattering? From $ git log --topo-order --oneline --merges 97e72bacbeb9574f09f7475a62ef45c3e228183e and the test's history we get

97e72bacbe Merge #80499 <-- bad 41e26007da Merge #79976 #80408 #80730 6c893ef380 Merge #80363 #80741 <-- 🔥 run 2951494c21 Merge #80612 #80690 ccc1a27d21 Merge #80723 f06da926cd Merge #77329 #79968 #80274 #80341 #80502 #80654 <-- 🔥 run 7092ff4dab Merge #80303 #80667 #80702 <-- ✅ run 3831e2760c Merge #80269 #80679 7c45e41a8a Merge #80660 232e975c14 Merge #78193 #79788 #80696 efad149fa6 Merge #80366 #80640 2ce801c21a Merge #80644 a3ca09014c Merge #80606 72e485705a Merge #78961 <-- good

I think #80690 #77329 look like the most likely root causes here. I'll give them a shot.

tbg commented 2 years ago

Confirmed that https://github.com/cockroachdb/cockroach/commit/f06da926cd39a2c3b89b061c7bf356658d3524e1 is the first failing commit, which with high likelihood points to self-delegated snapshots as the troublemaker. https://github.com/cockroachdb/cockroach/pull/77329

Reassigning to the KV team for investigation.

cc @cockroachdb/kv

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 2c264bd12c15b8811553adc9d90f9230065f3ff3:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1096,tpcc.go:931,test_runner.go:876: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1096
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:931
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ a35b848edb3f5562199b0f56bc27a6c576073d5d:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1096,tpcc.go:931,test_runner.go:876: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1096
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:931
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ ce66acdbba801de88f1dd645eaedeb3834f23dbd:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1096,tpcc.go:931,test_runner.go:876: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1096
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:931
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 92947c29c55ff909f50a5e625811d34a1bbe71f7:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:876: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 7dd1c51f6b5802e32bafd82e46747f349836592f:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:884: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

aayushshah15 commented 2 years ago

Based on the simple benchmark in https://github.com/cockroachdb/cockroach/pull/81713, it doesn't seem like there's anything odd going at the level of replication changes at least.

benchdiff --new=selfDelegatedSnapshots --old=beforeDelegatedSnaps --post-checkout='make buildshort' --count=100 --run='EmptyRebalance' ./pkg/kv/kvserver;
test binaries already exist for '102bdc5'; skipping build
test binaries already exist for 'c5e3bcf'; skipping build

  pkg=1/1 iter=100/100 cockroachdb/cockroach/pkg/kv/kvserver \

name                          old time/op    new time/op    delta
EmptyRebalance/add-remove-24    39.3ms ± 2%    39.6ms ± 2%  +0.92%  (p=0.000 n=98+97)

name                          old alloc/op   new alloc/op   delta
EmptyRebalance/add-remove-24    3.24MB ± 5%    3.29MB ± 5%  +1.28%  (p=0.000 n=97+94)

name                          old allocs/op  new allocs/op  delta
EmptyRebalance/add-remove-24     20.2k ± 1%     20.5k ± 1%  +1.65%  (p=0.000 n=94+93)
cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 48e48db89eb3cd6bc38f3631364c516181811280:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:884: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 12198a51408e7333cd4f96b221e6734239479765:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:884: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 7a17498a9679853612cb88d82a4a3952d1015f94:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:884: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ d9fe5f67c75b1b59fc297bf4509a139c640b835b:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:884: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 1e2cc61b58dc14386bb68dca59814874648931c2:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:884: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ e6815947a050e32f21e983aa30dc74ab2a247af3:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:884: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ e6815947a050e32f21e983aa30dc74ab2a247af3:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:884: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 662cc5c3070e6d64d155a9cc9f33253ee5d99ee9:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:884: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 2181204e9c7ac6b316573073b6b8010f43920f8b:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:884: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #82125 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-kv branch-release-21.2 release-blocker] - #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 90b5db50e1e1cdb0315d8b094081d261e6dcb336:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:884: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #82125 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-kv branch-release-21.2 release-blocker sync-me sync-me-5] - #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 0fdac09ca1119b494661e1e1f64ea291d8649782:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:884: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 2c08debd6d8c89019f26a3e58cf3b5d3d97b6495:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:884: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 5bd3fdd33a351564e7af5e64ce931e82db5e23a1:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:884: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 175eebd03e5f2674a2d1f6bfea8bd72f131a6532:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:884: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ c070f2c70e0c92573a99dcb80b8a0cb0532bd06f:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:884: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 3b04ced1801d02b5dd50dab0c93f957c5c1c5720:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:884: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 45ee8026c1997df738ee184fe6a606e927c0a8c5:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:884: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 45ee8026c1997df738ee184fe6a606e927c0a8c5:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:884: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ b0a8d28feb8befa417933d0a2ee99511b16414a1:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:884: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ aadbaf97b4e6092ad6978a28e2735715d64d9f10:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:884: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 47df1bc3a28f705e56ff4efc3d81c6fc90e096b5:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:884: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 8d34ef1ea15850ee1c70470610b6652df4c317de:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:897: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 8d34ef1ea15850ee1c70470610b6652df4c317de:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:897: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 8d34ef1ea15850ee1c70470610b6652df4c317de:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:897: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 377b113bd7e7b679c55a0f93e0d8ab5dbb6f2e97:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:897: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1 no-test-failure-activity]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 13cb2f6c40e3146fed8d931f65f89da9b42ce2c1:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:896: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1 no-test-failure-activity]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 7a1ced4a49d28f66f381e78733bcab510e2cbafe:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:896: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1 no-test-failure-activity]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 7af6c0d3e2085cafda90d5f0b15d06f6a360f154:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:896: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1 no-test-failure-activity]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ fc7ae339c3f85ad2a2b28643e47d7c20768fe237:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:896: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1 no-test-failure-activity]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ a611410e3babe97a14152381b89060e1bafa2e27:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:896: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #80856 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [C-test-failure O-roachtest O-robot T-bulkio T-kv branch-release-21.1 no-test-failure-activity]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 7f8fef4c3f24ec2b6f9a3db1eb38756da68b3d5a:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:896: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ ac11a55499120ffd33016feebc47b4ed43a21007:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1103,tpcc.go:938,test_runner.go:896: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1103
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:938
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ e1e99da678e29db4ef21813423f012273c99add2:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1101,tpcc.go:936,test_runner.go:896: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1101
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:936
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 78caa5c3a8ecd0d0a612798a106b626edb3db2ee:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1101,tpcc.go:936,test_runner.go:896: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1101
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:936
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 4e8134033890c956d39020f28a0e99b21b32649f:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1101,tpcc.go:936,test_runner.go:896: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1101
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:936
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 3c9b17113488d2ee6929936aa6ec48396f3ed71c:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1101,tpcc.go:936,test_runner.go:896: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1101
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:936
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ b9a165c03643ed43a0d45b16c0a795244543a2fe:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:127,tpcc.go:1101,tpcc.go:936,test_runner.go:896: monitor failure: monitor task failed: Non-zero exit code: 1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1101
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:936
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) monitor task failed
        Wraps: (5) Non-zero exit code: 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

aayushshah15 commented 2 years ago

This failure is almost certainly due to https://github.com/cockroachdb/cockroach/issues/83687 and will be fixed by https://github.com/cockroachdb/cockroach/pull/83686.