Closed cockroach-teamcity closed 1 year ago
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ f4c16510270c6902e69c3e26e3c125532b416c1b:
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:142
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:258
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6309
| runtime.main
| /usr/local/go/src/runtime/proc.go:208
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
cluster.go:1301,context.go:91,cluster.go:1289,test_runner.go:866: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3699635-1636442973-02-n2cpu8 --oneshot --ignore-empty-nodes: exit status 1 2: skipped
1: dead (exit status 137)
Error: UNCLASSIFIED_PROBLEM: 1: dead (exit status 137)
(1) UNCLASSIFIED_PROBLEM
Wraps: (2) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod.Monitor
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:635
| main.glob..func14
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:582
| main.wrap.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:123
| github.com/spf13/cobra.(*Command).execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
| github.com/spf13/cobra.(*Command).ExecuteC
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
| github.com/spf13/cobra.(*Command).Execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
| main.main
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1183
| runtime.main
| /usr/local/go/src/runtime/proc.go:225
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (3) 1: dead (exit status 137)
Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) | See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
/cc @cockroachdb/kv-triage
@sumeerbhola this is the admission control I/O overload test https://github.com/cockroachdb/cockroach/blob/7e5741bfc663e11427eac9f6618804562aec2351/pkg/cmd/roachtest/tests/kv.go#L151-L152
It was unskipped in 7e5741bfc663e11427eac9f6618804562aec2351. Looks like the node is crashing despite admission control being enabled.
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 13669f9c9bd92a4c3b0378a558d7735f122c4e72:
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:142
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:258
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6309
| runtime.main
| /usr/local/go/src/runtime/proc.go:208
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
cluster.go:1301,context.go:91,cluster.go:1289,test_runner.go:866: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3727704-1636874044-02-n2cpu8 --oneshot --ignore-empty-nodes: exit status 1 2: skipped
1: dead (exit status 137)
Error: UNCLASSIFIED_PROBLEM: 1: dead (exit status 137)
(1) UNCLASSIFIED_PROBLEM
Wraps: (2) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod.Monitor
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:596
| main.glob..func14
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:569
| main.wrap.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:123
| github.com/spf13/cobra.(*Command).execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
| github.com/spf13/cobra.(*Command).ExecuteC
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
| github.com/spf13/cobra.(*Command).Execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
| main.main
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1170
| runtime.main
| /usr/local/go/src/runtime/proc.go:225
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (3) 1: dead (exit status 137)
Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ a5e32e1d18aa07a65eb44063177a6c196623f360:
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:142
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:258
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
cluster.go:1308,context.go:91,cluster.go:1296,test_runner.go:866: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3781442-1637651447-02-n2cpu8 --oneshot --ignore-empty-nodes: exit status 1 2: skipped
1: dead (exit status 137)
Error: UNCLASSIFIED_PROBLEM: 1: dead (exit status 137)
(1) UNCLASSIFIED_PROBLEM
Wraps: (2) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod.Monitor
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:568
| main.glob..func14
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:518
| main.wrap.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:69
| github.com/spf13/cobra.(*Command).execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
| github.com/spf13/cobra.(*Command).ExecuteC
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
| github.com/spf13/cobra.(*Command).Execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
| main.main
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:909
| runtime.main
| /usr/local/go/src/runtime/proc.go:255
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (3) 1: dead (exit status 137)
Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ a132b15e6c9705a6922c2e476936597c6670e072:
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:142
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:258
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:778
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
cluster.go:1308,context.go:91,cluster.go:1296,test_runner.go:867: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3788142-1637737745-01-n2cpu8 --oneshot --ignore-empty-nodes: exit status 1 2: skipped
1: dead (exit status 137)
Error: UNCLASSIFIED_PROBLEM: 1: dead (exit status 137)
(1) UNCLASSIFIED_PROBLEM
Wraps: (2) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod.Monitor
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:568
| main.glob..func14
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:518
| main.wrap.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:69
| github.com/spf13/cobra.(*Command).execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
| github.com/spf13/cobra.(*Command).ExecuteC
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
| github.com/spf13/cobra.(*Command).Execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
| main.main
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:909
| runtime.main
| /usr/local/go/src/runtime/proc.go:255
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (3) 1: dead (exit status 137)
Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 40f11fead0a0453969634f8ddb0502c1f78b2806:
| 1184.0s 0 0.0 178.4 0.0 0.0 0.0 0.0 write
| 1185.0s 0 0.0 178.2 0.0 0.0 0.0 0.0 write
| 1186.0s 0 0.0 178.1 0.0 0.0 0.0 0.0 write
| 1187.0s 0 0.0 177.9 0.0 0.0 0.0 0.0 write
| 1188.0s 0 0.0 177.8 0.0 0.0 0.0 0.0 write
| 1189.0s 0 0.0 177.6 0.0 0.0 0.0 0.0 write
| 1190.0s 0 0.0 177.5 0.0 0.0 0.0 0.0 write
| 1191.0s 0 8.0 177.3 31138.5 36507.2 36507.2 36507.2 write
| 1192.0s 0 611.3 177.7 27917.3 36507.2 36507.2 36507.2 write
| 1193.0s 0 331.2 177.8 26843.5 36507.2 36507.2 38654.7 write
| 1194.0s 0 324.1 178.0 27917.3 38654.7 38654.7 38654.7 write
| 1195.0s 0 334.0 178.1 28991.0 38654.7 38654.7 38654.7 write
| 1196.0s 0 343.0 178.2 30064.8 40802.2 40802.2 40802.2 write
| 1197.0s 0 14.0 178.1 32212.3 40802.2 40802.2 40802.2 write
Wraps: (4) exit status 20
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
monitor.go:128,kv.go:142,kv.go:258,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:142
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:258
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
cluster.go:1343,context.go:91,cluster.go:1333,test_runner.go:867: dead node detection: 1: dead (exit status 137)
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 2c014c47c1a242f504f6d595bfd79c0edc20b90a:
| 746.0s 0 0.0 192.1 0.0 0.0 0.0 0.0 write
| 747.0s 0 36.0 191.9 20401.1 21474.8 21474.8 21474.8 write
| 748.0s 0 215.0 192.0 21474.8 21474.8 21474.8 22548.6 write
| 749.0s 0 189.8 192.0 21474.8 23622.3 23622.3 23622.3 write
| 750.0s 0 197.1 192.0 22548.6 22548.6 23622.3 23622.3 write
| 751.0s 0 50.0 191.8 21474.8 22548.6 22548.6 22548.6 write
| 752.0s 0 23.0 191.6 23622.3 23622.3 23622.3 23622.3 write
| 753.0s 0 53.0 191.4 23622.3 24696.1 24696.1 24696.1 write
| 754.0s 0 31.0 191.2 24696.1 24696.1 24696.1 24696.1 write
| 755.0s 0 0.0 190.9 0.0 0.0 0.0 0.0 write
| 756.0s 0 11.0 190.7 26843.5 26843.5 26843.5 26843.5 write
| 757.0s 0 20.0 190.4 27917.3 27917.3 27917.3 27917.3 write
| 758.0s 0 21.0 190.2 27917.3 27917.3 27917.3 27917.3 write
| 759.0s 0 12.0 190.0 28991.0 28991.0 28991.0 28991.0 write
Wraps: (4) exit status 20
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
monitor.go:128,kv.go:142,kv.go:258,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:142
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:258
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
cluster.go:1343,context.go:91,cluster.go:1333,test_runner.go:867: dead node detection: 1: dead (exit status 137)
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 7097a9015f1a09c7dee4fbdbcc6bde82121f657b:
| 693.0s 0 82.9 190.5 47244.6 47244.6 47244.6 47244.6 write
| 694.0s 0 66.0 190.3 47244.6 47244.6 47244.6 47244.6 write
| 695.0s 0 69.0 190.1 47244.6 47244.6 47244.6 47244.6 write
| 696.0s 0 69.0 189.9 47244.6 49392.1 49392.1 49392.1 write
| 697.0s 0 53.0 189.7 40802.2 49392.1 49392.1 49392.1 write
| 698.0s 0 565.6 190.3 25769.8 47244.6 49392.1 49392.1 write
| 699.0s 0 307.2 190.4 25769.8 47244.6 49392.1 51539.6 write
| 700.0s 0 318.3 190.6 23622.3 47244.6 49392.1 51539.6 write
| _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
| 701.0s 0 325.8 190.8 25769.8 49392.1 51539.6 51539.6 write
| 702.0s 0 322.1 191.0 22548.6 49392.1 51539.6 53687.1 write
| 703.0s 0 387.7 191.3 22548.6 49392.1 53687.1 53687.1 write
| 704.0s 0 307.0 191.4 24696.1 53687.1 53687.1 55834.6 write
| 705.0s 0 240.0 191.5 18253.6 53687.1 55834.6 57982.1 write
Wraps: (4) exit status 20
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
monitor.go:128,kv.go:142,kv.go:258,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:142
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:258
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
cluster.go:1343,context.go:91,cluster.go:1333,test_runner.go:867: dead node detection: 1: dead (exit status 137)
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ a80cfbee826f70988381a0a85c0fe7aba0115484:
| 769.0s 0 45.0 194.9 20401.1 20401.1 20401.1 20401.1 write
| 770.0s 0 22.0 194.6 20401.1 20401.1 20401.1 20401.1 write
| 771.0s 0 31.0 194.4 21474.8 21474.8 21474.8 21474.8 write
| 772.0s 0 25.0 194.2 22548.6 22548.6 22548.6 22548.6 write
| 773.0s 0 0.0 194.0 0.0 0.0 0.0 0.0 write
| 774.0s 0 3.0 193.7 24696.1 24696.1 24696.1 24696.1 write
| 775.0s 0 14.0 193.5 25769.8 25769.8 25769.8 25769.8 write
| 776.0s 0 0.0 193.2 0.0 0.0 0.0 0.0 write
| 777.0s 0 0.0 193.0 0.0 0.0 0.0 0.0 write
| 778.0s 0 8.0 192.8 27917.3 27917.3 27917.3 27917.3 write
| 779.0s 0 2.0 192.5 28991.0 28991.0 28991.0 28991.0 write
| 780.0s 0 17.0 192.3 30064.8 30064.8 30064.8 30064.8 write
| _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
| 781.0s 0 7.0 192.0 31138.5 31138.5 31138.5 31138.5 write
Wraps: (4) exit status 20
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
monitor.go:128,kv.go:142,kv.go:258,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:142
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:258
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
cluster.go:1343,context.go:91,cluster.go:1333,test_runner.go:867: dead node detection: 1: dead (exit status 137)
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 2b8b0f64c63a376784926667f5fbffa4c4fcc60b:
| 1145.0s 0 194.9 183.4 16643.0 19327.4 19327.4 19327.4 write
| 1146.0s 0 246.9 183.5 17179.9 19327.4 20401.1 20401.1 write
| 1147.0s 0 224.1 183.5 18253.6 19327.4 20401.1 20401.1 write
| 1148.0s 0 213.1 183.6 18253.6 19327.4 19327.4 19327.4 write
| 1149.0s 0 226.9 183.6 18253.6 19327.4 19327.4 19327.4 write
| 1150.0s 0 162.0 183.6 18253.6 19327.4 19327.4 19327.4 write
| 1151.0s 0 157.1 183.5 18253.6 19327.4 19327.4 19327.4 write
| 1152.0s 0 54.0 183.4 19327.4 19327.4 19327.4 19327.4 write
| 1153.0s 0 47.0 183.3 20401.1 20401.1 20401.1 20401.1 write
| 1154.0s 0 30.0 183.2 21474.8 21474.8 21474.8 21474.8 write
| 1155.0s 0 42.0 183.1 22548.6 22548.6 22548.6 22548.6 write
| 1156.0s 0 41.0 182.9 22548.6 22548.6 22548.6 22548.6 write
| 1157.0s 0 44.0 182.8 23622.3 23622.3 23622.3 23622.3 write
| 1158.0s 0 0.0 182.7 0.0 0.0 0.0 0.0 write
Wraps: (4) exit status 20
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
monitor.go:128,kv.go:142,kv.go:258,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:142
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:258
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
cluster.go:1343,context.go:91,cluster.go:1333,test_runner.go:867: dead node detection: 1: dead (exit status 137)
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 260b7930e717c76bc421ae3f34e9a939fc896b50:
| 1129.0s 0 95.0 174.0 47244.6 49392.1 49392.1 49392.1 write
| 1130.0s 0 153.0 174.0 47244.6 49392.1 49392.1 49392.1 write
| 1131.0s 0 141.0 174.0 45097.2 47244.6 49392.1 49392.1 write
| 1132.0s 0 561.5 174.3 20401.1 45097.2 47244.6 47244.6 write
| 1133.0s 0 247.3 174.4 19327.4 45097.2 49392.1 49392.1 write
| 1134.0s 0 259.0 174.4 19327.4 45097.2 47244.6 47244.6 write
| 1135.0s 0 283.0 174.5 16106.1 47244.6 49392.1 51539.6 write
| 1136.0s 0 274.9 174.6 18253.6 47244.6 49392.1 49392.1 write
| 1137.0s 0 229.1 174.7 12884.9 49392.1 51539.6 53687.1 write
| 1138.0s 0 241.9 174.7 6442.5 47244.6 51539.6 53687.1 write
| 1139.0s 0 255.9 174.8 8053.1 51539.6 53687.1 53687.1 write
| 1140.0s 0 288.1 174.9 8589.9 51539.6 53687.1 55834.6 write
| _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
| 1141.0s 0 232.2 175.0 9126.8 51539.6 53687.1 55834.6 write
Wraps: (4) exit status 20
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
monitor.go:128,kv.go:142,kv.go:258,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:142
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:258
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
cluster.go:1343,context.go:91,cluster.go:1333,test_runner.go:867: dead node detection: 1: dead (exit status 137)
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 835ceca5d25d4a62233ddde4f493dbcf68302f1e:
| 1051.0s 0 221.9 182.4 18253.6 18253.6 18253.6 18253.6 write
| 1052.0s 0 223.9 182.4 18253.6 19327.4 19327.4 19327.4 write
| 1053.0s 0 181.2 182.4 18253.6 19327.4 19327.4 19327.4 write
| 1054.0s 0 104.0 182.4 19327.4 19327.4 19327.4 19327.4 write
| 1055.0s 0 42.0 182.2 20401.1 20401.1 20401.1 20401.1 write
| 1056.0s 0 58.0 182.1 19327.4 20401.1 21474.8 21474.8 write
| 1057.0s 0 15.0 182.0 20401.1 20401.1 20401.1 20401.1 write
| 1058.0s 0 13.0 181.8 21474.8 21474.8 21474.8 21474.8 write
| 1059.0s 0 9.0 181.6 22548.6 22548.6 22548.6 22548.6 write
| 1060.0s 0 14.0 181.5 23622.3 23622.3 23622.3 23622.3 write
| _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
| 1061.0s 0 16.0 181.3 24696.1 24696.1 24696.1 24696.1 write
| 1062.0s 0 11.0 181.2 25769.8 25769.8 25769.8 25769.8 write
| 1063.0s 0 6.0 181.0 26843.5 26843.5 26843.5 26843.5 write
Wraps: (4) exit status 20
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
monitor.go:128,kv.go:142,kv.go:258,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:142
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:258
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
cluster.go:1339,context.go:91,cluster.go:1329,test_runner.go:867: dead node detection: 1: dead (exit status 137)
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
The node has log statements of the form
E211206 07:58:48.611382 22427 kv/kvserver/replica_write.go:285 ⋮ [n1,s1,r870/1:‹/Table/56/1/-26{99748…-81319…}›] 133853 range unavailable: have been waiting 15.00s for proposing command Put [‹/Table/56/1/-2694689559611899965/0›,‹/Min›), EndTxn(commit) [‹/Table/56/1/-2694689559611899965/0›], [txn: d6b6a4ad], [can-forward-ts].
E211206 07:58:48.611382 22427 kv/kvserver/replica_write.go:285 ⋮ [n1,s1,r870/1:‹/Table/56/1/-26{99748…-81319…}›] 133853 +This range is likely unavailable.
I211206 07:58:48.625194 28782 kv/kvserver/replica_write.go:187 ⋮ [n1,s1,r936/1:‹/Table/56/1/-5{11385…-09542…}›] 133868 slow command Put [‹/Table/56/1/-5112559483361806260/0›,‹/Min›), EndTxn(commit) [‹/Table/56/1/-5112559483361806260/0›], [txn: f24eb0f8], [can-forward-ts] finished after 16.46s with error <nil>
W211206 07:58:52.898875 352 kv/kvserver/spanlatch/manager.go:526 ⋮ [n1,s1,r35/1:‹/Table/{39-40}›] 133942 have been waiting 15s to acquire ‹write› latch ‹/Local/RangeID/35/r/AbortSpan/"b37d0d1f-9258-4b86-8fed-1dcfe466fb97"@0,0›, held by ‹read› latch ‹/Local/RangeID/35/r/AbortSpan/"b37d0d1f-9258-4b86-8fed-1dcfe466fb97"@0,0›
Which should not be caused by overload since admission control is on and should have throttled KV requests before they reached the above code (and corresponding latency measurements). There are also numerous statements indicating raft level slowness.
W211206 07:58:54.030732 262 kv/kvserver/store_raft.go:520 ⋮ [n1,s1,r591/1:‹/Table/56/1/78{59676…-78104…}›] 133978 handle raft ready: 0.8s [applied=5, batches=1, state_assertions=0]
I'll rerun this manually and look at the metrics etc.
The slowness is correlated with write stalls due to "memtable count limit reached".
BatchRequest
is incorrect. This increases the likelihood of write stalls.
I211206 07:58:32.602147 341 util/admission/granter.go:1609 â‹® [-] 125495 IO overload on store 1 (files 917, sub-levels 26): admitted: 8886, added: 458649546, removed (0, 127797425), admit: (1237.990889, 1738)
I211206 07:58:47.601883 341 util/admission/granter.go:1609 â‹® [-] 133502 IO overload on store 1 (files 866, sub-levels 21): admitted: 1738, added: 534509826, removed (487858690, 307828057), admit: (500.463356, 1119)
I211206 07:59:02.602118 341 util/admission/granter.go:1609 â‹® [-] 134029 IO overload on store 1 (files 1206, sub-levels 28): admitted: 1149, added: 527068422, removed (84938142, 196383099), admit: (214.055871, 666)
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 5cd5be83218475ef0cc3c62a856b8d9d22cf142b:
| 1322.0s 0 344.0 172.4 16643.0 60129.5 64424.5 64424.5 write
| 1323.0s 0 316.2 172.5 21474.8 62277.0 64424.5 66572.0 write
| 1324.0s 0 207.2 172.5 15569.3 62277.0 66572.0 66572.0 write
| 1325.0s 0 255.9 172.6 13421.8 62277.0 66572.0 68719.5 write
| 1326.0s 0 425.8 172.8 14495.5 64424.5 68719.5 68719.5 write
| 1327.0s 0 401.0 172.9 14495.5 64424.5 68719.5 73014.4 write
| 1328.0s 0 321.0 173.1 12884.9 66572.0 68719.5 73014.4 write
| 1329.0s 0 121.1 173.0 13421.8 66572.0 73014.4 73014.4 write
| 1330.0s 0 60.0 172.9 13958.6 57982.1 64424.5 73014.4 write
| 1331.0s 0 288.8 173.0 15569.3 64424.5 73014.4 73014.4 write
| 1332.0s 0 419.3 173.2 16643.0 62277.0 73014.4 77309.4 write
| 1333.0s 0 451.0 173.4 14495.5 57982.1 73014.4 77309.4 write
| 1334.0s 0 320.5 173.5 11811.2 21474.8 23622.3 31138.5 write
| 1335.0s 0 0.0 173.4 0.0 0.0 0.0 0.0 write
Wraps: (4) exit status 20
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
monitor.go:128,kv.go:151,kv.go:266,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:151
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:266
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
cluster.go:1339,context.go:91,cluster.go:1329,test_runner.go:867: dead node detection: 1: dead (exit status 137)
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ acf40e97f64a97338cbebd20d7fbe176c4e37c34:
| 1494.0s 0 297.2 171.0 25769.8 49392.1 49392.1 49392.1 write
| 1495.0s 0 319.0 171.1 26843.5 49392.1 51539.6 51539.6 write
| 1496.0s 0 372.0 171.2 26843.5 51539.6 51539.6 51539.6 write
| 1497.0s 0 369.1 171.3 22548.6 51539.6 53687.1 53687.1 write
| 1498.0s 0 369.9 171.4 20401.1 53687.1 53687.1 53687.1 write
| 1499.0s 0 375.0 171.6 18253.6 53687.1 55834.6 55834.6 write
| 1500.0s 0 342.8 171.7 21474.8 55834.6 55834.6 55834.6 write
| _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
| 1501.0s 0 185.2 171.7 16643.0 53687.1 55834.6 55834.6 write
| 1502.0s 0 387.7 171.9 16643.0 55834.6 57982.1 57982.1 write
| 1503.0s 0 14.0 171.7 5368.7 49392.1 55834.6 55834.6 write
| 1504.0s 0 410.1 171.9 18253.6 57982.1 60129.5 60129.5 write
| 1505.0s 0 439.0 172.1 12348.0 60129.5 60129.5 62277.0 write
| 1506.0s 0 225.3 172.1 12884.9 60129.5 62277.0 62277.0 write
Wraps: (4) exit status 20
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
monitor.go:128,kv.go:151,kv.go:266,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:151
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:266
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
cluster.go:1339,context.go:91,cluster.go:1329,test_runner.go:867: dead node detection: 1: dead (exit status 137)
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ dd7fd4e5a371ce4437fce383b9674cba2e212fe6:
| 1755.0s 0 0.0 166.2 0.0 0.0 0.0 0.0 write
| 1756.0s 0 12.0 166.1 30064.8 30064.8 30064.8 30064.8 write
| 1757.0s 0 18.0 166.0 31138.5 31138.5 31138.5 31138.5 write
| 1758.0s 0 6.0 166.0 32212.3 32212.3 32212.3 32212.3 write
| 1759.0s 0 6.0 165.9 32212.3 33286.0 33286.0 33286.0 write
| 1760.0s 0 13.0 165.8 33286.0 33286.0 33286.0 33286.0 write
| _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
| 1761.0s 0 84.0 165.7 25769.8 34359.7 34359.7 34359.7 write
| 1762.0s 0 639.6 166.0 26843.5 36507.2 36507.2 36507.2 write
| 1763.0s 0 325.2 166.1 27917.3 36507.2 36507.2 36507.2 write
| 1764.0s 0 270.0 166.2 26843.5 36507.2 38654.7 38654.7 write
| 1765.0s 0 337.0 166.2 26843.5 38654.7 38654.7 38654.7 write
| 1766.0s 0 302.1 166.3 26843.5 38654.7 38654.7 40802.2 write
| 1767.0s 0 322.0 166.4 27917.3 40802.2 40802.2 40802.2 write
Wraps: (4) exit status 20
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
monitor.go:128,kv.go:151,kv.go:266,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:151
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:266
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
cluster.go:1339,context.go:91,cluster.go:1329,test_runner.go:867: dead node detection: 1: dead (exit status 137)
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 2bf850847554519dc139289615e1a061b6a251de:
| 1025.0s 0 95.0 179.6 18253.6 18253.6 18253.6 18253.6 write
| 1026.0s 0 43.0 179.4 19327.4 19327.4 19327.4 19327.4 write
| 1027.0s 0 2.0 179.3 20401.1 20401.1 20401.1 20401.1 write
| 1028.0s 0 0.0 179.1 0.0 0.0 0.0 0.0 write
| 1029.0s 0 0.0 178.9 0.0 0.0 0.0 0.0 write
| 1030.0s 0 0.0 178.7 0.0 0.0 0.0 0.0 write
| 1031.0s 0 0.0 178.6 0.0 0.0 0.0 0.0 write
| 1032.0s 0 0.0 178.4 0.0 0.0 0.0 0.0 write
| 1033.0s 0 0.0 178.2 0.0 0.0 0.0 0.0 write
| 1034.0s 0 0.0 178.0 0.0 0.0 0.0 0.0 write
| 1035.0s 0 0.0 177.9 0.0 0.0 0.0 0.0 write
| 1036.0s 0 0.0 177.7 0.0 0.0 0.0 0.0 write
| 1037.0s 0 0.0 177.5 0.0 0.0 0.0 0.0 write
| 1038.0s 0 0.0 177.4 0.0 0.0 0.0 0.0 write
| 1039.0s 0 0.0 177.2 0.0 0.0 0.0 0.0 write
| 1040.0s 0 0.0 177.0 0.0 0.0 0.0 0.0 write
| _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
| 1041.0s 0 0.0 176.8 0.0 0.0 0.0 0.0 write
| 1042.0s 0 39.0 176.7 28991.0 34359.7 36507.2 36507.2 write
| 1043.0s 0 392.3 176.9 26843.5 36507.2 36507.2 36507.2 write
| 1044.0s 0 312.1 177.1 27917.3 36507.2 36507.2 36507.2 write
| 1045.0s 0 325.9 177.2 27917.3 38654.7 38654.7 38654.7 write
| 1046.0s 0 335.1 177.3 28991.0 38654.7 38654.7 38654.7 write
Wraps: (4) secondary error attachment
| exit status 20
| (1) exit status 20
| Error types: (1) *exec.ExitError
Wraps: (5) context canceled
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString
monitor.go:128,kv.go:151,kv.go:266,test_runner.go:779: monitor failure: unexpected node event: 1: dead (exit status 137)
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:151
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:266
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (2) monitor failure
Wraps: (3) unexpected node event: 1: dead (exit status 137)
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString
cluster.go:1339,context.go:91,cluster.go:1329,test_runner.go:867: dead node detection: 1: dead (exit status 137)
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 8c1528fff5c132b971428bc7f18bf0ea5eb16799:
| 1638.0s 0 14.0 167.9 28991.0 28991.0 28991.0 28991.0 write
| 1639.0s 0 17.0 167.8 30064.8 30064.8 30064.8 30064.8 write
| 1640.0s 0 15.0 167.7 31138.5 31138.5 31138.5 31138.5 write
| _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
| 1641.0s 0 5.0 167.6 32212.3 32212.3 32212.3 32212.3 write
| 1642.0s 0 38.0 167.5 26843.5 33286.0 33286.0 33286.0 write
| 1643.0s 0 466.1 167.7 24696.1 33286.0 34359.7 34359.7 write
| 1644.0s 0 350.5 167.8 25769.8 34359.7 36507.2 36507.2 write
| 1645.0s 0 169.1 167.8 24696.1 34359.7 36507.2 36507.2 write
| 1646.0s 0 321.9 167.9 25769.8 36507.2 36507.2 36507.2 write
| 1647.0s 0 260.1 167.9 26843.5 36507.2 38654.7 38654.7 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ f3710ae65e327d0795123029db679f8c9f3e610e:
| 721.0s 0 0.0 189.9 0.0 0.0 0.0 0.0 write
| 722.0s 0 0.0 189.6 0.0 0.0 0.0 0.0 write
| 723.0s 0 0.0 189.4 0.0 0.0 0.0 0.0 write
| 724.0s 0 0.0 189.1 0.0 0.0 0.0 0.0 write
| 725.0s 0 0.0 188.8 0.0 0.0 0.0 0.0 write
| 726.0s 0 0.0 188.6 0.0 0.0 0.0 0.0 write
| 727.0s 0 155.0 188.5 25769.8 33286.0 34359.7 34359.7 write
| 728.0s 0 400.4 188.8 25769.8 33286.0 34359.7 34359.7 write
| 729.0s 0 290.2 189.0 25769.8 34359.7 36507.2 36507.2 write
| 730.0s 0 301.1 189.1 27917.3 36507.2 36507.2 36507.2 write
| 731.0s 0 307.8 189.3 25769.8 36507.2 36507.2 38654.7 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 01c0c739dd83b3e662d715404f2193d747f4fb9e:
| 1081.0s 0 19.0 183.7 23622.3 24696.1 24696.1 24696.1 write
| 1082.0s 0 6.0 183.5 24696.1 24696.1 24696.1 24696.1 write
| 1083.0s 0 12.0 183.4 25769.8 25769.8 25769.8 25769.8 write
| 1084.0s 0 14.0 183.2 26843.5 26843.5 26843.5 26843.5 write
| 1085.0s 0 16.0 183.1 27917.3 27917.3 27917.3 27917.3 write
| 1086.0s 0 17.0 182.9 28991.0 28991.0 28991.0 28991.0 write
| 1087.0s 0 31.0 182.8 30064.8 30064.8 30064.8 30064.8 write
| 1088.0s 0 27.0 182.6 31138.5 31138.5 31138.5 31138.5 write
| 1089.0s 0 13.0 182.5 31138.5 32212.3 32212.3 32212.3 write
| 1090.0s 0 7.0 182.3 32212.3 32212.3 32212.3 32212.3 write
| 1091.0s 0 15.0 182.2 33286.0 33286.0 33286.0 33286.0 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 5ac733bb4927020bc1c52da24b2591742fde8e1f:
| 1489.0s 0 100.1 167.1 42949.7 42949.7 42949.7 42949.7 write
| 1490.0s 0 93.9 167.0 42949.7 42949.7 42949.7 42949.7 write
| 1491.0s 0 14.0 166.9 42949.7 42949.7 42949.7 42949.7 write
| 1492.0s 0 256.8 167.0 22548.6 42949.7 42949.7 42949.7 write
| 1493.0s 0 456.5 167.2 23622.3 40802.2 42949.7 45097.2 write
| 1494.0s 0 285.3 167.3 24696.1 42949.7 45097.2 45097.2 write
| 1495.0s 0 305.0 167.4 24696.1 38654.7 45097.2 47244.6 write
| 1496.0s 0 327.0 167.5 23622.3 40802.2 45097.2 47244.6 write
| 1497.0s 0 306.9 167.6 25769.8 42949.7 47244.6 49392.1 write
| 1498.0s 0 332.2 167.7 24696.1 42949.7 47244.6 47244.6 write
| 1499.0s 0 85.0 167.6 24696.1 42949.7 49392.1 49392.1 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 3471dc50db8406f3faa638940492dbe0cab53435:
| 799.0s 0 18.0 182.3 36507.2 36507.2 36507.2 36507.2 write
| 800.0s 0 18.0 182.1 36507.2 36507.2 36507.2 36507.2 write
| _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
| 801.0s 0 13.0 181.9 38654.7 38654.7 38654.7 38654.7 write
| 802.0s 0 0.0 181.7 0.0 0.0 0.0 0.0 write
| 803.0s 0 399.5 181.9 31138.5 38654.7 40802.2 40802.2 write
| 804.0s 0 288.9 182.1 31138.5 38654.7 40802.2 40802.2 write
| 805.0s 0 313.2 182.2 32212.3 40802.2 40802.2 40802.2 write
| 806.0s 0 287.8 182.4 32212.3 42949.7 42949.7 42949.7 write
| 807.0s 0 359.9 182.6 32212.3 42949.7 42949.7 45097.2 write
| 808.0s 0 264.2 182.7 32212.3 42949.7 45097.2 45097.2 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ fa9480a8b661b31a3c5e4c3428595b3e7c320700:
| 1218.0s 0 6.0 177.2 28991.0 28991.0 28991.0 28991.0 write
| 1219.0s 0 18.0 177.1 30064.8 30064.8 30064.8 30064.8 write
| 1220.0s 0 13.0 176.9 31138.5 31138.5 31138.5 31138.5 write
| _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
| 1221.0s 0 13.0 176.8 31138.5 32212.3 32212.3 32212.3 write
| 1222.0s 0 141.0 176.8 26843.5 32212.3 33286.0 33286.0 write
| 1223.0s 0 342.5 176.9 24696.1 33286.0 33286.0 33286.0 write
| 1224.0s 0 324.8 177.0 25769.8 34359.7 34359.7 34359.7 write
| 1225.0s 0 315.2 177.1 26843.5 34359.7 36507.2 36507.2 write
| 1226.0s 0 322.9 177.3 26843.5 36507.2 36507.2 36507.2 write
| 1227.0s 0 344.1 177.4 26843.5 36507.2 38654.7 38654.7 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 65c6377f4f486d537ef7dfc59f8f9c78f1a6018a:
| 1414.0s 0 170.2 179.7 18253.6 23622.3 24696.1 24696.1 write
| 1415.0s 0 304.8 179.8 19327.4 23622.3 23622.3 66572.0 write
| 1416.0s 0 282.1 179.9 18253.6 23622.3 66572.0 73014.4 write
| 1417.0s 0 241.9 179.9 18253.6 21474.8 22548.6 40802.2 write
| 1418.0s 0 209.1 180.0 16643.0 19327.4 66572.0 73014.4 write
| 1419.0s 0 212.0 180.0 14495.5 18253.6 18253.6 18253.6 write
| 1420.0s 0 199.0 180.0 15032.4 18253.6 19327.4 19327.4 write
| _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
| 1421.0s 0 219.1 180.0 16106.1 17179.9 20401.1 20401.1 write
| 1422.0s 0 225.0 180.1 16643.0 20401.1 21474.8 21474.8 write
| 1423.0s 0 124.1 180.0 18253.6 21474.8 21474.8 21474.8 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ e3be114fc6fe8eda07aafbc7ee43cc452456f270:
| 1437.0s 0 347.7 174.7 16106.1 26843.5 27917.3 27917.3 write
| 1438.0s 0 325.8 174.8 17179.9 26843.5 27917.3 27917.3 write
| 1439.0s 0 321.2 174.9 17179.9 26843.5 27917.3 27917.3 write
| 1440.0s 0 340.1 175.0 17179.9 25769.8 27917.3 27917.3 write
| _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
| 1441.0s 0 112.1 174.9 18253.6 25769.8 27917.3 62277.0 write
| 1442.0s 0 187.0 174.9 19327.4 25769.8 26843.5 26843.5 write
| 1443.0s 0 300.7 175.0 19327.4 26843.5 27917.3 27917.3 write
| 1444.0s 0 245.2 175.1 19327.4 26843.5 26843.5 60129.5 write
| 1445.0s 0 257.9 175.1 19327.4 25769.8 62277.0 73014.4 write
| 1446.0s 0 231.1 175.2 19327.4 24696.1 24696.1 25769.8 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ bbd8658d09681135c30e07650798768ca0063f09:
| 1069.0s 0 9.0 180.3 28991.0 28991.0 28991.0 28991.0 write
| 1070.0s 0 8.0 180.2 30064.8 30064.8 30064.8 30064.8 write
| 1071.0s 0 9.0 180.0 31138.5 31138.5 31138.5 31138.5 write
| 1072.0s 0 30.0 179.9 31138.5 32212.3 32212.3 32212.3 write
| 1073.0s 0 469.9 180.1 24696.1 32212.3 33286.0 33286.0 write
| 1074.0s 0 287.3 180.2 25769.8 33286.0 34359.7 34359.7 write
| 1075.0s 0 320.1 180.4 25769.8 34359.7 36507.2 36507.2 write
| 1076.0s 0 306.9 180.5 25769.8 36507.2 36507.2 36507.2 write
| 1077.0s 0 350.6 180.6 26843.5 36507.2 38654.7 38654.7 write
| 1078.0s 0 335.3 180.8 27917.3 38654.7 38654.7 38654.7 write
| 1079.0s 0 151.1 180.8 27917.3 38654.7 38654.7 38654.7 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ d6b99e92bf55b6f4a0d79800d67924e04d0b2a6d:
| 708.0s 0 86.9 181.9 40802.2 40802.2 40802.2 40802.2 write
| 709.0s 0 44.0 181.7 42949.7 42949.7 42949.7 42949.7 write
| 710.0s 0 172.0 181.7 21474.8 42949.7 42949.7 42949.7 write
| 711.0s 0 496.3 182.2 25769.8 42949.7 42949.7 45097.2 write
| 712.0s 0 366.1 182.4 24696.1 42949.7 45097.2 45097.2 write
| 713.0s 0 234.3 182.5 24696.1 45097.2 45097.2 47244.6 write
| 714.0s 0 358.7 182.8 24696.1 45097.2 45097.2 47244.6 write
| 715.0s 0 362.1 183.0 26843.5 47244.6 47244.6 47244.6 write
| 716.0s 0 324.1 183.2 26843.5 47244.6 49392.1 49392.1 write
| 717.0s 0 397.8 183.5 24696.1 47244.6 49392.1 51539.6 write
| 718.0s 0 47.0 183.3 24696.1 42949.7 45097.2 45097.2 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 5ad21e3896ee809e9c3ebc28bb22166f1275acca:
| 1488.0s 0 8.0 168.8 26843.5 26843.5 26843.5 26843.5 write
| 1489.0s 0 16.0 168.7 27917.3 27917.3 27917.3 27917.3 write
| 1490.0s 0 22.0 168.6 28991.0 28991.0 28991.0 28991.0 write
| 1491.0s 0 27.0 168.5 30064.8 30064.8 30064.8 30064.8 write
| 1492.0s 0 2.0 168.4 30064.8 30064.8 30064.8 30064.8 write
| 1493.0s 0 179.9 168.4 24696.1 31138.5 32212.3 32212.3 write
| 1494.0s 0 523.2 168.7 24696.1 32212.3 33286.0 33286.0 write
| 1495.0s 0 329.3 168.8 23622.3 33286.0 34359.7 34359.7 write
| 1496.0s 0 304.1 168.9 24696.1 34359.7 34359.7 36507.2 write
| 1497.0s 0 288.0 168.9 25769.8 36507.2 36507.2 36507.2 write
| 1498.0s 0 161.1 168.9 26843.5 36507.2 36507.2 36507.2 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 4b41789120e019ab015e6dbb924df763897ebadb:
| 1536.0s 0 246.1 173.8 17179.9 18253.6 18253.6 18253.6 write
| 1537.0s 0 202.0 173.9 18253.6 19327.4 19327.4 19327.4 write
| 1538.0s 0 248.0 173.9 18253.6 19327.4 19327.4 19327.4 write
| 1539.0s 0 218.0 173.9 18253.6 19327.4 19327.4 19327.4 write
| 1540.0s 0 243.0 174.0 18253.6 18253.6 19327.4 19327.4 write
| _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
| 1541.0s 0 184.0 174.0 18253.6 18253.6 19327.4 19327.4 write
| 1542.0s 0 111.1 173.9 18253.6 19327.4 19327.4 19327.4 write
| 1543.0s 0 43.0 173.9 19327.4 19327.4 19327.4 19327.4 write
| 1544.0s 0 21.0 173.8 20401.1 20401.1 20401.1 20401.1 write
| 1545.0s 0 29.0 173.7 21474.8 21474.8 21474.8 21474.8 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 912964e02ddd951c77d4f71981ae18b3894e9084:
| 1496.0s 0 270.0 171.9 34359.7 45097.2 47244.6 47244.6 write
| 1497.0s 0 300.2 172.0 34359.7 45097.2 47244.6 47244.6 write
| 1498.0s 0 268.9 172.0 36507.2 47244.6 47244.6 49392.1 write
| 1499.0s 0 347.8 172.1 36507.2 47244.6 49392.1 49392.1 write
| 1500.0s 0 354.1 172.3 36507.2 49392.1 49392.1 49392.1 write
| _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
| 1501.0s 0 369.8 172.4 36507.2 49392.1 51539.6 51539.6 write
| 1502.0s 0 304.2 172.5 30064.8 51539.6 51539.6 51539.6 write
| 1503.0s 0 128.0 172.5 16106.1 51539.6 51539.6 53687.1 write
| 1504.0s 0 301.9 172.5 27917.3 51539.6 53687.1 53687.1 write
| 1505.0s 0 197.2 172.6 11811.2 51539.6 53687.1 55834.6 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ da01e4c0545f191a0573e1d097ff0366769e0d6b:
| 726.0s 0 23.0 192.8 25769.8 25769.8 25769.8 25769.8 write
| 727.0s 0 0.0 192.5 0.0 0.0 0.0 0.0 write
| 728.0s 0 11.0 192.2 27917.3 27917.3 27917.3 27917.3 write
| 729.0s 0 8.0 192.0 28991.0 28991.0 28991.0 28991.0 write
| 730.0s 0 13.0 191.7 30064.8 30064.8 30064.8 30064.8 write
| 731.0s 0 17.0 191.5 31138.5 31138.5 31138.5 31138.5 write
| 732.0s 0 18.0 191.3 31138.5 32212.3 32212.3 32212.3 write
| 733.0s 0 17.0 191.0 32212.3 32212.3 32212.3 32212.3 write
| 734.0s 0 18.0 190.8 33286.0 33286.0 33286.0 33286.0 write
| 735.0s 0 18.0 190.6 34359.7 34359.7 34359.7 34359.7 write
| 736.0s 0 17.0 190.3 36507.2 36507.2 36507.2 36507.2 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ e1068d77afbd39b162978281c9da7cbea49c1c3a:
| 1145.0s 0 181.9 177.8 18253.6 18253.6 18253.6 18253.6 write
| 1146.0s 0 98.1 177.8 18253.6 18253.6 18253.6 18253.6 write
| 1147.0s 0 27.0 177.6 19327.4 19327.4 19327.4 19327.4 write
| 1148.0s 0 30.0 177.5 19327.4 19327.4 19327.4 19327.4 write
| 1149.0s 0 15.0 177.4 20401.1 20401.1 20401.1 20401.1 write
| 1150.0s 0 14.0 177.2 21474.8 21474.8 21474.8 21474.8 write
| 1151.0s 0 18.0 177.1 22548.6 22548.6 22548.6 22548.6 write
| 1152.0s 0 8.0 176.9 23622.3 23622.3 23622.3 23622.3 write
| 1153.0s 0 14.0 176.8 24696.1 24696.1 24696.1 24696.1 write
| 1154.0s 0 1.0 176.6 25769.8 25769.8 25769.8 25769.8 write
| 1155.0s 0 6.0 176.5 26843.5 26843.5 26843.5 26843.5 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ c4c5ca2fdd5a641433a85a28d4dfd3bd4443015d:
| 1384.0s 0 41.0 170.6 49392.1 49392.1 49392.1 49392.1 write
| 1385.0s 0 67.0 170.5 51539.6 51539.6 51539.6 51539.6 write
| 1386.0s 0 37.0 170.4 51539.6 51539.6 51539.6 51539.6 write
| 1387.0s 0 42.0 170.4 51539.6 51539.6 53687.1 53687.1 write
| 1388.0s 0 366.6 170.5 40802.2 51539.6 53687.1 53687.1 write
| 1389.0s 0 255.1 170.6 40802.2 53687.1 53687.1 53687.1 write
| 1390.0s 0 330.7 170.7 40802.2 53687.1 55834.6 55834.6 write
| 1391.0s 0 347.1 170.8 40802.2 55834.6 55834.6 55834.6 write
| 1392.0s 0 346.9 170.9 40802.2 55834.6 55834.6 57982.1 write
| 1393.0s 0 345.1 171.1 38654.7 55834.6 57982.1 57982.1 write
| 1394.0s 0 60.1 171.0 31138.5 57982.1 57982.1 57982.1 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ e5d1c374c31dc0e80a596c570da8dc45d73f80b8:
| 935.0s 0 23.0 183.8 28991.0 28991.0 28991.0 28991.0 write
| 936.0s 0 7.0 183.6 30064.8 30064.8 30064.8 30064.8 write
| 937.0s 0 12.0 183.4 31138.5 31138.5 31138.5 31138.5 write
| 938.0s 0 42.0 183.3 27917.3 32212.3 32212.3 32212.3 write
| 939.0s 0 634.1 183.7 24696.1 32212.3 33286.0 33286.0 write
| 940.0s 0 307.5 183.9 24696.1 33286.0 34359.7 34359.7 write
| _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
| 941.0s 0 374.8 184.1 25769.8 34359.7 34359.7 36507.2 write
| 942.0s 0 327.3 184.2 25769.8 36507.2 36507.2 36507.2 write
| 943.0s 0 319.8 184.4 25769.8 36507.2 36507.2 38654.7 write
| 944.0s 0 330.1 184.5 25769.8 36507.2 38654.7 38654.7 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 8548987813ff9e1b8a9878023d3abfc6911c16db:
| 1383.0s 0 40.0 172.0 38654.7 38654.7 38654.7 38654.7 write
| 1384.0s 0 113.0 172.0 38654.7 38654.7 38654.7 38654.7 write
| 1385.0s 0 81.0 171.9 34359.7 38654.7 40802.2 40802.2 write
| 1386.0s 0 690.8 172.3 20401.1 38654.7 40802.2 40802.2 write
| 1387.0s 0 324.6 172.4 23622.3 40802.2 40802.2 42949.7 write
| 1388.0s 0 308.0 172.5 21474.8 40802.2 42949.7 42949.7 write
| 1389.0s 0 325.8 172.6 20401.1 42949.7 42949.7 42949.7 write
| 1390.0s 0 279.1 172.7 20401.1 42949.7 45097.2 45097.2 write
| 1391.0s 0 282.9 172.8 18253.6 42949.7 45097.2 45097.2 write
| 1392.0s 0 387.0 172.9 16643.0 45097.2 45097.2 47244.6 write
| 1393.0s 0 289.9 173.0 15569.3 45097.2 47244.6 47244.6 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| main/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| main/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| main/pkg/cmd/roachtest/test_runner.go:780
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| main/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| main/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| GOROOT/src/runtime/proc.go:6498
| runtime.main
| GOROOT/src/runtime/proc.go:238
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 83e2df701688745fe7e7d8a0d0e5e7a4ba8633c8:
| 1278.0s 0 0.0 173.8 0.0 0.0 0.0 0.0 write
| 1279.0s 0 0.0 173.6 0.0 0.0 0.0 0.0 write
| 1280.0s 0 0.0 173.5 0.0 0.0 0.0 0.0 write
| _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
| 1281.0s 0 73.0 173.4 28991.0 38654.7 38654.7 38654.7 write
| 1282.0s 0 621.6 173.8 28991.0 38654.7 38654.7 38654.7 write
| 1283.0s 0 307.5 173.9 30064.8 38654.7 40802.2 40802.2 write
| 1284.0s 0 331.0 174.0 30064.8 40802.2 40802.2 40802.2 write
| 1285.0s 0 372.1 174.1 30064.8 40802.2 42949.7 42949.7 write
| 1286.0s 0 377.9 174.3 30064.8 42949.7 42949.7 42949.7 write
| 1287.0s 0 294.0 174.4 32212.3 42949.7 42949.7 45097.2 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ bb85ddc4fc0ad8761d7aba61315da22b433022e3:
| 1036.0s 0 312.4 178.4 20401.1 55834.6 57982.1 57982.1 write
| 1037.0s 0 144.0 178.4 23622.3 57982.1 57982.1 60129.5 write
| 1038.0s 0 389.6 178.6 21474.8 57982.1 60129.5 60129.5 write
| 1039.0s 0 62.1 178.5 28991.0 60129.5 60129.5 62277.0 write
| 1040.0s 0 436.4 178.7 15032.4 60129.5 62277.0 62277.0 write
| _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
| 1041.0s 0 475.9 179.0 13421.8 60129.5 62277.0 64424.5 write
| 1042.0s 0 302.1 179.1 14495.5 62277.0 64424.5 64424.5 write
| 1043.0s 0 172.2 179.1 13958.6 57982.1 64424.5 64424.5 write
| 1044.0s 0 8.0 178.9 11811.2 51539.6 51539.6 51539.6 write
| 1045.0s 0 479.3 179.2 15569.3 62277.0 66572.0 66572.0 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 8548987813ff9e1b8a9878023d3abfc6911c16db:
| _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
| 1581.0s 0 313.0 170.1 13421.8 55834.6 57982.1 57982.1 write
| 1582.0s 0 445.5 170.3 13421.8 57982.1 60129.5 60129.5 write
| 1583.0s 0 409.4 170.5 13421.8 57982.1 60129.5 60129.5 write
| 1584.0s 0 79.1 170.4 16643.0 60129.5 62277.0 62277.0 write
| 1585.0s 0 74.1 170.3 16106.1 60129.5 62277.0 62277.0 write
| 1586.0s 0 402.3 170.5 16106.1 62277.0 64424.5 64424.5 write
| 1587.0s 0 362.9 170.6 16643.0 62277.0 64424.5 64424.5 write
| 1588.0s 0 347.0 170.7 16643.0 60129.5 64424.5 66572.0 write
| 1589.0s 0 275.1 170.8 14495.5 21474.8 27917.3 36507.2 write
| 1590.0s 0 100.1 170.7 15032.4 22548.6 49392.1 49392.1 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| main/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| main/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| main/pkg/cmd/roachtest/test_runner.go:780
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| main/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| main/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| GOROOT/src/runtime/proc.go:6498
| runtime.main
| GOROOT/src/runtime/proc.go:238
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 16859b99637d1adedb915a67998749dbfa4c0663:
| 1384.0s 0 0.0 174.9 0.0 0.0 0.0 0.0 write
| 1385.0s 0 0.0 174.8 0.0 0.0 0.0 0.0 write
| 1386.0s 0 3.0 174.6 32212.3 36507.2 36507.2 36507.2 write
| 1387.0s 0 450.4 174.8 32212.3 38654.7 40802.2 40802.2 write
| 1388.0s 0 350.0 175.0 32212.3 38654.7 40802.2 40802.2 write
| 1389.0s 0 283.9 175.0 32212.3 40802.2 42949.7 42949.7 write
| 1390.0s 0 325.1 175.1 33286.0 40802.2 42949.7 42949.7 write
| 1391.0s 0 178.1 175.1 33286.0 42949.7 42949.7 42949.7 write
| 1392.0s 0 308.1 175.2 33286.0 42949.7 45097.2 45097.2 write
| 1393.0s 0 350.7 175.4 33286.0 45097.2 45097.2 47244.6 write
| 1394.0s 0 185.2 175.4 34359.7 45097.2 47244.6 47244.6 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 8548987813ff9e1b8a9878023d3abfc6911c16db:
| 1141.0s 0 18.0 177.0 24696.1 24696.1 24696.1 24696.1 write
| 1142.0s 0 28.0 176.9 25769.8 25769.8 25769.8 25769.8 write
| 1143.0s 0 0.0 176.7 0.0 0.0 0.0 0.0 write
| 1144.0s 0 1.0 176.6 27917.3 27917.3 27917.3 27917.3 write
| 1145.0s 0 0.0 176.4 0.0 0.0 0.0 0.0 write
| 1146.0s 0 0.0 176.3 0.0 0.0 0.0 0.0 write
| 1147.0s 0 327.5 176.4 22548.6 30064.8 30064.8 31138.5 write
| 1148.0s 0 409.5 176.6 22548.6 31138.5 32212.3 32212.3 write
| 1149.0s 0 338.2 176.7 22548.6 31138.5 33286.0 33286.0 write
| 1150.0s 0 310.1 176.8 23622.3 33286.0 33286.0 34359.7 write
| 1151.0s 0 113.1 176.8 24696.1 33286.0 34359.7 34359.7 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| main/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| main/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| main/pkg/cmd/roachtest/test_runner.go:780
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| main/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| main/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| GOROOT/src/runtime/proc.go:6498
| runtime.main
| GOROOT/src/runtime/proc.go:238
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ f20c1a8d824f0d00f18c0d58ed8b4bc4955e0969:
| 1059.0s 0 38.0 188.7 23622.3 23622.3 23622.3 23622.3 write
| 1060.0s 0 16.0 188.5 23622.3 23622.3 23622.3 23622.3 write
| _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
| 1061.0s 0 0.0 188.3 0.0 0.0 0.0 0.0 write
| 1062.0s 0 0.0 188.1 0.0 0.0 0.0 0.0 write
| 1063.0s 0 0.0 188.0 0.0 0.0 0.0 0.0 write
| 1064.0s 0 17.0 187.8 27917.3 27917.3 27917.3 27917.3 write
| 1065.0s 0 19.0 187.6 28991.0 28991.0 28991.0 28991.0 write
| 1066.0s 0 24.0 187.5 30064.8 30064.8 30064.8 30064.8 write
| 1067.0s 0 22.0 187.3 30064.8 31138.5 31138.5 31138.5 write
| 1068.0s 0 21.0 187.2 32212.3 32212.3 32212.3 32212.3 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| /usr/local/go/src/runtime/proc.go:6498
| runtime.main
| /usr/local/go/src/runtime/proc.go:238
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 8548987813ff9e1b8a9878023d3abfc6911c16db:
| 1399.0s 0 87.0 171.6 47244.6 47244.6 47244.6 47244.6 write
| 1400.0s 0 147.8 171.6 20401.1 47244.6 47244.6 47244.6 write
| _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
| 1401.0s 0 557.6 171.8 38654.7 47244.6 49392.1 49392.1 write
| 1402.0s 0 326.1 172.0 40802.2 49392.1 49392.1 49392.1 write
| 1403.0s 0 316.1 172.1 40802.2 49392.1 51539.6 51539.6 write
| 1404.0s 0 321.0 172.2 40802.2 51539.6 51539.6 51539.6 write
| 1405.0s 0 372.0 172.3 40802.2 51539.6 53687.1 53687.1 write
| 1406.0s 0 278.9 172.4 42949.7 53687.1 53687.1 53687.1 write
| 1407.0s 0 333.3 172.5 27917.3 53687.1 53687.1 55834.6 write
| 1408.0s 0 272.0 172.6 15569.3 53687.1 55834.6 55834.6 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| main/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| main/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| main/pkg/cmd/roachtest/test_runner.go:780
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| main/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| main/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| GOROOT/src/runtime/proc.go:6498
| runtime.main
| GOROOT/src/runtime/proc.go:238
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 8548987813ff9e1b8a9878023d3abfc6911c16db:
| 1532.0s 0 219.1 173.5 23622.3 40802.2 40802.2 40802.2 write
| 1533.0s 0 210.0 173.5 24696.1 40802.2 42949.7 42949.7 write
| 1534.0s 0 366.8 173.7 22548.6 42949.7 42949.7 45097.2 write
| 1535.0s 0 438.6 173.8 12884.9 42949.7 45097.2 45097.2 write
| 1536.0s 0 394.3 174.0 13421.8 45097.2 45097.2 47244.6 write
| 1537.0s 0 92.1 173.9 13421.8 42949.7 45097.2 45097.2 write
| 1538.0s 0 38.0 173.8 16106.1 45097.2 47244.6 47244.6 write
| 1539.0s 0 444.4 174.0 15032.4 47244.6 49392.1 49392.1 write
| 1540.0s 0 319.9 174.1 13421.8 47244.6 49392.1 49392.1 write
| _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
| 1541.0s 0 291.1 174.2 17179.9 49392.1 51539.6 51539.6 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| main/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| main/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| main/pkg/cmd/roachtest/test_runner.go:779
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| main/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| main/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| GOROOT/src/runtime/proc.go:6498
| runtime.main
| GOROOT/src/runtime/proc.go:238
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 8548987813ff9e1b8a9878023d3abfc6911c16db:
| 1293.0s 0 396.2 174.6 17179.9 66572.0 68719.5 73014.4 write
| 1294.0s 0 175.1 174.6 13958.6 64424.5 68719.5 73014.4 write
| 1295.0s 0 461.5 174.8 14495.5 68719.5 73014.4 73014.4 write
| 1296.0s 0 0.0 174.7 0.0 0.0 0.0 0.0 write
| 1297.0s 0 245.8 174.7 16106.1 68719.5 73014.4 73014.4 write
| 1298.0s 0 341.8 174.8 16643.0 73014.4 73014.4 77309.4 write
| 1299.0s 0 394.0 175.0 14495.5 66572.0 73014.4 77309.4 write
| 1300.0s 0 433.7 175.2 18253.6 73014.4 77309.4 77309.4 write
| _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
| 1301.0s 0 140.3 175.2 18253.6 73014.4 77309.4 77309.4 write
| 1302.0s 0 2.0 175.0 15032.4 21474.8 21474.8 21474.8 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| main/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| main/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| main/pkg/cmd/roachtest/test_runner.go:779
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| main/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| main/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| GOROOT/src/runtime/proc.go:6498
| runtime.main
| GOROOT/src/runtime/proc.go:238
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ a7b1c17c240c935056f87f29dfc1ee75611c371e:
| 1294.0s 0 0.0 175.9 0.0 0.0 0.0 0.0 write
| 1295.0s 0 14.0 175.8 45097.2 45097.2 45097.2 45097.2 write
| 1296.0s 0 20.0 175.7 45097.2 45097.2 45097.2 45097.2 write
| 1297.0s 0 17.0 175.6 45097.2 45097.2 45097.2 45097.2 write
| 1298.0s 0 444.1 175.8 36507.2 47244.6 47244.6 47244.6 write
| 1299.0s 0 416.8 176.0 38654.7 47244.6 47244.6 47244.6 write
| 1300.0s 0 342.1 176.1 38654.7 47244.6 49392.1 49392.1 write
| _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
| 1301.0s 0 328.2 176.2 40802.2 49392.1 49392.1 49392.1 write
| 1302.0s 0 335.8 176.3 38654.7 49392.1 51539.6 51539.6 write
| 1303.0s 0 251.2 176.4 38654.7 51539.6 51539.6 51539.6 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| main/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| main/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| main/pkg/cmd/roachtest/test_runner.go:779
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| main/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| main/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| GOROOT/src/runtime/proc.go:6498
| runtime.main
| GOROOT/src/runtime/proc.go:238
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 84a0e500920e6d0b935d2007de256e42822e94c7:
| 1616.0s 0 0.0 172.6 0.0 0.0 0.0 0.0 write
| 1617.0s 0 0.0 172.5 0.0 0.0 0.0 0.0 write
| 1618.0s 0 0.0 172.4 0.0 0.0 0.0 0.0 write
| 1619.0s 0 0.0 172.3 0.0 0.0 0.0 0.0 write
| 1620.0s 0 2.0 172.1 32212.3 32212.3 32212.3 32212.3 write
| _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
| 1621.0s 0 17.0 172.1 33286.0 33286.0 33286.0 33286.0 write
| 1622.0s 0 17.0 172.0 33286.0 34359.7 34359.7 34359.7 write
| 1623.0s 0 17.0 171.9 34359.7 34359.7 34359.7 34359.7 write
| 1624.0s 0 34.0 171.8 36507.2 36507.2 36507.2 36507.2 write
| 1625.0s 0 108.9 171.7 36507.2 36507.2 36507.2 36507.2 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| main/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| main/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| main/pkg/cmd/roachtest/test_runner.go:779
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| main/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| main/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| GOROOT/src/runtime/proc.go:6498
| runtime.main
| GOROOT/src/runtime/proc.go:238
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 80f1c2ce09389f1d7e97376964d3f2a922405b1b:
| 814.0s 0 0.0 187.9 0.0 0.0 0.0 0.0 write
| 815.0s 0 0.0 187.6 0.0 0.0 0.0 0.0 write
| 816.0s 0 0.0 187.4 0.0 0.0 0.0 0.0 write
| 817.0s 0 22.0 187.2 36507.2 38654.7 38654.7 38654.7 write
| 818.0s 0 367.5 187.4 30064.8 38654.7 38654.7 38654.7 write
| 819.0s 0 301.2 187.6 32212.3 38654.7 40802.2 40802.2 write
| 820.0s 0 372.5 187.8 32212.3 40802.2 40802.2 40802.2 write
| _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
| 821.0s 0 309.3 187.9 33286.0 40802.2 42949.7 42949.7 write
| 822.0s 0 340.9 188.1 33286.0 42949.7 42949.7 42949.7 write
| 823.0s 0 124.1 188.1 31138.5 42949.7 42949.7 42949.7 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| main/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| main/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| main/pkg/cmd/roachtest/test_runner.go:779
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| main/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| main/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| GOROOT/src/runtime/proc.go:6498
| runtime.main
| GOROOT/src/runtime/proc.go:238
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 260be01361ed7264a870e530397ed3529cd014f7:
| 1342.0s 0 19.0 176.9 19327.4 19327.4 19327.4 19327.4 write
| 1343.0s 0 0.0 176.7 0.0 0.0 0.0 0.0 write
| 1344.0s 0 0.0 176.6 0.0 0.0 0.0 0.0 write
| 1345.0s 0 0.0 176.5 0.0 0.0 0.0 0.0 write
| 1346.0s 0 0.0 176.3 0.0 0.0 0.0 0.0 write
| 1347.0s 0 0.0 176.2 0.0 0.0 0.0 0.0 write
| 1348.0s 0 0.0 176.1 0.0 0.0 0.0 0.0 write
| 1349.0s 0 0.0 176.0 0.0 0.0 0.0 0.0 write
| 1350.0s 0 0.0 175.8 0.0 0.0 0.0 0.0 write
| 1351.0s 0 0.0 175.7 0.0 0.0 0.0 0.0 write
| 1352.0s 0 0.0 175.6 0.0 0.0 0.0 0.0 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| main/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| main/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| main/pkg/cmd/roachtest/test_runner.go:779
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| main/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| main/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| GOROOT/src/runtime/proc.go:6498
| runtime.main
| GOROOT/src/runtime/proc.go:238
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ bbb473c8f304ac20fec51ff0a0d04e128383bcf6:
| 985.0s 0 314.1 186.5 20401.1 47244.6 49392.1 49392.1 write
| 986.0s 0 300.2 186.6 19327.4 49392.1 49392.1 51539.6 write
| 987.0s 0 377.8 186.8 19327.4 49392.1 51539.6 51539.6 write
| 988.0s 0 346.2 187.0 19327.4 51539.6 51539.6 53687.1 write
| 989.0s 0 373.8 187.2 14495.5 51539.6 53687.1 53687.1 write
| 990.0s 0 419.8 187.4 16106.1 53687.1 53687.1 55834.6 write
| 991.0s 0 29.0 187.3 9126.8 53687.1 55834.6 55834.6 write
| 992.0s 0 218.8 187.3 17179.9 53687.1 55834.6 55834.6 write
| 993.0s 0 266.8 187.4 19327.4 55834.6 55834.6 57982.1 write
| 994.0s 0 471.7 187.7 13421.8 55834.6 57982.1 57982.1 write
| 995.0s 0 260.2 187.7 13958.6 55834.6 60129.5 60129.5 write
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 2. Command with error:
| ``````
| ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
monitor.go:127,kv.go:154,kv.go:269,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
(1) attached stack trace
-- stack trace:
| main.(*monitorImpl).WaitE
| main/pkg/cmd/roachtest/monitor.go:115
| main.(*monitorImpl).Wait
| main/pkg/cmd/roachtest/monitor.go:123
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
| main.(*testRunner).runTest.func2
| main/pkg/cmd/roachtest/test_runner.go:779
Wraps: (2) monitor failure
Wraps: (3) attached stack trace
-- stack trace:
| main.(*monitorImpl).wait.func2
| main/pkg/cmd/roachtest/monitor.go:171
Wraps: (4) monitor task failed
Wraps: (5) attached stack trace
-- stack trace:
| main.init
| main/pkg/cmd/roachtest/monitor.go:80
| runtime.doInit
| GOROOT/src/runtime/proc.go:6498
| runtime.main
| GOROOT/src/runtime/proc.go:238
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1581
Wraps: (6) t.Fatal() was called
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 05d36621ae805e644d98ef6b35d453ae16255ccc:
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) | See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
Same failure on other branches
- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]
/cc @cockroachdb/kv-triage
This test on roachdash | Improve this report!
Jira issue: CRDB-11123
Epic CRDB-20293