cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.06k stars 3.8k forks source link

roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] #72375

Closed cockroach-teamcity closed 1 year ago

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 05d36621ae805e644d98ef6b35d453ae16255ccc:

          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:142
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:258
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6309
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:208
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1371
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

    cluster.go:1300,context.go:91,cluster.go:1288,test_runner.go:866: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3674517-1635924337-02-n2cpu8 --oneshot --ignore-empty-nodes: exit status 1 2: skipped
        1: dead (exit status 137)
        Error: UNCLASSIFIED_PROBLEM: 1: dead (exit status 137)
        (1) UNCLASSIFIED_PROBLEM
        Wraps: (2) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod.Monitor
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:641
          | main.glob..func14
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:610
          | main.wrap.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:122
          | github.com/spf13/cobra.(*Command).execute
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
          | github.com/spf13/cobra.(*Command).ExecuteC
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
          | github.com/spf13/cobra.(*Command).Execute
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
          | main.main
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1282
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:225
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1371
        Wraps: (3) 1: dead (exit status 137)
        Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) | See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-11123

Epic CRDB-20293

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ f4c16510270c6902e69c3e26e3c125532b416c1b:

          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:142
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:258
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6309
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:208
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1371
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

    cluster.go:1301,context.go:91,cluster.go:1289,test_runner.go:866: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3699635-1636442973-02-n2cpu8 --oneshot --ignore-empty-nodes: exit status 1 2: skipped
        1: dead (exit status 137)
        Error: UNCLASSIFIED_PROBLEM: 1: dead (exit status 137)
        (1) UNCLASSIFIED_PROBLEM
        Wraps: (2) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod.Monitor
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:635
          | main.glob..func14
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:582
          | main.wrap.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:123
          | github.com/spf13/cobra.(*Command).execute
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
          | github.com/spf13/cobra.(*Command).ExecuteC
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
          | github.com/spf13/cobra.(*Command).Execute
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
          | main.main
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1183
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:225
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1371
        Wraps: (3) 1: dead (exit status 137)
        Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) | See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

tbg commented 2 years ago

@sumeerbhola this is the admission control I/O overload test https://github.com/cockroachdb/cockroach/blob/7e5741bfc663e11427eac9f6618804562aec2351/pkg/cmd/roachtest/tests/kv.go#L151-L152

It was unskipped in 7e5741bfc663e11427eac9f6618804562aec2351. Looks like the node is crashing despite admission control being enabled.

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 13669f9c9bd92a4c3b0378a558d7735f122c4e72:

          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:142
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:258
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6309
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:208
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1371
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

    cluster.go:1301,context.go:91,cluster.go:1289,test_runner.go:866: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3727704-1636874044-02-n2cpu8 --oneshot --ignore-empty-nodes: exit status 1 2: skipped
        1: dead (exit status 137)
        Error: UNCLASSIFIED_PROBLEM: 1: dead (exit status 137)
        (1) UNCLASSIFIED_PROBLEM
        Wraps: (2) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod.Monitor
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:596
          | main.glob..func14
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:569
          | main.wrap.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:123
          | github.com/spf13/cobra.(*Command).execute
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
          | github.com/spf13/cobra.(*Command).ExecuteC
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
          | github.com/spf13/cobra.(*Command).Execute
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
          | main.main
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1170
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:225
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1371
        Wraps: (3) 1: dead (exit status 137)
        Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ a5e32e1d18aa07a65eb44063177a6c196623f360:

          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:142
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:258
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

    cluster.go:1308,context.go:91,cluster.go:1296,test_runner.go:866: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3781442-1637651447-02-n2cpu8 --oneshot --ignore-empty-nodes: exit status 1 2: skipped
        1: dead (exit status 137)
        Error: UNCLASSIFIED_PROBLEM: 1: dead (exit status 137)
        (1) UNCLASSIFIED_PROBLEM
        Wraps: (2) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod.Monitor
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:568
          | main.glob..func14
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:518
          | main.wrap.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:69
          | github.com/spf13/cobra.(*Command).execute
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
          | github.com/spf13/cobra.(*Command).ExecuteC
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
          | github.com/spf13/cobra.(*Command).Execute
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
          | main.main
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:909
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:255
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (3) 1: dead (exit status 137)
        Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ a132b15e6c9705a6922c2e476936597c6670e072:

          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:142
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:258
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:778
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

    cluster.go:1308,context.go:91,cluster.go:1296,test_runner.go:867: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3788142-1637737745-01-n2cpu8 --oneshot --ignore-empty-nodes: exit status 1 2: skipped
        1: dead (exit status 137)
        Error: UNCLASSIFIED_PROBLEM: 1: dead (exit status 137)
        (1) UNCLASSIFIED_PROBLEM
        Wraps: (2) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod.Monitor
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:568
          | main.glob..func14
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:518
          | main.wrap.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:69
          | github.com/spf13/cobra.(*Command).execute
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
          | github.com/spf13/cobra.(*Command).ExecuteC
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
          | github.com/spf13/cobra.(*Command).Execute
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
          | main.main
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:909
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:255
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (3) 1: dead (exit status 137)
        Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 40f11fead0a0453969634f8ddb0502c1f78b2806:

          |  1184.0s        0            0.0          178.4      0.0      0.0      0.0      0.0 write
          |  1185.0s        0            0.0          178.2      0.0      0.0      0.0      0.0 write
          |  1186.0s        0            0.0          178.1      0.0      0.0      0.0      0.0 write
          |  1187.0s        0            0.0          177.9      0.0      0.0      0.0      0.0 write
          |  1188.0s        0            0.0          177.8      0.0      0.0      0.0      0.0 write
          |  1189.0s        0            0.0          177.6      0.0      0.0      0.0      0.0 write
          |  1190.0s        0            0.0          177.5      0.0      0.0      0.0      0.0 write
          |  1191.0s        0            8.0          177.3  31138.5  36507.2  36507.2  36507.2 write
          |  1192.0s        0          611.3          177.7  27917.3  36507.2  36507.2  36507.2 write
          |  1193.0s        0          331.2          177.8  26843.5  36507.2  36507.2  38654.7 write
          |  1194.0s        0          324.1          178.0  27917.3  38654.7  38654.7  38654.7 write
          |  1195.0s        0          334.0          178.1  28991.0  38654.7  38654.7  38654.7 write
          |  1196.0s        0          343.0          178.2  30064.8  40802.2  40802.2  40802.2 write
          |  1197.0s        0           14.0          178.1  32212.3  40802.2  40802.2  40802.2 write
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

    monitor.go:128,kv.go:142,kv.go:258,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:142
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:258
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

    cluster.go:1343,context.go:91,cluster.go:1333,test_runner.go:867: dead node detection: 1: dead (exit status 137)
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 2c014c47c1a242f504f6d595bfd79c0edc20b90a:

          |   746.0s        0            0.0          192.1      0.0      0.0      0.0      0.0 write
          |   747.0s        0           36.0          191.9  20401.1  21474.8  21474.8  21474.8 write
          |   748.0s        0          215.0          192.0  21474.8  21474.8  21474.8  22548.6 write
          |   749.0s        0          189.8          192.0  21474.8  23622.3  23622.3  23622.3 write
          |   750.0s        0          197.1          192.0  22548.6  22548.6  23622.3  23622.3 write
          |   751.0s        0           50.0          191.8  21474.8  22548.6  22548.6  22548.6 write
          |   752.0s        0           23.0          191.6  23622.3  23622.3  23622.3  23622.3 write
          |   753.0s        0           53.0          191.4  23622.3  24696.1  24696.1  24696.1 write
          |   754.0s        0           31.0          191.2  24696.1  24696.1  24696.1  24696.1 write
          |   755.0s        0            0.0          190.9      0.0      0.0      0.0      0.0 write
          |   756.0s        0           11.0          190.7  26843.5  26843.5  26843.5  26843.5 write
          |   757.0s        0           20.0          190.4  27917.3  27917.3  27917.3  27917.3 write
          |   758.0s        0           21.0          190.2  27917.3  27917.3  27917.3  27917.3 write
          |   759.0s        0           12.0          190.0  28991.0  28991.0  28991.0  28991.0 write
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

    monitor.go:128,kv.go:142,kv.go:258,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:142
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:258
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

    cluster.go:1343,context.go:91,cluster.go:1333,test_runner.go:867: dead node detection: 1: dead (exit status 137)
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 7097a9015f1a09c7dee4fbdbcc6bde82121f657b:

          |   693.0s        0           82.9          190.5  47244.6  47244.6  47244.6  47244.6 write
          |   694.0s        0           66.0          190.3  47244.6  47244.6  47244.6  47244.6 write
          |   695.0s        0           69.0          190.1  47244.6  47244.6  47244.6  47244.6 write
          |   696.0s        0           69.0          189.9  47244.6  49392.1  49392.1  49392.1 write
          |   697.0s        0           53.0          189.7  40802.2  49392.1  49392.1  49392.1 write
          |   698.0s        0          565.6          190.3  25769.8  47244.6  49392.1  49392.1 write
          |   699.0s        0          307.2          190.4  25769.8  47244.6  49392.1  51539.6 write
          |   700.0s        0          318.3          190.6  23622.3  47244.6  49392.1  51539.6 write
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |   701.0s        0          325.8          190.8  25769.8  49392.1  51539.6  51539.6 write
          |   702.0s        0          322.1          191.0  22548.6  49392.1  51539.6  53687.1 write
          |   703.0s        0          387.7          191.3  22548.6  49392.1  53687.1  53687.1 write
          |   704.0s        0          307.0          191.4  24696.1  53687.1  53687.1  55834.6 write
          |   705.0s        0          240.0          191.5  18253.6  53687.1  55834.6  57982.1 write
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

    monitor.go:128,kv.go:142,kv.go:258,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:142
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:258
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

    cluster.go:1343,context.go:91,cluster.go:1333,test_runner.go:867: dead node detection: 1: dead (exit status 137)
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ a80cfbee826f70988381a0a85c0fe7aba0115484:

          |   769.0s        0           45.0          194.9  20401.1  20401.1  20401.1  20401.1 write
          |   770.0s        0           22.0          194.6  20401.1  20401.1  20401.1  20401.1 write
          |   771.0s        0           31.0          194.4  21474.8  21474.8  21474.8  21474.8 write
          |   772.0s        0           25.0          194.2  22548.6  22548.6  22548.6  22548.6 write
          |   773.0s        0            0.0          194.0      0.0      0.0      0.0      0.0 write
          |   774.0s        0            3.0          193.7  24696.1  24696.1  24696.1  24696.1 write
          |   775.0s        0           14.0          193.5  25769.8  25769.8  25769.8  25769.8 write
          |   776.0s        0            0.0          193.2      0.0      0.0      0.0      0.0 write
          |   777.0s        0            0.0          193.0      0.0      0.0      0.0      0.0 write
          |   778.0s        0            8.0          192.8  27917.3  27917.3  27917.3  27917.3 write
          |   779.0s        0            2.0          192.5  28991.0  28991.0  28991.0  28991.0 write
          |   780.0s        0           17.0          192.3  30064.8  30064.8  30064.8  30064.8 write
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |   781.0s        0            7.0          192.0  31138.5  31138.5  31138.5  31138.5 write
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

    monitor.go:128,kv.go:142,kv.go:258,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:142
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:258
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

    cluster.go:1343,context.go:91,cluster.go:1333,test_runner.go:867: dead node detection: 1: dead (exit status 137)
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 2b8b0f64c63a376784926667f5fbffa4c4fcc60b:

          |  1145.0s        0          194.9          183.4  16643.0  19327.4  19327.4  19327.4 write
          |  1146.0s        0          246.9          183.5  17179.9  19327.4  20401.1  20401.1 write
          |  1147.0s        0          224.1          183.5  18253.6  19327.4  20401.1  20401.1 write
          |  1148.0s        0          213.1          183.6  18253.6  19327.4  19327.4  19327.4 write
          |  1149.0s        0          226.9          183.6  18253.6  19327.4  19327.4  19327.4 write
          |  1150.0s        0          162.0          183.6  18253.6  19327.4  19327.4  19327.4 write
          |  1151.0s        0          157.1          183.5  18253.6  19327.4  19327.4  19327.4 write
          |  1152.0s        0           54.0          183.4  19327.4  19327.4  19327.4  19327.4 write
          |  1153.0s        0           47.0          183.3  20401.1  20401.1  20401.1  20401.1 write
          |  1154.0s        0           30.0          183.2  21474.8  21474.8  21474.8  21474.8 write
          |  1155.0s        0           42.0          183.1  22548.6  22548.6  22548.6  22548.6 write
          |  1156.0s        0           41.0          182.9  22548.6  22548.6  22548.6  22548.6 write
          |  1157.0s        0           44.0          182.8  23622.3  23622.3  23622.3  23622.3 write
          |  1158.0s        0            0.0          182.7      0.0      0.0      0.0      0.0 write
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

    monitor.go:128,kv.go:142,kv.go:258,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:142
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:258
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

    cluster.go:1343,context.go:91,cluster.go:1333,test_runner.go:867: dead node detection: 1: dead (exit status 137)
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 260b7930e717c76bc421ae3f34e9a939fc896b50:

          |  1129.0s        0           95.0          174.0  47244.6  49392.1  49392.1  49392.1 write
          |  1130.0s        0          153.0          174.0  47244.6  49392.1  49392.1  49392.1 write
          |  1131.0s        0          141.0          174.0  45097.2  47244.6  49392.1  49392.1 write
          |  1132.0s        0          561.5          174.3  20401.1  45097.2  47244.6  47244.6 write
          |  1133.0s        0          247.3          174.4  19327.4  45097.2  49392.1  49392.1 write
          |  1134.0s        0          259.0          174.4  19327.4  45097.2  47244.6  47244.6 write
          |  1135.0s        0          283.0          174.5  16106.1  47244.6  49392.1  51539.6 write
          |  1136.0s        0          274.9          174.6  18253.6  47244.6  49392.1  49392.1 write
          |  1137.0s        0          229.1          174.7  12884.9  49392.1  51539.6  53687.1 write
          |  1138.0s        0          241.9          174.7   6442.5  47244.6  51539.6  53687.1 write
          |  1139.0s        0          255.9          174.8   8053.1  51539.6  53687.1  53687.1 write
          |  1140.0s        0          288.1          174.9   8589.9  51539.6  53687.1  55834.6 write
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |  1141.0s        0          232.2          175.0   9126.8  51539.6  53687.1  55834.6 write
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

    monitor.go:128,kv.go:142,kv.go:258,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:142
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:258
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

    cluster.go:1343,context.go:91,cluster.go:1333,test_runner.go:867: dead node detection: 1: dead (exit status 137)
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 835ceca5d25d4a62233ddde4f493dbcf68302f1e:

          |  1051.0s        0          221.9          182.4  18253.6  18253.6  18253.6  18253.6 write
          |  1052.0s        0          223.9          182.4  18253.6  19327.4  19327.4  19327.4 write
          |  1053.0s        0          181.2          182.4  18253.6  19327.4  19327.4  19327.4 write
          |  1054.0s        0          104.0          182.4  19327.4  19327.4  19327.4  19327.4 write
          |  1055.0s        0           42.0          182.2  20401.1  20401.1  20401.1  20401.1 write
          |  1056.0s        0           58.0          182.1  19327.4  20401.1  21474.8  21474.8 write
          |  1057.0s        0           15.0          182.0  20401.1  20401.1  20401.1  20401.1 write
          |  1058.0s        0           13.0          181.8  21474.8  21474.8  21474.8  21474.8 write
          |  1059.0s        0            9.0          181.6  22548.6  22548.6  22548.6  22548.6 write
          |  1060.0s        0           14.0          181.5  23622.3  23622.3  23622.3  23622.3 write
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |  1061.0s        0           16.0          181.3  24696.1  24696.1  24696.1  24696.1 write
          |  1062.0s        0           11.0          181.2  25769.8  25769.8  25769.8  25769.8 write
          |  1063.0s        0            6.0          181.0  26843.5  26843.5  26843.5  26843.5 write
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

    monitor.go:128,kv.go:142,kv.go:258,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:142
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:258
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

    cluster.go:1339,context.go:91,cluster.go:1329,test_runner.go:867: dead node detection: 1: dead (exit status 137)
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

sumeerbhola commented 2 years ago

The node has log statements of the form

E211206 07:58:48.611382 22427 kv/kvserver/replica_write.go:285 ⋮ [n1,s1,r870/1:‹/Table/56/1/-26{99748…-81319…}›] 133853  range unavailable: have been waiting 15.00s for proposing command Put [‹/Table/56/1/-2694689559611899965/0›,‹/Min›), EndTxn(commit) [‹/Table/56/1/-2694689559611899965/0›], [txn: d6b6a4ad], [can-forward-ts].
E211206 07:58:48.611382 22427 kv/kvserver/replica_write.go:285 ⋮ [n1,s1,r870/1:‹/Table/56/1/-26{99748…-81319…}›] 133853 +This range is likely unavailable.

I211206 07:58:48.625194 28782 kv/kvserver/replica_write.go:187 ⋮ [n1,s1,r936/1:‹/Table/56/1/-5{11385…-09542…}›] 133868  slow command Put [‹/Table/56/1/-5112559483361806260/0›,‹/Min›), EndTxn(commit) [‹/Table/56/1/-5112559483361806260/0›], [txn: f24eb0f8], [can-forward-ts] finished after 16.46s with error <nil>

W211206 07:58:52.898875 352 kv/kvserver/spanlatch/manager.go:526 ⋮ [n1,s1,r35/1:‹/Table/{39-40}›] 133942  have been waiting 15s to acquire ‹write› latch ‹/Local/RangeID/35/r/AbortSpan/"b37d0d1f-9258-4b86-8fed-1dcfe466fb97"@0,0›, held by ‹read› latch ‹/Local/RangeID/35/r/AbortSpan/"b37d0d1f-9258-4b86-8fed-1dcfe466fb97"@0,0›

Which should not be caused by overload since admission control is on and should have throttled KV requests before they reached the above code (and corresponding latency measurements). There are also numerous statements indicating raft level slowness.

W211206 07:58:54.030732 262 kv/kvserver/store_raft.go:520 ⋮ [n1,s1,r591/1:‹/Table/56/1/78{59676…-78104…}›] 133978  handle raft ready: 0.8s [applied=5, batches=1, state_assertions=0]

I'll rerun this manually and look at the metrics etc.

sumeerbhola commented 2 years ago

The slowness is correlated with write stalls due to "memtable count limit reached".

I211206 07:58:47.601883 341 util/admission/granter.go:1609 â‹® [-] 133502 IO overload on store 1 (files 866, sub-levels 21): admitted: 1738, added: 534509826, removed (487858690, 307828057), admit: (500.463356, 1119)

I211206 07:59:02.602118 341 util/admission/granter.go:1609 â‹® [-] 134029 IO overload on store 1 (files 1206, sub-levels 28): admitted: 1149, added: 527068422, removed (84938142, 196383099), admit: (214.055871, 666)

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 5cd5be83218475ef0cc3c62a856b8d9d22cf142b:

          |  1322.0s        0          344.0          172.4  16643.0  60129.5  64424.5  64424.5 write
          |  1323.0s        0          316.2          172.5  21474.8  62277.0  64424.5  66572.0 write
          |  1324.0s        0          207.2          172.5  15569.3  62277.0  66572.0  66572.0 write
          |  1325.0s        0          255.9          172.6  13421.8  62277.0  66572.0  68719.5 write
          |  1326.0s        0          425.8          172.8  14495.5  64424.5  68719.5  68719.5 write
          |  1327.0s        0          401.0          172.9  14495.5  64424.5  68719.5  73014.4 write
          |  1328.0s        0          321.0          173.1  12884.9  66572.0  68719.5  73014.4 write
          |  1329.0s        0          121.1          173.0  13421.8  66572.0  73014.4  73014.4 write
          |  1330.0s        0           60.0          172.9  13958.6  57982.1  64424.5  73014.4 write
          |  1331.0s        0          288.8          173.0  15569.3  64424.5  73014.4  73014.4 write
          |  1332.0s        0          419.3          173.2  16643.0  62277.0  73014.4  77309.4 write
          |  1333.0s        0          451.0          173.4  14495.5  57982.1  73014.4  77309.4 write
          |  1334.0s        0          320.5          173.5  11811.2  21474.8  23622.3  31138.5 write
          |  1335.0s        0            0.0          173.4      0.0      0.0      0.0      0.0 write
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

    monitor.go:128,kv.go:151,kv.go:266,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:151
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:266
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

    cluster.go:1339,context.go:91,cluster.go:1329,test_runner.go:867: dead node detection: 1: dead (exit status 137)
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ acf40e97f64a97338cbebd20d7fbe176c4e37c34:

          |  1494.0s        0          297.2          171.0  25769.8  49392.1  49392.1  49392.1 write
          |  1495.0s        0          319.0          171.1  26843.5  49392.1  51539.6  51539.6 write
          |  1496.0s        0          372.0          171.2  26843.5  51539.6  51539.6  51539.6 write
          |  1497.0s        0          369.1          171.3  22548.6  51539.6  53687.1  53687.1 write
          |  1498.0s        0          369.9          171.4  20401.1  53687.1  53687.1  53687.1 write
          |  1499.0s        0          375.0          171.6  18253.6  53687.1  55834.6  55834.6 write
          |  1500.0s        0          342.8          171.7  21474.8  55834.6  55834.6  55834.6 write
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |  1501.0s        0          185.2          171.7  16643.0  53687.1  55834.6  55834.6 write
          |  1502.0s        0          387.7          171.9  16643.0  55834.6  57982.1  57982.1 write
          |  1503.0s        0           14.0          171.7   5368.7  49392.1  55834.6  55834.6 write
          |  1504.0s        0          410.1          171.9  18253.6  57982.1  60129.5  60129.5 write
          |  1505.0s        0          439.0          172.1  12348.0  60129.5  60129.5  62277.0 write
          |  1506.0s        0          225.3          172.1  12884.9  60129.5  62277.0  62277.0 write
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

    monitor.go:128,kv.go:151,kv.go:266,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:151
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:266
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

    cluster.go:1339,context.go:91,cluster.go:1329,test_runner.go:867: dead node detection: 1: dead (exit status 137)
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ dd7fd4e5a371ce4437fce383b9674cba2e212fe6:

          |  1755.0s        0            0.0          166.2      0.0      0.0      0.0      0.0 write
          |  1756.0s        0           12.0          166.1  30064.8  30064.8  30064.8  30064.8 write
          |  1757.0s        0           18.0          166.0  31138.5  31138.5  31138.5  31138.5 write
          |  1758.0s        0            6.0          166.0  32212.3  32212.3  32212.3  32212.3 write
          |  1759.0s        0            6.0          165.9  32212.3  33286.0  33286.0  33286.0 write
          |  1760.0s        0           13.0          165.8  33286.0  33286.0  33286.0  33286.0 write
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |  1761.0s        0           84.0          165.7  25769.8  34359.7  34359.7  34359.7 write
          |  1762.0s        0          639.6          166.0  26843.5  36507.2  36507.2  36507.2 write
          |  1763.0s        0          325.2          166.1  27917.3  36507.2  36507.2  36507.2 write
          |  1764.0s        0          270.0          166.2  26843.5  36507.2  38654.7  38654.7 write
          |  1765.0s        0          337.0          166.2  26843.5  38654.7  38654.7  38654.7 write
          |  1766.0s        0          302.1          166.3  26843.5  38654.7  38654.7  40802.2 write
          |  1767.0s        0          322.0          166.4  27917.3  40802.2  40802.2  40802.2 write
        Wraps: (4) exit status 20
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

    monitor.go:128,kv.go:151,kv.go:266,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:151
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:266
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

    cluster.go:1339,context.go:91,cluster.go:1329,test_runner.go:867: dead node detection: 1: dead (exit status 137)
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 2bf850847554519dc139289615e1a061b6a251de:

          |  1025.0s        0           95.0          179.6  18253.6  18253.6  18253.6  18253.6 write
          |  1026.0s        0           43.0          179.4  19327.4  19327.4  19327.4  19327.4 write
          |  1027.0s        0            2.0          179.3  20401.1  20401.1  20401.1  20401.1 write
          |  1028.0s        0            0.0          179.1      0.0      0.0      0.0      0.0 write
          |  1029.0s        0            0.0          178.9      0.0      0.0      0.0      0.0 write
          |  1030.0s        0            0.0          178.7      0.0      0.0      0.0      0.0 write
          |  1031.0s        0            0.0          178.6      0.0      0.0      0.0      0.0 write
          |  1032.0s        0            0.0          178.4      0.0      0.0      0.0      0.0 write
          |  1033.0s        0            0.0          178.2      0.0      0.0      0.0      0.0 write
          |  1034.0s        0            0.0          178.0      0.0      0.0      0.0      0.0 write
          |  1035.0s        0            0.0          177.9      0.0      0.0      0.0      0.0 write
          |  1036.0s        0            0.0          177.7      0.0      0.0      0.0      0.0 write
          |  1037.0s        0            0.0          177.5      0.0      0.0      0.0      0.0 write
          |  1038.0s        0            0.0          177.4      0.0      0.0      0.0      0.0 write
          |  1039.0s        0            0.0          177.2      0.0      0.0      0.0      0.0 write
          |  1040.0s        0            0.0          177.0      0.0      0.0      0.0      0.0 write
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |  1041.0s        0            0.0          176.8      0.0      0.0      0.0      0.0 write
          |  1042.0s        0           39.0          176.7  28991.0  34359.7  36507.2  36507.2 write
          |  1043.0s        0          392.3          176.9  26843.5  36507.2  36507.2  36507.2 write
          |  1044.0s        0          312.1          177.1  27917.3  36507.2  36507.2  36507.2 write
          |  1045.0s        0          325.9          177.2  27917.3  38654.7  38654.7  38654.7 write
          |  1046.0s        0          335.1          177.3  28991.0  38654.7  38654.7  38654.7 write
        Wraps: (4) secondary error attachment
          | exit status 20
          | (1) exit status 20
          | Error types: (1) *exec.ExitError
        Wraps: (5) context canceled
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

    monitor.go:128,kv.go:151,kv.go:266,test_runner.go:779: monitor failure: unexpected node event: 1: dead (exit status 137)
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:151
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:266
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (2) monitor failure
        Wraps: (3) unexpected node event: 1: dead (exit status 137)
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

    cluster.go:1339,context.go:91,cluster.go:1329,test_runner.go:867: dead node detection: 1: dead (exit status 137)
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 8c1528fff5c132b971428bc7f18bf0ea5eb16799:

          |  1638.0s        0           14.0          167.9  28991.0  28991.0  28991.0  28991.0 write
          |  1639.0s        0           17.0          167.8  30064.8  30064.8  30064.8  30064.8 write
          |  1640.0s        0           15.0          167.7  31138.5  31138.5  31138.5  31138.5 write
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |  1641.0s        0            5.0          167.6  32212.3  32212.3  32212.3  32212.3 write
          |  1642.0s        0           38.0          167.5  26843.5  33286.0  33286.0  33286.0 write
          |  1643.0s        0          466.1          167.7  24696.1  33286.0  34359.7  34359.7 write
          |  1644.0s        0          350.5          167.8  25769.8  34359.7  36507.2  36507.2 write
          |  1645.0s        0          169.1          167.8  24696.1  34359.7  36507.2  36507.2 write
          |  1646.0s        0          321.9          167.9  25769.8  36507.2  36507.2  36507.2 write
          |  1647.0s        0          260.1          167.9  26843.5  36507.2  38654.7  38654.7 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ f3710ae65e327d0795123029db679f8c9f3e610e:

          |   721.0s        0            0.0          189.9      0.0      0.0      0.0      0.0 write
          |   722.0s        0            0.0          189.6      0.0      0.0      0.0      0.0 write
          |   723.0s        0            0.0          189.4      0.0      0.0      0.0      0.0 write
          |   724.0s        0            0.0          189.1      0.0      0.0      0.0      0.0 write
          |   725.0s        0            0.0          188.8      0.0      0.0      0.0      0.0 write
          |   726.0s        0            0.0          188.6      0.0      0.0      0.0      0.0 write
          |   727.0s        0          155.0          188.5  25769.8  33286.0  34359.7  34359.7 write
          |   728.0s        0          400.4          188.8  25769.8  33286.0  34359.7  34359.7 write
          |   729.0s        0          290.2          189.0  25769.8  34359.7  36507.2  36507.2 write
          |   730.0s        0          301.1          189.1  27917.3  36507.2  36507.2  36507.2 write
          |   731.0s        0          307.8          189.3  25769.8  36507.2  36507.2  38654.7 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 01c0c739dd83b3e662d715404f2193d747f4fb9e:

          |  1081.0s        0           19.0          183.7  23622.3  24696.1  24696.1  24696.1 write
          |  1082.0s        0            6.0          183.5  24696.1  24696.1  24696.1  24696.1 write
          |  1083.0s        0           12.0          183.4  25769.8  25769.8  25769.8  25769.8 write
          |  1084.0s        0           14.0          183.2  26843.5  26843.5  26843.5  26843.5 write
          |  1085.0s        0           16.0          183.1  27917.3  27917.3  27917.3  27917.3 write
          |  1086.0s        0           17.0          182.9  28991.0  28991.0  28991.0  28991.0 write
          |  1087.0s        0           31.0          182.8  30064.8  30064.8  30064.8  30064.8 write
          |  1088.0s        0           27.0          182.6  31138.5  31138.5  31138.5  31138.5 write
          |  1089.0s        0           13.0          182.5  31138.5  32212.3  32212.3  32212.3 write
          |  1090.0s        0            7.0          182.3  32212.3  32212.3  32212.3  32212.3 write
          |  1091.0s        0           15.0          182.2  33286.0  33286.0  33286.0  33286.0 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 5ac733bb4927020bc1c52da24b2591742fde8e1f:

          |  1489.0s        0          100.1          167.1  42949.7  42949.7  42949.7  42949.7 write
          |  1490.0s        0           93.9          167.0  42949.7  42949.7  42949.7  42949.7 write
          |  1491.0s        0           14.0          166.9  42949.7  42949.7  42949.7  42949.7 write
          |  1492.0s        0          256.8          167.0  22548.6  42949.7  42949.7  42949.7 write
          |  1493.0s        0          456.5          167.2  23622.3  40802.2  42949.7  45097.2 write
          |  1494.0s        0          285.3          167.3  24696.1  42949.7  45097.2  45097.2 write
          |  1495.0s        0          305.0          167.4  24696.1  38654.7  45097.2  47244.6 write
          |  1496.0s        0          327.0          167.5  23622.3  40802.2  45097.2  47244.6 write
          |  1497.0s        0          306.9          167.6  25769.8  42949.7  47244.6  49392.1 write
          |  1498.0s        0          332.2          167.7  24696.1  42949.7  47244.6  47244.6 write
          |  1499.0s        0           85.0          167.6  24696.1  42949.7  49392.1  49392.1 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 3471dc50db8406f3faa638940492dbe0cab53435:

          |   799.0s        0           18.0          182.3  36507.2  36507.2  36507.2  36507.2 write
          |   800.0s        0           18.0          182.1  36507.2  36507.2  36507.2  36507.2 write
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |   801.0s        0           13.0          181.9  38654.7  38654.7  38654.7  38654.7 write
          |   802.0s        0            0.0          181.7      0.0      0.0      0.0      0.0 write
          |   803.0s        0          399.5          181.9  31138.5  38654.7  40802.2  40802.2 write
          |   804.0s        0          288.9          182.1  31138.5  38654.7  40802.2  40802.2 write
          |   805.0s        0          313.2          182.2  32212.3  40802.2  40802.2  40802.2 write
          |   806.0s        0          287.8          182.4  32212.3  42949.7  42949.7  42949.7 write
          |   807.0s        0          359.9          182.6  32212.3  42949.7  42949.7  45097.2 write
          |   808.0s        0          264.2          182.7  32212.3  42949.7  45097.2  45097.2 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ fa9480a8b661b31a3c5e4c3428595b3e7c320700:

          |  1218.0s        0            6.0          177.2  28991.0  28991.0  28991.0  28991.0 write
          |  1219.0s        0           18.0          177.1  30064.8  30064.8  30064.8  30064.8 write
          |  1220.0s        0           13.0          176.9  31138.5  31138.5  31138.5  31138.5 write
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |  1221.0s        0           13.0          176.8  31138.5  32212.3  32212.3  32212.3 write
          |  1222.0s        0          141.0          176.8  26843.5  32212.3  33286.0  33286.0 write
          |  1223.0s        0          342.5          176.9  24696.1  33286.0  33286.0  33286.0 write
          |  1224.0s        0          324.8          177.0  25769.8  34359.7  34359.7  34359.7 write
          |  1225.0s        0          315.2          177.1  26843.5  34359.7  36507.2  36507.2 write
          |  1226.0s        0          322.9          177.3  26843.5  36507.2  36507.2  36507.2 write
          |  1227.0s        0          344.1          177.4  26843.5  36507.2  38654.7  38654.7 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 65c6377f4f486d537ef7dfc59f8f9c78f1a6018a:

          |  1414.0s        0          170.2          179.7  18253.6  23622.3  24696.1  24696.1 write
          |  1415.0s        0          304.8          179.8  19327.4  23622.3  23622.3  66572.0 write
          |  1416.0s        0          282.1          179.9  18253.6  23622.3  66572.0  73014.4 write
          |  1417.0s        0          241.9          179.9  18253.6  21474.8  22548.6  40802.2 write
          |  1418.0s        0          209.1          180.0  16643.0  19327.4  66572.0  73014.4 write
          |  1419.0s        0          212.0          180.0  14495.5  18253.6  18253.6  18253.6 write
          |  1420.0s        0          199.0          180.0  15032.4  18253.6  19327.4  19327.4 write
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |  1421.0s        0          219.1          180.0  16106.1  17179.9  20401.1  20401.1 write
          |  1422.0s        0          225.0          180.1  16643.0  20401.1  21474.8  21474.8 write
          |  1423.0s        0          124.1          180.0  18253.6  21474.8  21474.8  21474.8 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ e3be114fc6fe8eda07aafbc7ee43cc452456f270:

          |  1437.0s        0          347.7          174.7  16106.1  26843.5  27917.3  27917.3 write
          |  1438.0s        0          325.8          174.8  17179.9  26843.5  27917.3  27917.3 write
          |  1439.0s        0          321.2          174.9  17179.9  26843.5  27917.3  27917.3 write
          |  1440.0s        0          340.1          175.0  17179.9  25769.8  27917.3  27917.3 write
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |  1441.0s        0          112.1          174.9  18253.6  25769.8  27917.3  62277.0 write
          |  1442.0s        0          187.0          174.9  19327.4  25769.8  26843.5  26843.5 write
          |  1443.0s        0          300.7          175.0  19327.4  26843.5  27917.3  27917.3 write
          |  1444.0s        0          245.2          175.1  19327.4  26843.5  26843.5  60129.5 write
          |  1445.0s        0          257.9          175.1  19327.4  25769.8  62277.0  73014.4 write
          |  1446.0s        0          231.1          175.2  19327.4  24696.1  24696.1  25769.8 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ bbd8658d09681135c30e07650798768ca0063f09:

          |  1069.0s        0            9.0          180.3  28991.0  28991.0  28991.0  28991.0 write
          |  1070.0s        0            8.0          180.2  30064.8  30064.8  30064.8  30064.8 write
          |  1071.0s        0            9.0          180.0  31138.5  31138.5  31138.5  31138.5 write
          |  1072.0s        0           30.0          179.9  31138.5  32212.3  32212.3  32212.3 write
          |  1073.0s        0          469.9          180.1  24696.1  32212.3  33286.0  33286.0 write
          |  1074.0s        0          287.3          180.2  25769.8  33286.0  34359.7  34359.7 write
          |  1075.0s        0          320.1          180.4  25769.8  34359.7  36507.2  36507.2 write
          |  1076.0s        0          306.9          180.5  25769.8  36507.2  36507.2  36507.2 write
          |  1077.0s        0          350.6          180.6  26843.5  36507.2  38654.7  38654.7 write
          |  1078.0s        0          335.3          180.8  27917.3  38654.7  38654.7  38654.7 write
          |  1079.0s        0          151.1          180.8  27917.3  38654.7  38654.7  38654.7 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ d6b99e92bf55b6f4a0d79800d67924e04d0b2a6d:

          |   708.0s        0           86.9          181.9  40802.2  40802.2  40802.2  40802.2 write
          |   709.0s        0           44.0          181.7  42949.7  42949.7  42949.7  42949.7 write
          |   710.0s        0          172.0          181.7  21474.8  42949.7  42949.7  42949.7 write
          |   711.0s        0          496.3          182.2  25769.8  42949.7  42949.7  45097.2 write
          |   712.0s        0          366.1          182.4  24696.1  42949.7  45097.2  45097.2 write
          |   713.0s        0          234.3          182.5  24696.1  45097.2  45097.2  47244.6 write
          |   714.0s        0          358.7          182.8  24696.1  45097.2  45097.2  47244.6 write
          |   715.0s        0          362.1          183.0  26843.5  47244.6  47244.6  47244.6 write
          |   716.0s        0          324.1          183.2  26843.5  47244.6  49392.1  49392.1 write
          |   717.0s        0          397.8          183.5  24696.1  47244.6  49392.1  51539.6 write
          |   718.0s        0           47.0          183.3  24696.1  42949.7  45097.2  45097.2 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 5ad21e3896ee809e9c3ebc28bb22166f1275acca:

          |  1488.0s        0            8.0          168.8  26843.5  26843.5  26843.5  26843.5 write
          |  1489.0s        0           16.0          168.7  27917.3  27917.3  27917.3  27917.3 write
          |  1490.0s        0           22.0          168.6  28991.0  28991.0  28991.0  28991.0 write
          |  1491.0s        0           27.0          168.5  30064.8  30064.8  30064.8  30064.8 write
          |  1492.0s        0            2.0          168.4  30064.8  30064.8  30064.8  30064.8 write
          |  1493.0s        0          179.9          168.4  24696.1  31138.5  32212.3  32212.3 write
          |  1494.0s        0          523.2          168.7  24696.1  32212.3  33286.0  33286.0 write
          |  1495.0s        0          329.3          168.8  23622.3  33286.0  34359.7  34359.7 write
          |  1496.0s        0          304.1          168.9  24696.1  34359.7  34359.7  36507.2 write
          |  1497.0s        0          288.0          168.9  25769.8  36507.2  36507.2  36507.2 write
          |  1498.0s        0          161.1          168.9  26843.5  36507.2  36507.2  36507.2 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 4b41789120e019ab015e6dbb924df763897ebadb:

          |  1536.0s        0          246.1          173.8  17179.9  18253.6  18253.6  18253.6 write
          |  1537.0s        0          202.0          173.9  18253.6  19327.4  19327.4  19327.4 write
          |  1538.0s        0          248.0          173.9  18253.6  19327.4  19327.4  19327.4 write
          |  1539.0s        0          218.0          173.9  18253.6  19327.4  19327.4  19327.4 write
          |  1540.0s        0          243.0          174.0  18253.6  18253.6  19327.4  19327.4 write
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |  1541.0s        0          184.0          174.0  18253.6  18253.6  19327.4  19327.4 write
          |  1542.0s        0          111.1          173.9  18253.6  19327.4  19327.4  19327.4 write
          |  1543.0s        0           43.0          173.9  19327.4  19327.4  19327.4  19327.4 write
          |  1544.0s        0           21.0          173.8  20401.1  20401.1  20401.1  20401.1 write
          |  1545.0s        0           29.0          173.7  21474.8  21474.8  21474.8  21474.8 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 912964e02ddd951c77d4f71981ae18b3894e9084:

          |  1496.0s        0          270.0          171.9  34359.7  45097.2  47244.6  47244.6 write
          |  1497.0s        0          300.2          172.0  34359.7  45097.2  47244.6  47244.6 write
          |  1498.0s        0          268.9          172.0  36507.2  47244.6  47244.6  49392.1 write
          |  1499.0s        0          347.8          172.1  36507.2  47244.6  49392.1  49392.1 write
          |  1500.0s        0          354.1          172.3  36507.2  49392.1  49392.1  49392.1 write
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |  1501.0s        0          369.8          172.4  36507.2  49392.1  51539.6  51539.6 write
          |  1502.0s        0          304.2          172.5  30064.8  51539.6  51539.6  51539.6 write
          |  1503.0s        0          128.0          172.5  16106.1  51539.6  51539.6  53687.1 write
          |  1504.0s        0          301.9          172.5  27917.3  51539.6  53687.1  53687.1 write
          |  1505.0s        0          197.2          172.6  11811.2  51539.6  53687.1  55834.6 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ da01e4c0545f191a0573e1d097ff0366769e0d6b:

          |   726.0s        0           23.0          192.8  25769.8  25769.8  25769.8  25769.8 write
          |   727.0s        0            0.0          192.5      0.0      0.0      0.0      0.0 write
          |   728.0s        0           11.0          192.2  27917.3  27917.3  27917.3  27917.3 write
          |   729.0s        0            8.0          192.0  28991.0  28991.0  28991.0  28991.0 write
          |   730.0s        0           13.0          191.7  30064.8  30064.8  30064.8  30064.8 write
          |   731.0s        0           17.0          191.5  31138.5  31138.5  31138.5  31138.5 write
          |   732.0s        0           18.0          191.3  31138.5  32212.3  32212.3  32212.3 write
          |   733.0s        0           17.0          191.0  32212.3  32212.3  32212.3  32212.3 write
          |   734.0s        0           18.0          190.8  33286.0  33286.0  33286.0  33286.0 write
          |   735.0s        0           18.0          190.6  34359.7  34359.7  34359.7  34359.7 write
          |   736.0s        0           17.0          190.3  36507.2  36507.2  36507.2  36507.2 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ e1068d77afbd39b162978281c9da7cbea49c1c3a:

          |  1145.0s        0          181.9          177.8  18253.6  18253.6  18253.6  18253.6 write
          |  1146.0s        0           98.1          177.8  18253.6  18253.6  18253.6  18253.6 write
          |  1147.0s        0           27.0          177.6  19327.4  19327.4  19327.4  19327.4 write
          |  1148.0s        0           30.0          177.5  19327.4  19327.4  19327.4  19327.4 write
          |  1149.0s        0           15.0          177.4  20401.1  20401.1  20401.1  20401.1 write
          |  1150.0s        0           14.0          177.2  21474.8  21474.8  21474.8  21474.8 write
          |  1151.0s        0           18.0          177.1  22548.6  22548.6  22548.6  22548.6 write
          |  1152.0s        0            8.0          176.9  23622.3  23622.3  23622.3  23622.3 write
          |  1153.0s        0           14.0          176.8  24696.1  24696.1  24696.1  24696.1 write
          |  1154.0s        0            1.0          176.6  25769.8  25769.8  25769.8  25769.8 write
          |  1155.0s        0            6.0          176.5  26843.5  26843.5  26843.5  26843.5 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ c4c5ca2fdd5a641433a85a28d4dfd3bd4443015d:

          |  1384.0s        0           41.0          170.6  49392.1  49392.1  49392.1  49392.1 write
          |  1385.0s        0           67.0          170.5  51539.6  51539.6  51539.6  51539.6 write
          |  1386.0s        0           37.0          170.4  51539.6  51539.6  51539.6  51539.6 write
          |  1387.0s        0           42.0          170.4  51539.6  51539.6  53687.1  53687.1 write
          |  1388.0s        0          366.6          170.5  40802.2  51539.6  53687.1  53687.1 write
          |  1389.0s        0          255.1          170.6  40802.2  53687.1  53687.1  53687.1 write
          |  1390.0s        0          330.7          170.7  40802.2  53687.1  55834.6  55834.6 write
          |  1391.0s        0          347.1          170.8  40802.2  55834.6  55834.6  55834.6 write
          |  1392.0s        0          346.9          170.9  40802.2  55834.6  55834.6  57982.1 write
          |  1393.0s        0          345.1          171.1  38654.7  55834.6  57982.1  57982.1 write
          |  1394.0s        0           60.1          171.0  31138.5  57982.1  57982.1  57982.1 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ e5d1c374c31dc0e80a596c570da8dc45d73f80b8:

          |   935.0s        0           23.0          183.8  28991.0  28991.0  28991.0  28991.0 write
          |   936.0s        0            7.0          183.6  30064.8  30064.8  30064.8  30064.8 write
          |   937.0s        0           12.0          183.4  31138.5  31138.5  31138.5  31138.5 write
          |   938.0s        0           42.0          183.3  27917.3  32212.3  32212.3  32212.3 write
          |   939.0s        0          634.1          183.7  24696.1  32212.3  33286.0  33286.0 write
          |   940.0s        0          307.5          183.9  24696.1  33286.0  34359.7  34359.7 write
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |   941.0s        0          374.8          184.1  25769.8  34359.7  34359.7  36507.2 write
          |   942.0s        0          327.3          184.2  25769.8  36507.2  36507.2  36507.2 write
          |   943.0s        0          319.8          184.4  25769.8  36507.2  36507.2  38654.7 write
          |   944.0s        0          330.1          184.5  25769.8  36507.2  38654.7  38654.7 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 8548987813ff9e1b8a9878023d3abfc6911c16db:

          |  1383.0s        0           40.0          172.0  38654.7  38654.7  38654.7  38654.7 write
          |  1384.0s        0          113.0          172.0  38654.7  38654.7  38654.7  38654.7 write
          |  1385.0s        0           81.0          171.9  34359.7  38654.7  40802.2  40802.2 write
          |  1386.0s        0          690.8          172.3  20401.1  38654.7  40802.2  40802.2 write
          |  1387.0s        0          324.6          172.4  23622.3  40802.2  40802.2  42949.7 write
          |  1388.0s        0          308.0          172.5  21474.8  40802.2  42949.7  42949.7 write
          |  1389.0s        0          325.8          172.6  20401.1  42949.7  42949.7  42949.7 write
          |  1390.0s        0          279.1          172.7  20401.1  42949.7  45097.2  45097.2 write
          |  1391.0s        0          282.9          172.8  18253.6  42949.7  45097.2  45097.2 write
          |  1392.0s        0          387.0          172.9  16643.0  45097.2  45097.2  47244.6 write
          |  1393.0s        0          289.9          173.0  15569.3  45097.2  47244.6  47244.6 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:780
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 83e2df701688745fe7e7d8a0d0e5e7a4ba8633c8:

          |  1278.0s        0            0.0          173.8      0.0      0.0      0.0      0.0 write
          |  1279.0s        0            0.0          173.6      0.0      0.0      0.0      0.0 write
          |  1280.0s        0            0.0          173.5      0.0      0.0      0.0      0.0 write
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |  1281.0s        0           73.0          173.4  28991.0  38654.7  38654.7  38654.7 write
          |  1282.0s        0          621.6          173.8  28991.0  38654.7  38654.7  38654.7 write
          |  1283.0s        0          307.5          173.9  30064.8  38654.7  40802.2  40802.2 write
          |  1284.0s        0          331.0          174.0  30064.8  40802.2  40802.2  40802.2 write
          |  1285.0s        0          372.1          174.1  30064.8  40802.2  42949.7  42949.7 write
          |  1286.0s        0          377.9          174.3  30064.8  42949.7  42949.7  42949.7 write
          |  1287.0s        0          294.0          174.4  32212.3  42949.7  42949.7  45097.2 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ bb85ddc4fc0ad8761d7aba61315da22b433022e3:

          |  1036.0s        0          312.4          178.4  20401.1  55834.6  57982.1  57982.1 write
          |  1037.0s        0          144.0          178.4  23622.3  57982.1  57982.1  60129.5 write
          |  1038.0s        0          389.6          178.6  21474.8  57982.1  60129.5  60129.5 write
          |  1039.0s        0           62.1          178.5  28991.0  60129.5  60129.5  62277.0 write
          |  1040.0s        0          436.4          178.7  15032.4  60129.5  62277.0  62277.0 write
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |  1041.0s        0          475.9          179.0  13421.8  60129.5  62277.0  64424.5 write
          |  1042.0s        0          302.1          179.1  14495.5  62277.0  64424.5  64424.5 write
          |  1043.0s        0          172.2          179.1  13958.6  57982.1  64424.5  64424.5 write
          |  1044.0s        0            8.0          178.9  11811.2  51539.6  51539.6  51539.6 write
          |  1045.0s        0          479.3          179.2  15569.3  62277.0  66572.0  66572.0 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 8548987813ff9e1b8a9878023d3abfc6911c16db:

          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |  1581.0s        0          313.0          170.1  13421.8  55834.6  57982.1  57982.1 write
          |  1582.0s        0          445.5          170.3  13421.8  57982.1  60129.5  60129.5 write
          |  1583.0s        0          409.4          170.5  13421.8  57982.1  60129.5  60129.5 write
          |  1584.0s        0           79.1          170.4  16643.0  60129.5  62277.0  62277.0 write
          |  1585.0s        0           74.1          170.3  16106.1  60129.5  62277.0  62277.0 write
          |  1586.0s        0          402.3          170.5  16106.1  62277.0  64424.5  64424.5 write
          |  1587.0s        0          362.9          170.6  16643.0  62277.0  64424.5  64424.5 write
          |  1588.0s        0          347.0          170.7  16643.0  60129.5  64424.5  66572.0 write
          |  1589.0s        0          275.1          170.8  14495.5  21474.8  27917.3  36507.2 write
          |  1590.0s        0          100.1          170.7  15032.4  22548.6  49392.1  49392.1 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:780
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 16859b99637d1adedb915a67998749dbfa4c0663:

          |  1384.0s        0            0.0          174.9      0.0      0.0      0.0      0.0 write
          |  1385.0s        0            0.0          174.8      0.0      0.0      0.0      0.0 write
          |  1386.0s        0            3.0          174.6  32212.3  36507.2  36507.2  36507.2 write
          |  1387.0s        0          450.4          174.8  32212.3  38654.7  40802.2  40802.2 write
          |  1388.0s        0          350.0          175.0  32212.3  38654.7  40802.2  40802.2 write
          |  1389.0s        0          283.9          175.0  32212.3  40802.2  42949.7  42949.7 write
          |  1390.0s        0          325.1          175.1  33286.0  40802.2  42949.7  42949.7 write
          |  1391.0s        0          178.1          175.1  33286.0  42949.7  42949.7  42949.7 write
          |  1392.0s        0          308.1          175.2  33286.0  42949.7  45097.2  45097.2 write
          |  1393.0s        0          350.7          175.4  33286.0  45097.2  45097.2  47244.6 write
          |  1394.0s        0          185.2          175.4  34359.7  45097.2  47244.6  47244.6 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 8548987813ff9e1b8a9878023d3abfc6911c16db:

          |  1141.0s        0           18.0          177.0  24696.1  24696.1  24696.1  24696.1 write
          |  1142.0s        0           28.0          176.9  25769.8  25769.8  25769.8  25769.8 write
          |  1143.0s        0            0.0          176.7      0.0      0.0      0.0      0.0 write
          |  1144.0s        0            1.0          176.6  27917.3  27917.3  27917.3  27917.3 write
          |  1145.0s        0            0.0          176.4      0.0      0.0      0.0      0.0 write
          |  1146.0s        0            0.0          176.3      0.0      0.0      0.0      0.0 write
          |  1147.0s        0          327.5          176.4  22548.6  30064.8  30064.8  31138.5 write
          |  1148.0s        0          409.5          176.6  22548.6  31138.5  32212.3  32212.3 write
          |  1149.0s        0          338.2          176.7  22548.6  31138.5  33286.0  33286.0 write
          |  1150.0s        0          310.1          176.8  23622.3  33286.0  33286.0  34359.7 write
          |  1151.0s        0          113.1          176.8  24696.1  33286.0  34359.7  34359.7 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:780
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ f20c1a8d824f0d00f18c0d58ed8b4bc4955e0969:

          |  1059.0s        0           38.0          188.7  23622.3  23622.3  23622.3  23622.3 write
          |  1060.0s        0           16.0          188.5  23622.3  23622.3  23622.3  23622.3 write
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |  1061.0s        0            0.0          188.3      0.0      0.0      0.0      0.0 write
          |  1062.0s        0            0.0          188.1      0.0      0.0      0.0      0.0 write
          |  1063.0s        0            0.0          188.0      0.0      0.0      0.0      0.0 write
          |  1064.0s        0           17.0          187.8  27917.3  27917.3  27917.3  27917.3 write
          |  1065.0s        0           19.0          187.6  28991.0  28991.0  28991.0  28991.0 write
          |  1066.0s        0           24.0          187.5  30064.8  30064.8  30064.8  30064.8 write
          |  1067.0s        0           22.0          187.3  30064.8  31138.5  31138.5  31138.5 write
          |  1068.0s        0           21.0          187.2  32212.3  32212.3  32212.3  32212.3 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:780
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     /usr/local/go/src/runtime/proc.go:6498
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:238
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 8548987813ff9e1b8a9878023d3abfc6911c16db:

          |  1399.0s        0           87.0          171.6  47244.6  47244.6  47244.6  47244.6 write
          |  1400.0s        0          147.8          171.6  20401.1  47244.6  47244.6  47244.6 write
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |  1401.0s        0          557.6          171.8  38654.7  47244.6  49392.1  49392.1 write
          |  1402.0s        0          326.1          172.0  40802.2  49392.1  49392.1  49392.1 write
          |  1403.0s        0          316.1          172.1  40802.2  49392.1  51539.6  51539.6 write
          |  1404.0s        0          321.0          172.2  40802.2  51539.6  51539.6  51539.6 write
          |  1405.0s        0          372.0          172.3  40802.2  51539.6  53687.1  53687.1 write
          |  1406.0s        0          278.9          172.4  42949.7  53687.1  53687.1  53687.1 write
          |  1407.0s        0          333.3          172.5  27917.3  53687.1  53687.1  55834.6 write
          |  1408.0s        0          272.0          172.6  15569.3  53687.1  55834.6  55834.6 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:780: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:780
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 8548987813ff9e1b8a9878023d3abfc6911c16db:

          |  1532.0s        0          219.1          173.5  23622.3  40802.2  40802.2  40802.2 write
          |  1533.0s        0          210.0          173.5  24696.1  40802.2  42949.7  42949.7 write
          |  1534.0s        0          366.8          173.7  22548.6  42949.7  42949.7  45097.2 write
          |  1535.0s        0          438.6          173.8  12884.9  42949.7  45097.2  45097.2 write
          |  1536.0s        0          394.3          174.0  13421.8  45097.2  45097.2  47244.6 write
          |  1537.0s        0           92.1          173.9  13421.8  42949.7  45097.2  45097.2 write
          |  1538.0s        0           38.0          173.8  16106.1  45097.2  47244.6  47244.6 write
          |  1539.0s        0          444.4          174.0  15032.4  47244.6  49392.1  49392.1 write
          |  1540.0s        0          319.9          174.1  13421.8  47244.6  49392.1  49392.1 write
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |  1541.0s        0          291.1          174.2  17179.9  49392.1  51539.6  51539.6 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:779
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 8548987813ff9e1b8a9878023d3abfc6911c16db:

          |  1293.0s        0          396.2          174.6  17179.9  66572.0  68719.5  73014.4 write
          |  1294.0s        0          175.1          174.6  13958.6  64424.5  68719.5  73014.4 write
          |  1295.0s        0          461.5          174.8  14495.5  68719.5  73014.4  73014.4 write
          |  1296.0s        0            0.0          174.7      0.0      0.0      0.0      0.0 write
          |  1297.0s        0          245.8          174.7  16106.1  68719.5  73014.4  73014.4 write
          |  1298.0s        0          341.8          174.8  16643.0  73014.4  73014.4  77309.4 write
          |  1299.0s        0          394.0          175.0  14495.5  66572.0  73014.4  77309.4 write
          |  1300.0s        0          433.7          175.2  18253.6  73014.4  77309.4  77309.4 write
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |  1301.0s        0          140.3          175.2  18253.6  73014.4  77309.4  77309.4 write
          |  1302.0s        0            2.0          175.0  15032.4  21474.8  21474.8  21474.8 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:779
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ a7b1c17c240c935056f87f29dfc1ee75611c371e:

          |  1294.0s        0            0.0          175.9      0.0      0.0      0.0      0.0 write
          |  1295.0s        0           14.0          175.8  45097.2  45097.2  45097.2  45097.2 write
          |  1296.0s        0           20.0          175.7  45097.2  45097.2  45097.2  45097.2 write
          |  1297.0s        0           17.0          175.6  45097.2  45097.2  45097.2  45097.2 write
          |  1298.0s        0          444.1          175.8  36507.2  47244.6  47244.6  47244.6 write
          |  1299.0s        0          416.8          176.0  38654.7  47244.6  47244.6  47244.6 write
          |  1300.0s        0          342.1          176.1  38654.7  47244.6  49392.1  49392.1 write
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |  1301.0s        0          328.2          176.2  40802.2  49392.1  49392.1  49392.1 write
          |  1302.0s        0          335.8          176.3  38654.7  49392.1  51539.6  51539.6 write
          |  1303.0s        0          251.2          176.4  38654.7  51539.6  51539.6  51539.6 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:779
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 84a0e500920e6d0b935d2007de256e42822e94c7:

          |  1616.0s        0            0.0          172.6      0.0      0.0      0.0      0.0 write
          |  1617.0s        0            0.0          172.5      0.0      0.0      0.0      0.0 write
          |  1618.0s        0            0.0          172.4      0.0      0.0      0.0      0.0 write
          |  1619.0s        0            0.0          172.3      0.0      0.0      0.0      0.0 write
          |  1620.0s        0            2.0          172.1  32212.3  32212.3  32212.3  32212.3 write
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |  1621.0s        0           17.0          172.1  33286.0  33286.0  33286.0  33286.0 write
          |  1622.0s        0           17.0          172.0  33286.0  34359.7  34359.7  34359.7 write
          |  1623.0s        0           17.0          171.9  34359.7  34359.7  34359.7  34359.7 write
          |  1624.0s        0           34.0          171.8  36507.2  36507.2  36507.2  36507.2 write
          |  1625.0s        0          108.9          171.7  36507.2  36507.2  36507.2  36507.2 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:779
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 80f1c2ce09389f1d7e97376964d3f2a922405b1b:

          |   814.0s        0            0.0          187.9      0.0      0.0      0.0      0.0 write
          |   815.0s        0            0.0          187.6      0.0      0.0      0.0      0.0 write
          |   816.0s        0            0.0          187.4      0.0      0.0      0.0      0.0 write
          |   817.0s        0           22.0          187.2  36507.2  38654.7  38654.7  38654.7 write
          |   818.0s        0          367.5          187.4  30064.8  38654.7  38654.7  38654.7 write
          |   819.0s        0          301.2          187.6  32212.3  38654.7  40802.2  40802.2 write
          |   820.0s        0          372.5          187.8  32212.3  40802.2  40802.2  40802.2 write
          | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
          |   821.0s        0          309.3          187.9  33286.0  40802.2  42949.7  42949.7 write
          |   822.0s        0          340.9          188.1  33286.0  42949.7  42949.7  42949.7 write
          |   823.0s        0          124.1          188.1  31138.5  42949.7  42949.7  42949.7 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:779
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ 260be01361ed7264a870e530397ed3529cd014f7:

          |  1342.0s        0           19.0          176.9  19327.4  19327.4  19327.4  19327.4 write
          |  1343.0s        0            0.0          176.7      0.0      0.0      0.0      0.0 write
          |  1344.0s        0            0.0          176.6      0.0      0.0      0.0      0.0 write
          |  1345.0s        0            0.0          176.5      0.0      0.0      0.0      0.0 write
          |  1346.0s        0            0.0          176.3      0.0      0.0      0.0      0.0 write
          |  1347.0s        0            0.0          176.2      0.0      0.0      0.0      0.0 write
          |  1348.0s        0            0.0          176.1      0.0      0.0      0.0      0.0 write
          |  1349.0s        0            0.0          176.0      0.0      0.0      0.0      0.0 write
          |  1350.0s        0            0.0          175.8      0.0      0.0      0.0      0.0 write
          |  1351.0s        0            0.0          175.7      0.0      0.0      0.0      0.0 write
          |  1352.0s        0            0.0          175.6      0.0      0.0      0.0      0.0 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:779
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.kv0/enc=false/nodes=1/size=64kb/conc=4096 failed with artifacts on master @ bbb473c8f304ac20fec51ff0a0d04e128383bcf6:

          |   985.0s        0          314.1          186.5  20401.1  47244.6  49392.1  49392.1 write
          |   986.0s        0          300.2          186.6  19327.4  49392.1  49392.1  51539.6 write
          |   987.0s        0          377.8          186.8  19327.4  49392.1  51539.6  51539.6 write
          |   988.0s        0          346.2          187.0  19327.4  51539.6  51539.6  53687.1 write
          |   989.0s        0          373.8          187.2  14495.5  51539.6  53687.1  53687.1 write
          |   990.0s        0          419.8          187.4  16106.1  53687.1  53687.1  55834.6 write
          |   991.0s        0           29.0          187.3   9126.8  53687.1  55834.6  55834.6 write
          |   992.0s        0          218.8          187.3  17179.9  53687.1  55834.6  55834.6 write
          |   993.0s        0          266.8          187.4  19327.4  55834.6  55834.6  57982.1 write
          |   994.0s        0          471.7          187.7  13421.8  55834.6  57982.1  57982.1 write
          |   995.0s        0          260.2          187.7  13958.6  55834.6  60129.5  60129.5 write
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 2. Command with error:
          | ``````
          | ./workload run kv --init --histograms=perf/stats.json --concurrency=4096 --splits=1000 --duration=30m0s --read-percent=0 --min-block-bytes=65536 --max-block-bytes=65536 {pgurl:1-1}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

    monitor.go:127,kv.go:154,kv.go:269,test_runner.go:779: monitor failure: monitor task failed: t.Fatal() was called
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:154
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKV.func3
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:269
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:779
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | main.init
          |     main/pkg/cmd/roachtest/monitor.go:80
          | runtime.doInit
          |     GOROOT/src/runtime/proc.go:6498
          | runtime.main
          |     GOROOT/src/runtime/proc.go:238
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) t.Fatal() was called
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #70247 roachtest: kv0/enc=false/nodes=1/size=64kb/conc=4096 failed [admission control] [C-test-failure O-roachtest O-robot T-storage branch-release-21.2]

This test on roachdash | Improve this report!