roachtest: monitor_failure failed

[cockroach-teamcity]

roachtest.monitor_failure failed with artifacts on release-24.3.0-rc @ 6d0a627cbefef712d05815b90b738e01805a618e:

test kv0/enc=false/nodes=3/cpu=32/size=64kb failed: (cluster.go:2451).Run: context canceled
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu=32/size=64kb/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=aws
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

- #132978 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #132025 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #131734 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #130292 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #126792 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-44295

[cockroach-teamcity]

roachtest.monitor_failure failed with artifacts on release-24.3.0-rc @ 6d0a627cbefef712d05815b90b738e01805a618e:

test kv0/enc=false/nodes=3/cpu=32 failed: (cluster.go:2451).Run: context canceled
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu=32/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=aws
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

- #132978 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #132025 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #131734 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #130292 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #126792 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.monitor_failure failed with artifacts on release-24.3.0-rc @ 6d0a627cbefef712d05815b90b738e01805a618e:

test kv0/enc=false/nodes=3/cpu=32/size=4kb failed: (cluster.go:2451).Run: context canceled
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu=32/size=4kb/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=aws
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

- #132978 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #132025 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #131734 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #130292 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #126792 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.monitor_failure failed with artifacts on release-24.3.0-rc @ 6d0a627cbefef712d05815b90b738e01805a618e:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b failed: (monitor.go:149).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b_us-west-2b_eu-west-1b/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=aws
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=false
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

- #132978 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #132025 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #131734 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #130292 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #126792 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.monitor_failure failed with artifacts on release-24.3.0-rc @ 516de012609eb9463b2df0fdc78958ca0549074e:

test restore/tpce/400GB/aws/nodes=8/cpus=8 failed: (monitor.go:149).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/restore/tpce/400GB/aws/nodes=8/cpus=8/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=aws
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=false
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

- #132978 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #132025 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #131734 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #130292 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #126792 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.monitor_failure failed with artifacts on release-24.3.0-rc @ 516de012609eb9463b2df0fdc78958ca0549074e:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b failed: (monitor.go:149).Wait: monitor failure: context canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b_us-west-2b_eu-west-1b/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=aws
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=false
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

- #132978 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #132025 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #131734 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #130292 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #126792 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.monitor_failure failed with artifacts on release-24.3.0-rc @ c22e14594a68e8d7135d765726aacdad3eb1f6e0:

test kv0/enc=false/nodes=3/cpu=32/mt-shared-process failed: (cluster.go:2451).Run: context canceled
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu=32/mt-shared-process/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=aws
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

- #132978 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #132025 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #131734 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #130292 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #126792 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.monitor_failure failed with artifacts on release-24.3.0-rc @ c22e14594a68e8d7135d765726aacdad3eb1f6e0:

test kv95/enc=false/nodes=3/size=4kb failed: (cluster.go:2451).Run: context canceled
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/size=4kb/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=aws
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

- #132978 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #132025 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #131734 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #130292 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #126792 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.monitor_failure failed with artifacts on release-24.3.0-rc @ c22e14594a68e8d7135d765726aacdad3eb1f6e0:

test restore/tpce/8TB/aws/nodes=10/cpus=8 failed: (monitor.go:149).Wait: monitor failure: pq: query execution canceled
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/restore/tpce/8TB/aws/nodes=10/cpus=8/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=aws
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=false
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

- #132978 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #132025 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #131734 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #130292 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #126792 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.monitor_failure failed with artifacts on release-24.3.0-rc @ 49d902f212d10c8482298d777061ba153d5c16c9:

test kv95/enc=false/nodes=3/batch=16 failed: (cluster.go:2451).Run: context canceled
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/batch=16/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=aws
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

- #132978 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #132025 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #131734 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #130292 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #126792 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.monitor_failure failed with artifacts on release-24.3.0-rc @ 49d902f212d10c8482298d777061ba153d5c16c9:

test kv95/enc=false/nodes=3/size=4kb failed: (cluster.go:2451).Run: context canceled
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/size=4kb/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=aws
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

- #132978 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #132025 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #131734 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #130292 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #126792 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.monitor_failure failed with artifacts on release-24.3.0-rc @ dd28642dbab32aeaf42d647f0ed0a08780e8f8b0:

test ycsb/C/nodes=3 failed: (cluster.go:2451).Run: context canceled
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/ycsb/C/nodes=3/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=aws
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

- #132978 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #132025 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #131734 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #130292 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #126792 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.monitor_failure failed with artifacts on release-24.3.0-rc @ ef2ebe96f3a26fd6d6716e2ec4a0c8c6fe0fb567:

test c2c/tpcc/warehouses=1000/duration=60/cutover=30 failed: (monitor.go:149).Wait: monitor failure: context canceled
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
EOF [owner=test-eng]
test artifacts and logs in: /artifacts/c2c/tpcc/warehouses=1000/duration=60/cutover=30/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=aws
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=false
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

- #132978 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #132025 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #131734 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #130292 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #126792 roachtest: monitor_failure failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!