search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.18k stars 3.82k forks source link

roachtest: multi-region/mixed-version failed #121455

Open cockroach-teamcity opened 7 months ago

cockroach-teamcity commented 7 months ago

roachtest.multi-region/mixed-version failed with artifacts on master @ 2a5e231716c436781f12452d800651f51c6383b7:

(test_runner.go:1185).runTest: test timed out (36h0m0s)
test artifacts and logs in: /artifacts/multi-region/mixed-version/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-37272

renatolabs commented 7 months ago

The test timed out, but it's unfortunately not possible to know if it got stuck or if this was a regular execution and the test was too large:

Failed to publish artifacts: Artifact file 'artifacts.zip' has size 8.5 GB which exceeds maximum allowed size of 3 GB. Maximum artifact size is configured at the Administration -> Global Settings page.

We should find ways to make this test shorter.

cockroach-teamcity commented 4 months ago

roachtest.multi-region/mixed-version failed with artifacts on master @ 047a7ed79756eef53b8b9ab4c9dd9c5a463496c9:

(cluster.go:2417).Run: full command output in run_140450.372345979_n81_tpcc-workload-check-.log: COMMAND_PROBLEM: exit status 127
(mixedversion.go:620).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/multi-region/mixed-version/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.multi-region/mixed-version failed with artifacts on master @ 1893ac1d7188dc87dbb2b379b30172bf83ca4645:

(node_lister.go:68).WorkloadNode: workload node specified but no workload nodes were provisioned by the cluster
(mixedversion.go:646).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/multi-region/mixed-version/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.multi-region/mixed-version failed with artifacts on master @ 27468a22f82fb1e500bf7b35a62ca58d80abac39:

(cluster.go:2473).Run: full command output in run_150237.261300107_n81_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
(mixedversion.go:694).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/multi-region/mixed-version/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.multi-region/mixed-version failed with artifacts on master @ 1993fc04b5116f20f4814d637c7ce87b003687e4:

(mixedversion.go:695).Run: mixed-version test failure while running step 14 (restart node 22 with binary version master): cluster.StopE: timed out after 300s waiting for n22 to drain and shutdown
test artifacts and logs in: /artifacts/multi-region/mixed-version/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.multi-region/mixed-version failed with artifacts on master @ 5cc8013ada42f9ea03eda661a4f178c141f4f24d:

(mixedversion.go:710).Run: mixed-version test failure while running step 3 (start shared-process tenant "mixed-version-tenant-yn2yj"): ~ COCKROACH_INTERNAL_DISABLE_METAMORPHIC_TESTING=true COCKROACH_CONNECT_TIMEOUT=1200 ./cockroach sql --url 'postgres://root@localhost:26257?options=-ccluster%3Dmixed-version-tenant-yn2yj&sslcert=.%2Fcerts%2Fclient.root.crt&sslkey=.%2Fcerts%2Fclient.root.key&sslmode=verify-full&sslrootcert=.%2Fcerts%2Fca.crt' -e "CREATE SCHEDULE IF NOT EXISTS test_only_backup FOR BACKUP INTO 'gs://cockroachdb-backup-testing/roachprod-scheduled-backups/teamcity-16932716-1726621206-57-n81cpu4-geo/mixed-version-tenant-yn2yj/1726650565447818345?AUTH=implicit' RECURRING '*/15 * * * *' FULL BACKUP '@hourly' WITH SCHEDULE OPTIONS first_run = 'now'"
ERROR: service unavailable for target tenant (mixed-version-tenant-yn2yj)
SQLSTATE: 08000
HINT: Double check your "-ccluster=" connection option or your "cluster:" database name prefix.
Failed running "sql": COMMAND_PROBLEM: exit status 1 [owner=test-eng]
test artifacts and logs in: /artifacts/multi-region/mixed-version/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 weeks ago

roachtest.multi-region/mixed-version failed with artifacts on master @ 7495c0390f93478bc0f961285f41cd4ec85a6a76:

(test_runner.go:1316).runTest: test timed out (36h0m0s)
test artifacts and logs in: /artifacts/multi-region/mixed-version/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.multi-region/mixed-version failed with artifacts on master @ 17535c13cfed95db70cd8dfb1ba6a700686f57b1:

(test_runner.go:1316).runTest: test timed out (36h0m0s)
test artifacts and logs in: /artifacts/multi-region/mixed-version/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.multi-region/mixed-version failed with artifacts on master @ e83bc46aa42f2476b4b11b9703b8038c660dc980:

(mixedversion.go:759).Run: mixed-version test failure while running step 436 (wait for all nodes (:1-80) to acknowledge cluster version '24.2' on system tenant): timed out after 1h0m0s: expected n1 to be at cluster version 24.2, but is still at 24.1-upgrading-to-24.2-step-010: pq: query execution canceled
test artifacts and logs in: /artifacts/multi-region/mixed-version/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 day ago

roachtest.multi-region/mixed-version failed with artifacts on master @ cea3ff5562160a3bf2802da052da2aaa40e1ccc1:

(cluster.go:2456).Run: full command output in run_165321.985907466_n81_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
(mixedversion.go:759).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/multi-region/mixed-version/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!