search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.11k stars 3.81k forks source link

roachtest: schemachange/mixed-versions failed [job-rows-batch-insert error; let's ignore that error] #116737

Closed cockroach-teamcity closed 7 months ago

cockroach-teamcity commented 10 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ a1051b873ef16c1ee1bc996ca711d31ab5804a4b:

(mixedversion.go:539).Run: mixed-version test failure while running step 22 (run "run schemachange workload and validation in mixed version"): full command output in run_070359.530245957_n4_COCKROACHRANDOMSEED8.log: COMMAND_PROBLEM: exit status 134
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7) See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-13213601/schemachange-mixed-versions/1702968812530/1702969936044)

Same failure on other branches

- #116649 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-foundations branch-master] - #115050 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-1 T-sql-foundations branch-release-23.2]

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-34740

cockroach-teamcity commented 10 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ 6780f67db80cc24b1d16a4e652413f667eb103a5:

(mixedversion.go:539).Run: mixed-version test failure while running step 21 (run "run schemachange workload and validation in mixed version"): full command output in run_095140.725498696_n4_COCKROACHRANDOMSEED1.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7) See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-13262692/schemachange-mixed-versions/1703238192794/1703238891682)

Same failure on other branches

- #116649 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-foundations branch-master] - #115050 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-1 T-sql-foundations branch-release-23.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 10 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ 6780f67db80cc24b1d16a4e652413f667eb103a5:

(mixedversion.go:539).Run: mixed-version test failure while running step 21 (run "run schemachange workload and validation in mixed version"): full command output in run_230639.426151012_n4_COCKROACHRANDOMSEED1.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7) See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-13270095/schemachange-mixed-versions/1703285849211/1703286552070)

Same failure on other branches

- #116649 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-foundations branch-master] - #115050 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-1 T-sql-foundations branch-release-23.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 10 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ 6780f67db80cc24b1d16a4e652413f667eb103a5:

(mixedversion.go:539).Run: mixed-version test failure while running step 22 (run "run schemachange workload and validation in mixed version"): full command output in run_070208.972575349_n4_COCKROACHRANDOMSEED8.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7) See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-13289124/schemachange-mixed-versions/1703487058481/1703487876296)

Same failure on other branches

- #116649 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-foundations branch-master] - #115050 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-1 T-sql-foundations branch-release-23.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 10 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ 80ab3499ff6a4fc9d9b8d55008784aeb0615d5b0:

(mixedversion.go:539).Run: mixed-version test failure while running step 21 (run "run schemachange workload and validation in mixed version"): full command output in run_070201.031434827_n4_COCKROACHRANDOMSEED3.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7) See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-13326691/schemachange-mixed-versions/1703832703200/1703833471378)

Same failure on other branches

- #116649 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-foundations branch-master] - #115050 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-1 T-sql-foundations branch-release-23.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 10 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ 80ab3499ff6a4fc9d9b8d55008784aeb0615d5b0:

(mixedversion.go:539).Run: mixed-version test failure while running step 22 (run "run schemachange workload and validation in mixed version"): full command output in run_070132.739757092_n4_COCKROACHRANDOMSEED7.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7) See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-13351231/schemachange-mixed-versions/1704091875383/1704092675482)

Same failure on other branches

- #116649 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-foundations branch-master] - #115050 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-1 T-sql-foundations branch-release-23.2]

This test on roachdash | Improve this report!

annrpom commented 10 months ago 
"message": "***UNEXPECTED COMMIT ERROR; Received an unexpected commit error: ERROR: job-rows-batch-insert: null value in column \"payload\" violates not-null constraint (SQLSTATE 23502)"

This one might be due to V23_1StopWritingPayloadAndProgressToSystemJobs - where we stopped writing the payload column to the system.jobs table. The upgrade path for this roachtest looks like:

 install fixtures for version "v22.2.11" (1)
 start cluster at version "v22.2.11" (2)
 wait for nodes :1-4 to all have the same cluster version (same as binary version of node 1) (3)
 run "set up schemachange workload" (4)
 upgrade cluster from "v22.2.11" to "release-23.1"
    prevent auto-upgrades by setting `preserve_downgrade_option` (5)
    upgrade nodes :1-4 from "v22.2.11" to "release-23.1"
    restart node 1 with binary version release-23.1 (6)
    run "run schemachange workload and validation in mixed version" (7)
    restart node 4 with binary version release-23.1 (8)
    restart node 2 with binary version release-23.1 (9)
    restart node 3 with binary version release-23.1 (10)
    downgrade nodes :1-4 from "release-23.1" to "v22.2.11"
    restart node 4 with binary version v22.2.11 (11)
    restart node 2 with binary version v22.2.11 (12)
    restart node 3 with binary version v22.2.11 (13)
    run "run schemachange workload and validation in mixed version" (14)
    restart node 1 with binary version v22.2.11 (15)
    upgrade nodes :1-4 from "v22.2.11" to "release-23.1"
    restart node 4 with binary version release-23.1 (16)
    run "run schemachange workload and validation in mixed version" (17)
    restart node 1 with binary version release-23.1 (18)
    restart node 3 with binary version release-23.1 (19)
    restart node 2 with binary version release-23.1 (20)
    finalize upgrade by resetting `preserve_downgrade_option` (21)
    run "run schemachange workload and validation in mixed version" (22)
    wait for nodes :1-4 to all have the same cluster version (same as binary version of node 1) (23)
    run "run schemachange workload and validation after upgrade has finalized" (24)

We failed at step 22 - presumably due to the fact that this payload column got wiped during the upgrade to 23.1, was still wiped whenever we downgraded back to 22.2, and just failed the check constraint on the upgrade back to 23.1.

rafiss commented 10 months ago

For that last failure, it looks like @adityamaru suggested here (https://github.com/cockroachdb/cockroach/issues/115747#issuecomment-1855830157) that we should not fix it for 23.1.

annrpom commented 10 months ago

Apologies for the noise - closed since I saw several of these payload errors, but also just saw this one as well; so, keeping open:

Error: ***UNEXPECTED ERROR; Received an unexpected execution error.: ERROR: descriptor ID generator unavailable, migration in progress, retry later (SQLSTATE XXUUU)
cockroach-teamcity commented 10 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ d3d457a45f468c9e7befdf5787f8a99bf46ca56e:

(mixedversion.go:539).Run: mixed-version test failure while running step 22 (run "run schemachange workload and validation in mixed version"): full command output in run_070255.077226141_n4_COCKROACHRANDOMSEED3.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7) See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-13433039/schemachange-mixed-versions/1704610337907/1704611163396)

Same failure on other branches

- #117465 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-foundations branch-release-23.1.14-rc release-blocker] - #116649 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-foundations branch-master] - #115050 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-1 T-sql-foundations branch-release-23.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 10 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ 6c578a116b3a3dd7d5ebbd9352648773339d73b7:

(mixedversion.go:539).Run: mixed-version test failure while running step 22 (run "run schemachange workload and validation in mixed version"): full command output in run_070613.877655101_n4_COCKROACHRANDOMSEED1.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7) See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-13536588/schemachange-mixed-versions/1705215296040/1705216128863)

Same failure on other branches

- #117721 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-foundations branch-release-23.1.14-rc release-blocker] - #116649 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ 6c578a116b3a3dd7d5ebbd9352648773339d73b7:

(mixedversion.go:539).Run: mixed-version test failure while running step 22 (run "run schemachange workload and validation in mixed version"): full command output in run_070152.281877482_n4_COCKROACHRANDOMSEED4.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7) See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-13546548/schemachange-mixed-versions/1705301502804/1705302293537)

Same failure on other branches

- #117721 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-foundations branch-release-23.1.14-rc release-blocker] - #116649 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ 801ed373e467acaad112eb9abad1f7d3b706efcb:

(mixedversion.go:539).Run: mixed-version test failure while running step 22 (run "run schemachange workload and validation in mixed version"): full command output in run_070345.812787231_n4_COCKROACHRANDOMSEED1.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7) See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-13590700/schemachange-mixed-versions/1705560782400/1705561618300)

Same failure on other branches

- #117721 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-23.1.14-rc] - #116649 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ 5dc36f8be732be0b48050340c8f9efefb45e12e8:

(mixedversion.go:539).Run: mixed-version test failure while running step 22 (run "run schemachange workload and validation in mixed version"): full command output in run_070313.923231422_n4_COCKROACHRANDOMSEED6.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_metamorphicBuild=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7) See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-13606294/schemachange-mixed-versions/1705647166464/1705647982200)

Same failure on other branches

- #117721 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-23.1.14-rc] - #116649 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ 0a2b3a178fad81a0e93af53e0535f04aaecb12ce:

(mixedversion.go:561).Run: mixed-version test failure while running step 22 (run "run schemachange workload and validation in mixed version"): full command output in run_070554.343278177_n4_COCKROACHRANDOMSEED4.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #117721 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-23.1.14-rc] - #116649 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-master]

This test on roachdash | Improve this report!

annrpom commented 9 months ago

Recent (I looked at the past 3) failures are related to the payload column

rafiss commented 9 months ago

Were you referring to https://github.com/cockroachdb/cockroach/issues/115747#issuecomment-1855830157 ? Can we update this workload to tolerate that error and ignore it, since it looks like it won't be getting fixed in 23.1?

annrpom commented 9 months ago

Yes & yes I can look into that

cockroach-teamcity commented 9 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ 19ba39bf5c28cd15b89796332c0bbf6d4a7b0f50:

(mixedversion.go:561).Run: mixed-version test failure while running step 11 (run "run schemachange workload and validation in mixed version"): full command output in run_065443.391896809_n4_COCKROACHRANDOMSEED7.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #116649 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ 42a4b16971d1b0227f9ee9ce6fb472f37b8b8b6d:

(mixedversion.go:564).Run: mixed-version test failure while running step 20 (run "run schemachange workload and validation in mixed version"): full command output in run_070208.479383498_n4_COCKROACHRANDOMSEED5.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ 0fea8da5fc656d9108127450c91e5801e43be25f:

(mixedversion.go:564).Run: mixed-version test failure while running step 21 (run "run schemachange workload and validation in mixed version"): full command output in run_070114.140691354_n4_COCKROACHRANDOMSEED8.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ 1ba29d87c4ffe1e8f9c2670f6e31187dc9c9f638:

(mixedversion.go:588).Run: mixed-version test failure while running step 21 (run "run schemachange workload and validation in mixed version"): full command output in run_065928.643454589_n4_COCKROACHRANDOMSEED4.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #118621 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-23.1.15-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ 1ba29d87c4ffe1e8f9c2670f6e31187dc9c9f638:

(mixedversion.go:588).Run: mixed-version test failure while running step 20 (run "run schemachange workload and validation in mixed version"): full command output in run_070046.957899068_n4_COCKROACHRANDOMSEED6.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #118621 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-23.1.15-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ 9452b6e051ed8c0d1ba3fc5efdfe49c6d883f5e7:

(mixedversion.go:588).Run: mixed-version test failure while running step 22 (run "run schemachange workload and validation in mixed version"): full command output in run_070341.754895760_n4_COCKROACHRANDOMSEED2.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #118621 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-23.1.15-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ 9b3c1cabbcbfb491d33b6ea8943c0762a17892b6:

(mixedversion.go:588).Run: mixed-version test failure while running step 22 (run "run schemachange workload and validation in mixed version"): full command output in run_070254.337233011_n4_COCKROACHRANDOMSEED4.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #118621 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-23.1.15-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ 78de5d3123c3ee76fb2eaf37bb3f11f29b74c331:

(mixedversion.go:588).Run: mixed-version test failure while running step 22 (run "run schemachange workload and validation in mixed version"): full command output in run_070329.046512942_n4_COCKROACHRANDOMSEED5.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #118621 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-3 T-sql-foundations branch-release-23.1.15-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ 0c48ab97d8935fa4360a70803c80c1cef71fd958:

(mixedversion.go:594).Run: mixed-version test failure while running step 21 (run "run schemachange workload and validation in mixed version"): full command output in run_070127.468706390_n4_COCKROACHRANDOMSEED8.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #119194 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-1 T-sql-foundations branch-master release-blocker] - #118621 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-3 T-sql-foundations branch-release-23.1.15-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ b102a6dd3f49ef7bb92217d005ad890105f84d07:

(mixedversion.go:594).Run: mixed-version test failure while running step 21 (run "run schemachange workload and validation in mixed version"): full command output in run_070425.709795221_n4_COCKROACHRANDOMSEED2.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #119194 roachtest: schemachange/mixed-versions failed [CRDB and test both hang when leases fail] [C-test-failure O-roachtest O-robot P-1 T-sql-foundations branch-master release-blocker] - #118621 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-3 T-sql-foundations branch-release-23.1.15-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ 333253387f07fcbfc20697df5556224d44ca86c2:

(mixedversion.go:594).Run: mixed-version test failure while running step 21 (run "run schemachange workload and validation in mixed version"): full command output in run_070346.281596495_n4_COCKROACHRANDOMSEED6.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #119194 roachtest: schemachange/mixed-versions failed [CRDB and test both hang when leases fail] [C-test-failure O-roachtest O-robot P-1 T-sql-foundations branch-master release-blocker] - #118621 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-3 T-sql-foundations branch-release-23.1.15-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ 1ae85d2986e88578216a11776be952f63e593e87:

(mixedversion.go:594).Run: mixed-version test failure while running step 21 (run "run schemachange workload and validation in mixed version"): full command output in run_065751.552263917_n4_COCKROACHRANDOMSEED1.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #119194 roachtest: schemachange/mixed-versions failed [CRDB and test both hang when leases fail] [C-test-failure O-roachtest O-robot P-1 T-sql-foundations branch-master release-blocker] - #118621 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-3 T-sql-foundations branch-release-23.1.15-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ 1ae85d2986e88578216a11776be952f63e593e87:

(mixedversion.go:594).Run: mixed-version test failure while running step 20 (run "run schemachange workload and validation in mixed version"): full command output in run_065720.312687420_n4_COCKROACHRANDOMSEED1.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #119194 roachtest: schemachange/mixed-versions failed [CRDB and test both hang when leases fail] [C-test-failure O-roachtest O-robot P-1 T-sql-foundations branch-master release-blocker] - #118621 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-3 T-sql-foundations branch-release-23.1.15-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ 52968eab6105ddcba8ff09df6f579cb24847a2ac:

(mixedversion.go:594).Run: mixed-version test failure while running step 21 (run "run schemachange workload and validation in mixed version"): full command output in run_065948.387261603_n4_COCKROACHRANDOMSEED4.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #119194 roachtest: schemachange/mixed-versions failed [unsupported comparison operator: <geography> = <box2d>] [C-test-failure O-roachtest O-robot P-1 T-sql-foundations branch-master release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ e0fe00bd7e976fd9f33c097bd29b1fb2a64268fe:

(mixedversion.go:594).Run: mixed-version test failure while running step 21 (run "run schemachange workload and validation in mixed version"): full command output in run_060134.865551613_n4_COCKROACHRANDOMSEED5.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #119904 roachtest: schemachange/mixed-versions failed [job-rows-batch-insert error; let's ignore that error] [C-test-failure O-roachtest O-robot P-3 T-sql-foundations branch-release-23.1.17-rc] - #119194 roachtest: schemachange/mixed-versions failed [unsupported comparison operator: <geography> = <box2d>] [C-test-failure O-roachtest O-robot P-1 T-sql-foundations branch-master]

This test on roachdash | Improve this report!