roachtest: schemachange/mixed-versions failed

cockroach-teamcity commented 1 week ago

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar issue without the B-runtime-assertions-enabled label. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.schemachange/mixed-versions failed with artifacts on master @ 83589fb87caa92fb42e83994f1691978f37e4cbb:

(mixedversion.go:720).Run: mixed-version test failure while running step 18 (run "run schemachange workload and validation in mixed version"): full command output in run_064232.570264793_n4_COCKROACHRANDOMSEED9.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=true
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/sql-foundations _{This test on roachdash | Improve this report!

Jira issue: CRDB-42418}

fqazi commented 4 days ago

So, this failure is linked to renames occurring back to back:

 "message": "***UNEXPECTED ERROR; Received an unexpected execution error.: ERROR: function udf_w1_647(composite_w3_74[],composite_w4_114[],composite_w4_58[],composite_w4_721[],composite_w3_74,composite_w4_114,composite_w4_58,composite_w4_721,composite_w3_280[],composite_w4_263[],enum_w0_351[],composite_w3_280,composite_w4_263,enum_w0_351,composite_w4_354[],enum_w0_41[],enum_w4_101[],composite_w4_354,enum_w0_41,enum_w4_101,enum_w0_243[],enum_w1_126[],enum_w1_368[],enum_w0_243,enum_w1_126,enum_w1_368,composite_w0_288[],enum_w1_46[],enum_w2_220[],composite_w0_288,enum_w1_46,\"char\",bit,bool,box2d,char,bytes,string,date,decimal,float,geography,geometry,inet,int,int2,int2vector,int,interval,jsonb,name,oidvector,pg_lsn,refcursor,string,time,timestamp,timestamptz,timetz,tsquery,tsvector) does not exist (SQLSTATE 42883)",

the object we recently created via a rename:

ALTER FUNCTION schema_w3_478.udf_w1_583(public._composite_w3_74, public._composite_w4_114, public._composite_w4_58, public._composite_w4_721, public.composite_w3_74, public.composite_w4_114, public.composite_w4_58, public.composite_w4_721, schema_w1_15._composite_w3_280, schema_w1_15._composite_w4_263, schema_w1_15._enum_w0_351, schema_w1_15.composite_w3_280, schema_w1_15.composite_w4_263, schema_w1_15.enum_w0_351, schema_w3_61._composite_w4_354, schema_w3_61._enum_w0_41, schema_w3_61._enum_w4_101, schema_w3_61.composite_w4_354, schema_w3_61.enum_w0_41, schema_w3_61.enum_w4_101, schema_w2_103._enum_w0_243, schema_w2_103._enum_w1_126, schema_w2_103._enum_w1_368, schema_w2_103.enum_w0_243, schema_w2_103.enum_w1_126, schema_w2_103.enum_w1_368, schema_w0_77._composite_w0_288, schema_w0_77._enum_w1_46, schema_w0_77._enum_w2_220, schema_w0_77.composite_w0_288, schema_w0_77.enum_w1_46, \"char\", BIT, BOOL, BOX2D, BPCHAR, BYTES, STRING, DATE, DECIMAL, FLOAT8, GEOGRAPHY, GEOMETRY, INET, INT8, INT2, INT2VECTOR, INT8, INTERVAL, JSONB, NAME, OIDVECTOR, PG_LSN, REFCURSOR, STRING, TIME, TIMESTAMP, TIMESTAMPTZ, TIMETZ, TSQUERY, TSVECTOR) RENAME TO udf_w1_647

A simple repro here doesn't work

fqazi commented 4 days ago

Maybe the mixed version state was at play here:

  |                                                                  n1          n2          n3          n4
  | released versions                                                v24.2.1     v24.2.1     v24.2.1     master
  | logical binary versions                                          24.2        24.2        24.2        24.2-upgrading-to-24.3-step-016
  | cluster versions before failure (system)                         24.2        24.2        24.2        24.2
  | cluster versions after failure (system)                          24.2        24.2        24.2        24.2
  | cluster versions before failure (mixed-version-tenant-czrnv)     24.2        24.2        24.2        24.2
  | cluster versions after failure (mixed-version-tenant-czrnv)      24.2        24.2        24.2        24.2
Wraps: (2) attached stack trace

fqazi commented 4 days ago

I'm gonna stick this in the triage pile, lets see if we observe it again.

cockroach-teamcity commented 3 days ago

roachtest.schemachange/mixed-versions failed with artifacts on master @ 41084720464c4144f64d9ddcb46508b4d762c4e8:

(mixedversion.go:720).Run: mixed-version test failure while running step 16 (run "run schemachange workload and validation after upgrade has finalized"): full command output in run_083059.671973709_n4_COCKROACHRANDOMSEED3.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

Same failure on other branches

- #131335 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-foundations branch-release-24.1 release-blocker] - #130545 roachtest: schemachange/mixed-versions failed [don't use refcursor in mixed version state] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-23.2] - #130278 roachtest: schemachange/mixed-versions failed [computed column cannot reference a foreign key] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-24.2] - #129535 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-3 T-sql-foundations branch-release-24.2.1-rc] - #129308 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-3 T-sql-foundations branch-release-24.1.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 days ago

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.schemachange/mixed-versions failed with artifacts on master @ 67dc7a1c9bf117046b10513c3277bf7ccf0db975:

(mixedversion.go:737).Run: mixed-version test failure while running step 19 (run "run schemachange workload and validation in mixed version"): full command output in run_064632.864983100_n4_COCKROACHRANDOMSEED8.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=true
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131335 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-foundations branch-release-24.1 release-blocker] - #130545 roachtest: schemachange/mixed-versions failed [don't use refcursor in mixed version state] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-23.2] - #130278 roachtest: schemachange/mixed-versions failed [computed column cannot reference a foreign key] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-24.2] - #129535 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-3 T-sql-foundations branch-release-24.2.1-rc] - #129308 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-3 T-sql-foundations branch-release-24.1.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 day ago

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.schemachange/mixed-versions failed with artifacts on master @ 67dc7a1c9bf117046b10513c3277bf7ccf0db975:

(mixedversion.go:737).Run: mixed-version test failure while running step 11 (run "run schemachange workload and validation in mixed version"): full command output in run_123841.551664357_n4_COCKROACHRANDOMSEED8.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=true
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131335 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-foundations branch-release-24.1 release-blocker] - #130545 roachtest: schemachange/mixed-versions failed [don't use refcursor in mixed version state] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-23.2] - #130278 roachtest: schemachange/mixed-versions failed [computed column cannot reference a foreign key] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-24.2] - #129535 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-3 T-sql-foundations branch-release-24.2.1-rc] - #129308 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-3 T-sql-foundations branch-release-24.1.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 day ago

roachtest.schemachange/mixed-versions failed with artifacts on master @ 5400cb9a70e63bfe1aa2849a566c195ad63130d1:

(mixedversion.go:737).Run: mixed-version test failure while running step 27 (run "run schemachange workload and validation in mixed version"): full command output in run_073614.179943298_n4_COCKROACHRANDOMSEED2.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131335 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-foundations branch-release-24.1 release-blocker] - #130545 roachtest: schemachange/mixed-versions failed [don't use refcursor in mixed version state] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-23.2] - #130278 roachtest: schemachange/mixed-versions failed [computed column cannot reference a foreign key] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-24.2] - #129535 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-3 T-sql-foundations branch-release-24.2.1-rc] - #129308 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-3 T-sql-foundations branch-release-24.1.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 hour ago

roachtest.schemachange/mixed-versions failed with artifacts on master @ b6c13686495bbe9ad476b28033461ef7628e18a8:

(mixedversion.go:737).Run: mixed-version test failure while running step 12 (run "run schemachange workload and validation in mixed version"): full command output in run_061511.239883760_n4_COCKROACHRANDOMSEED4.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131335 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-foundations branch-release-24.1 release-blocker] - #130545 roachtest: schemachange/mixed-versions failed [don't use refcursor in mixed version state] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-23.2] - #130278 roachtest: schemachange/mixed-versions failed [computed column cannot reference a foreign key] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-release-24.2] - #129535 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-3 T-sql-foundations branch-release-24.2.1-rc] - #129308 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-3 T-sql-foundations branch-release-24.1.4-rc]

This test on roachdash | Improve this report!

cockroachdb / cockroach

roachtest: schemachange/mixed-versions failed #131162