search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.18k stars 3.82k forks source link

roachtest: decommission/mixed-versions failed #135786

Closed cockroach-teamcity closed 4 days ago

cockroach-teamcity commented 5 days ago

roachtest.decommission/mixed-versions failed with artifacts on master @ 8eeb7f2ae3b2cede564b46ca47e2353fd147c061:

(mixedversion.go:759).Run: mixed-version test failure while running step 5 (run "preload data"): failed to import fixtures: full command output in run_064928.744057746_n2_cockroach-workload-f.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/decommission/mixed-versions/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-44706

kvoli commented 5 days ago

Failure log:

run_064928.744057746_n2_cockroach-workload-f: 2024/11/20 06:49:28 cluster.go:2480: > ./cockroach workload fixtures import tpcc --warehouses=100 {pgurl:1}
I241120 06:49:29.482879 1 ccl/workloadccl/fixture.go:315  [-] 1  starting import of 9 tables
Error: importing fixture: importing table customer: pq: parsing parameters --seed=7523670811854905203 --warehouses=100 --families=false --fks=true --repair-order-ids=false: unknown flag: --repair-order-ids
run_064928.744057746_n2_cockroach-workload-f: 2024/11/20 06:49:29 cluster.go:2493: > result: COMMAND_PROBLEM: exit status 1

In the n1 logs:

I241120 06:49:29.906696 4380 1@util/log/event_log.go:32 ⋮ [T1,Vsystem,n1,job=1022410652286779393] 483 ={"Timestamp":1732085369906613440,"EventType":"import","JobID":1022410652286779393,"JobType":"IMPORT","Description":"‹IMPORT INTO tpcc.public.new_order CSV DATA ('workload:///csv/tpcc/new_order?families=false&fks=true&repair-order-ids=false&row-end=225000&row-start=0&seed=7523670811854905203&version=2.2.0&warehouses=100', 'workload:///csv/tpcc/new_order?families=false&fks=true&repair-order-ids=false&row-end=450000&row-start=225000&seed=7523670811854905203&version=2.2.0&warehouses=100', 'workload:///csv/tpcc/new_order?families=false&fks=true&repair-order-ids=false&row-end=675000&row-start=450000&seed=7523670811854905203&version=2.2.0&warehouses=100', 'workload:///csv/tpcc/new_order?families=false&fks=true&repair-order-ids=false&row-end=900000&row-start=675000&seed=7523670811854905203&version=2.2.0&warehouses=100') WITH OPTIONS (\"nullif\" = 'NULL')›","User":"‹roachprod›","DescriptorIDs":[191],"Status":"failed"}
E241120 06:49:29.906778 4380 jobs/registry.go:1555 ⋮ [T1,Vsystem,n1] 484  IMPORT job 1022410652286779393: stepping through state failed with unexpected error: parsing parameters ‹--seed=7523670811854905203 --warehouses=100 --families=false --fks=true --repair-order-ids=false›: ‹unknown flag: --repair-order-ids›
E241120 06:49:29.906778 4380 jobs/registry.go:1555 â‹® [T1,Vsystem,n1] 484 +(1) attached stack trace
E241120 06:49:29.906778 4380 jobs/registry.go:1555 â‹® [T1,Vsystem,n1] 484 +  -- stack trace:
E241120 06:49:29.906778 4380 jobs/registry.go:1555 â‹® [T1,Vsystem,n1] 484 +  | github.com/cockroachdb/cockroach/pkg/sql/importer.(*workloadReader).readFiles
E241120 06:49:29.906778 4380 jobs/registry.go:1555 â‹® [T1,Vsystem,n1] 484 +  |     github.com/cockroachdb/cockroach/pkg/sql/importer/read_import_workload.go:162
E241120 06:49:29.906778 4380 jobs/registry.go:1555 â‹® [T1,Vsystem,n1] 484 +  | github.com/cockroachdb/cockroach/pkg/sql/importer.runImport.func1
E241120 06:49:29.906778 4380 jobs/registry.go:1555 â‹® [T1,Vsystem,n1] 484 +  |     github.com/cockroachdb/cockroach/pkg/sql/importer/read_import_base.go:106
E241120 06:49:29.906778 4380 jobs/registry.go:1555 â‹® [T1,Vsystem,n1] 484 +  | github.com/cockroachdb/cockroach/pkg/sql/importer.runImport.Group.GoCtx.func4
E241120 06:49:29.906778 4380 jobs/registry.go:1555 â‹® [T1,Vsystem,n1] 484 +  |     github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168
E241120 06:49:29.906778 4380 jobs/registry.go:1555 â‹® [T1,Vsystem,n1] 484 +  | golang.org/x/sync/errgroup.(*Group).Go.func1
E241120 06:49:29.906778 4380 jobs/registry.go:1555 â‹® [T1,Vsystem,n1] 484 +  |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
E241120 06:49:29.906778 4380 jobs/registry.go:1555 â‹® [T1,Vsystem,n1] 484 +  | runtime.goexit
E241120 06:49:29.906778 4380 jobs/registry.go:1555 â‹® [T1,Vsystem,n1] 484 +  |     src/runtime/asm_amd64.s:1695
E241120 06:49:29.906778 4380 jobs/registry.go:1555 ⋮ [T1,Vsystem,n1] 484 +Wraps: (2) parsing parameters ‹--seed=7523670811854905203 --warehouses=100 --families=false --fks=true --repair-order-ids=false›
E241120 06:49:29.906778 4380 jobs/registry.go:1555 ⋮ [T1,Vsystem,n1] 484 +Wraps: (3) ‹unknown flag: --repair-order-ids›
E241120 06:49:29.906778 4380 jobs/registry.go:1555 â‹® [T1,Vsystem,n1] 484 +Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString
E241120 06:49:29.915828 4380 jobs/adopt.go:461 ⋮ [T1,Vsystem,n1] 485  job 1022410652286779393: adoption completed with error parsing parameters ‹--seed=7523670811854905203 --warehouses=100 --families=false --fks=true --repair-order-ids=false›: ‹unknown flag: --repair-order-ids›

Error comes from:

unknown flag: --repair-order-ids
kvoli commented 5 days ago

We probably need to exclude the version where this flag is unsupported w/ imports.

cockroach-teamcity commented 4 days ago

roachtest.decommission/mixed-versions failed with artifacts on master @ eb2d2e19eb29d2747d9e267bd0612a69d066adad:

(mixedversion.go:759).Run: mixed-version test failure while running step 4 (run "preload data"): failed to import fixtures: full command output in run_064715.310280551_n1_cockroach-workload-f.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/decommission/mixed-versions/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 days ago

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.decommission/mixed-versions failed with artifacts on master @ eb2d2e19eb29d2747d9e267bd0612a69d066adad:

(mixedversion.go:759).Run: mixed-version test failure while running step 5 (run "preload data"): failed to import fixtures: full command output in run_071404.291330222_n4_cockroach-workload-f.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/decommission/mixed-versions/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!