cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.88k stars 3.77k forks source link

roachtest: import-cancellation failed #112826

Closed cockroach-teamcity closed 10 months ago

cockroach-teamcity commented 10 months ago

roachtest.import-cancellation failed with artifacts on master @ 07915532f9a7def000127029c667c3a91ce90e47:

(import_cancellation.go:154).func2: full command output in run_200910.232468691_n1_workload-run-tpch-db.log: COMMAND_PROBLEM: exit status 1
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/import-cancellation/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7) See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-12281269/import-cancellation/1697916257934/1697920552941)

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!

Jira issue: CRDB-32620

yuzefovich commented 10 months ago
  |   | ./workload run tpch --db=csv --concurrency=1 --max-ops=44 {pgurl:1-6} --enable-checks=true --histograms=perf/stats.json --histograms-max-latency=8m20s
  |   | ```
  |   | <truncated> ... 1021 20:13:27.963365 130 workload/tpch/tpch.go:474  [-] 4  [q2] returned 100 rows after 218.95 seconds
  |   | I231021 20:15:21.207300 130 workload/tpch/tpch.go:474  [-] 5  [q3] returned 10 rows after 113.24 seconds
  |   | I231021 20:19:46.543699 130 workload/tpch/tpch.go:474  [-] 6  [q4] returned 5 rows after 265.34 seconds
  |   | I231021 20:22:27.963679 130 workload/tpch/tpch.go:474  [-] 7  [q5] returned 5 rows after 161.42 seconds
  |   | I231021 20:23:58.159391 130 workload/tpch/tpch.go:474  [-] 8  [q6] returned 1 rows after 90.20 seconds
  |   | I231021 20:27:07.837244 130 workload/tpch/tpch.go:474  [-] 9  [q7] returned 4 rows after 189.68 seconds
  |   | I231021 20:27:26.974098 130 workload/tpch/tpch.go:474  [-] 10  [q8] returned 2 rows after 19.14 seconds
  |   | I231021 20:30:04.681043 130 workload/tpch/tpch.go:474  [-] 11  [q9] returned 175 rows after 157.71 seconds
  |   | I231021 20:32:24.017414 130 workload/tpch/tpch.go:474  [-] 12  [q10] returned 20 rows after 139.34 seconds
  |   | Error: TPCH wrong output [q11] returned wrong number of rows: got 0, expected 1048

This is concerning.

cockroach-teamcity commented 10 months ago

roachtest.import-cancellation failed with artifacts on master @ 4f26bd1b3f90c4f1c9dd530f9692e8125207b4c8:

(import_cancellation.go:154).func2: full command output in run_183431.125551347_n1_workload-run-tpch-db.log: COMMAND_PROBLEM: exit status 1
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/import-cancellation/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7) See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-12292102/import-cancellation/1697996231236/1698001805972)

This test on roachdash | Improve this report!

DrewKimball commented 10 months ago

It was Q11 for this second failure, as well.

yuzefovich commented 10 months ago

Perhaps we simply need to skip this (and maybe others) query - enable-checks has hard-coded expected results for a particular backup, and it's possible that the import we're doing legitimately has different results.

DrewKimball commented 10 months ago

Is the data set initialized using workload init? See: https://github.com/cockroachdb/cockroach/issues/64172#issuecomment-826025901

DrewKimball commented 10 months ago

Is the data set initialized using workload init? See: https://github.com/cockroachdb/cockroach/issues/64172#issuecomment-826025901

Hm, looks like it's imported from a backup instead, which makes sense. But it looks like the test uses scale-factor=100, and I bet the hard-coded checks assume scale-factor-10.

cockroach-teamcity commented 10 months ago

roachtest.import-cancellation failed with artifacts on master @ 4f26bd1b3f90c4f1c9dd530f9692e8125207b4c8:

(import_cancellation.go:154).func2: full command output in run_182734.100536855_n1_workload-run-tpch-db.log: COMMAND_PROBLEM: exit status 1
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/import-cancellation/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7) See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-12302385/import-cancellation/1698082181794/1698087292504)

This test on roachdash | Improve this report!

yuzefovich commented 10 months ago

scale-factor-10

It's hard-coded at Scale Factor 1.

cockroach-teamcity commented 10 months ago

roachtest.import-cancellation failed with artifacts on master @ f3c7285d89afd63b2f7cae762638b86f237c1d72:

(import_cancellation.go:154).func2: full command output in run_192125.167698116_n1_workload-run-tpch-db.log: COMMAND_PROBLEM: exit status 1
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/import-cancellation/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7) See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-12320917/import-cancellation/1698172207028/1698177094322)

This test on roachdash | Improve this report!