search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.21k stars 3.82k forks source link

workload: ycsb hangs indefinitely #96875

Closed kvoli closed 1 year ago

kvoli commented 1 year ago

Describe the problem

When running ycsb, the workload fails to execute the ops due to a deadlock when acquiring a connection (it seems). This doesn't occur when running the same configuration against 3 nodes, rather than 7.

https://github.com/cockroachdb/cockroach/blob/3e65660091c0870988978032999acda5a675e5e7/pkg/workload/ycsb/ycsb.go#L497-L497

#   0x3a29c9b   github.com/jackc/puddle.(*Pool).Acquire+0x25b               github.com/jackc/puddle/external/com_github_jackc_puddle/pool.go:349
#   0x3a2e327   github.com/jackc/pgx/v4/pgxpool.(*Pool).Acquire+0x47            github.com/jackc/pgx/v4/pgxpool/external/com_github_jackc_pgx_v4/pgxpool/pool.go:414
#   0x458fb24   github.com/cockroachdb/cockroach/pkg/workload/ycsb.(*ycsb).Ops+0xee4    github.com/cockroachdb/cockroach/pkg/workload/ycsb/ycsb.go:497
#   0x4118052   github.com/cockroachdb/cockroach/pkg/workload/cli.runRun.func2+0x212    github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:440
#   0x4115e1e   github.com/cockroachdb/cockroach/pkg/workload/cli.runRun+0x59e      github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:457
#   0x4114b96   github.com/cockroachdb/cockroach/pkg/workload/cli.CmdHelper.func1+0x5d6 github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:224
#   0x41124ed   github.com/cockroachdb/cockroach/pkg/workload/cli.HandleErrs.func1+0x2d github.com/cockroachdb/cockroach/pkg/workload/cli/cli.go:87
#   0x40d3762   github.com/spf13/cobra.(*Command).execute+0x662             github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:860
#   0x40d3dfc   github.com/spf13/cobra.(*Command).ExecuteC+0x3bc            github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:974
#   0x4668ee8   github.com/spf13/cobra.(*Command).Execute+0x1e8             github.com/spf13/cobra/external/com_github_spf13_cobra/command.go:902
#   0x4668edd   github.com/cockroachdb/cockroach/pkg/cli.Run+0x1dd          github.com/cockroachdb/cockroach/pkg/cli/cli.go:300
#   0x4668e76   github.com/cockroachdb/cockroach/pkg/cli.doMain+0x176           github.com/cockroachdb/cockroach/pkg/cli/cli.go:143
#   0x4668b76   github.com/cockroachdb/cockroach/pkg/cli.Main+0x156         github.com/cockroachdb/cockroach/pkg/cli/cli.go:68
#   0x47f87d6   main.main+0x16                              main/pkg/cmd/cockroach/main.go:26
#   0x49e4d1    runtime.main+0x211                          GOROOT/src/runtime/proc.go:250

To Reproduce

./cockroach workload run ycsb --drop --insert-count=1000000 --workload=A --concurrency=144 --splits=7 --histograms=perf/stats.json --select-for-update=true --ramp=2m --duration=30m {pgurl:1-7}

I230209 15:15:10.680009 1 workload/cli/run.go:387  [T1] 1  DEPRECATION: the --init flag on "workload run" will no longer be supported after 19.2
I230209 15:15:10.833618 1 ccl/workloadccl/fixture.go:319  [T1] 2  starting import of 1 tables

I230209 15:15:38.872457 15 ccl/workloadccl/fixture.go:482  [T1] 3  imported 1.3 GiB in usertable table (1000006 rows, 0 index entries, took 28.022516955s, 47.62 MiB/s)
I230209 15:15:38.872661 1 ccl/workloadccl/fixture.go:327  [T1] 4  imported 1.3 GiB bytes in 1 tables (took 28.038936029s, 47.60 MiB/s)
I230209 15:15:38.984238 1 workload/workloadsql/workloadsql.go:148  [T1] 5  starting 7 splits
I230209 15:15:53.347325 1 workload/cli/run.go:622  [T1] 6  random seed: 12095074689715234192
I230209 15:15:53.347499 1 workload/cli/run.go:429  [T1] 7  creating load generator...
...
# hangs indefinitely here

Expected behavior The workload starts executing statements successfully when running the above command.

Environment:

Jira issue: CRDB-24383

blathers-crl[bot] commented 1 year ago

cc @cockroachdb/test-eng