Strange behaviour of --insert-count option in YCSB workload

[eivanov89]

Describe the problem

According the help it is a Number of rows to sequentially insert before beginning workload. I can assume that it either affects workloads which do insert operations or it forces some insertions before any workload.

1. In case of workload A (with updates) and workload C (read only) I don't see any insertions in metrics.
2. This flag affects results in mysterious way (concurrency 512):
   • uniform workload C: 138K op/s when --insert-count and 207K op/s when 10000000.
   • uniform workload A: 1K and 260K for --insert-count 1 vs 5000000
   • zipfian workload A: 1K and 24K for --insert-count 1 vs 5000000

Example command:

cockroach workload run ycsb --workload a --request-distribution uniform --max-ops 5000000 --insert-count 1 --display-every 10s postgresql://root@localhost:26257?sslmode=disable --concurrency 512

In cluster I have 8 servers, each one is 128 cores, 4 NVMe disks. There are 4 cockroach instances with 32 cores (set by taskset) and 1 disk per server. Replication factor is default.

Jira issue: CRDB-25230

[blathers-crl[bot]]

Hello, I am Blathers. I am here to help you get the issue triaged.

Hoot - a bug! Though bugs are the bane of my existence, rest assured the wretched thing will get the best of care here.

I have CC'd a few people who may be able to assist you:

• @dhartunian (found keywords: metrics)

If we have not gotten back to your issue within a few business days, you can try the following:

• Join our community slack channel and ask on #cockroachdb.
• Try find someone from here if you know they worked closely on the area and CC them.

_{:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[ajwerner]

I think the issue here is that --insert-count only applies to cockroach workload init ycsb and not cockroach workload run ycsb.

[eivanov89]

It seems to be a weird difference in 99%:

./cockroach/cockroach workload run ycsb --workload c --request-distribution uniform --max-ops 5000000 --insert-count 1 --display-every 10s postgresql://root@ydb-vla-dev04-000.search.yandex.net:26257?sslmode=disable --concurrency 1
I230310 12:22:46.804731 1 workload/cli/run.go:427  [-] 1  creating load generator...
I230310 12:22:46.824254 1 workload/cli/run.go:458  [-] 2  creating load generator... done (took 19.522111ms)
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
   10.0s        0            4.0            4.0    251.7    260.0    268.4    268.4 read
   20.0s        0            4.1            4.0    251.7    260.0    268.4    268.4 read
   30.0s        0            4.0            4.0    251.7    268.4    302.0    302.0 read
...
eivanov89@ydb-vla-dev02-000:~$ ./cockroach/cockroach workload run ycsb --workload c --request-distribution uniform --max-ops 5000000 --insert-count 1000 --display-every 10s postgresql://root@ydb-vla-dev04-000.search.yandex.net:26257?sslmode=disable --concurrency 1
I230310 12:25:55.670542 1 workload/cli/run.go:427  [-] 1  creating load generator...
I230310 12:25:55.690632 1 workload/cli/run.go:458  [-] 2  creating load generator... done (took 20.088996ms)
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
   10.0s        0          321.1          321.8      2.6      2.9      3.3    268.4 read
   20.0s        0          364.0          342.9      2.6      2.8      3.1    251.7 read
   30.0s        0          385.4          357.1      2.6      2.8      3.1      8.9 read

If I remove --insert-count then result is similar to --insert-count 10000 (the default value of insert-count).

[eivanov89]

I think the issue here is that --insert-count only applies to cockroach workload init ycsb and not cockroach workload run ycsb.

But then shouldn't there be a validation and error in case of cockroach workload run ycsb?

[ajwerner]

It does seem to also affect the workload. The workload does not validate that you've set it up correctly. The assumption of the workload, to the extent that I understand it, is that these flags are set to the same value between init and run. It looks like the flag both controls the initial data load and the number of rows available to the workload.

I think an ask for run to validate that the correct amount of data exists is a fine ask, and PRs are welcome.

[eivanov89]

It looks like the flag both controls the initial data load and the number of rows available to the workload.

Though imho the flag "--record-count" seems to be better to specify the initial data size. According the help

      --insert-count int                  Number of rows to sequentially insert before beginning workload. (default 10000)
      --insert-start int                  Key to start initial sequential insertions from. (default 0)
      --record-count int                  Key to start workload insertions from. Must be >= insert-start + insert-count. (Default: insert-start + insert-count)

So that I upload the data with combination of --insert-count and possibly --insert-start and then run workloads either by specifying same flags (a little bit confusing) or just --record-count.

[eivanov89]

It's very confusing that in the original YCSB same properties have different meaning:

• recordcount: the number of records in the dataset at the start of the workload. used when loading for all workloads. (default: 1000)
• insertstart: for parallel loads and runs, defines the starting record for this YCSB instance (default: 0)
• insertcount: for parallel loads and runs, defines the number of records for this YCSB instance (default: recordcount)