search

scylladb / scylla-bench

43 stars 36 forks source link

Unexpected writes. #72

Closed pveentjer closed 3 years ago

pveentjer commented 3 years ago

I'm running the following:

go/bin/scylla-bench -workload uniform -mode read -partition-count 10000000 -nodes 172.31.14.110 -concurrency 1600 -duration 30m -clustering-row-count 1 -host-selection-policy=token-aware -connection-count=1 2>&1 | tee -a scylla-bench-19-08-2021_10-48-38.log
go/bin/scylla-bench -workload uniform -mode read -partition-count 10000000 -nodes 172.31.14.110 -concurrency 1600 -duration 30m -clustering-row-count 1 -host-selection-policy=token-aware -connection-count=1 2>&1 | tee -a scylla-bench-19-08-2021_10-48-38.log
go/bin/scylla-bench -workload uniform -mode read -partition-count 10000000 -nodes 172.31.14.110 -concurrency 1600 -duration 30m -clustering-row-count 1 -host-selection-policy=token-aware -connection-count=1 2>&1 | tee -a scylla-bench-19-08-2021_10-48-38.log
go/bin/scylla-bench -workload uniform -mode read -partition-count 10000000 -nodes 172.31.14.110 -concurrency 1600 -duration 30m -clustering-row-count 1 -host-selection-policy=token-aware -connection-count=1 2>&1 | tee -a scylla-bench-19-08-2021_10-48-38.log

As you can see, I'm only reading. But when I go to Scylla monitor, I can see 1K writes/second on a single shard (shard 6).

image

This is unexpected since this is a readonly test.

pveentjer commented 3 years ago

I think there was still something in the background because I'm switching between cassandra-stress and scylla-bench. I created a small tool to kill everything that running on the load generator machines and the issue seems to be resolved.