shribe
commented
3 years ago Update: at about 3.5 hours, the producer to the single-node cluster quit as follows:
66.44 MiB/s; 696.64k records/s
67.77 MiB/s; 710.57k records/s
53.30 MiB/s; 558.87k records/s
0.00 MiB/s; 0.00k records/s
0.00 MiB/s; 0.00k records/s
0.00 MiB/s; 0.00k records/s
0.00 MiB/s; 0.00k records/s
0.00 MiB/s; 0.00k records/s
0.00 MiB/s; 0.00k records/s
0.00 MiB/s; 0.00k records/s
0.00 MiB/s; 0.00k records/s
0.00 MiB/s; 0.00k records/s
90.33 MiB/s; 947.22k records/s
71.89 MiB/s; 753.79k records/s
53.70 MiB/s; 563.08k records/s
26.85 MiB/s; 281.50k records/s
0.00 MiB/s; 0.00k records/s
0.00 MiB/s; 0.00k records/s
0.00 MiB/s; 0.00k records/s
304.17 MiB/s; 3189.45k records/s
142.15 MiB/s; 1490.52k records/s
101.49 MiB/s; 1064.16k records/s
produce error: EOF
emaxerrno
twmb
mmaslankaprv
dotnwat
isaacsanders
jcsp
Just running benchmarks, against a cluster of 3 replicas, in kubernetes, with all other settings the same as your single replica example. After a while of running at 600K-1.4M rps (depending on which k8s cluster), we usually get a run of 0rps reported, then it picks back up. After some small number of iterations of this, it stalls permanently. (Quitting and restarting client does not help.)
We have noticed that the redpanda containers are restarting, and the stalls to 0rps happen around the time of the restarts. The logs around the container quits seem to reflect memory allocation activity. It seems that if I increase memory allocation, it takes substantially longer for this to happen.
It seems that if a cluster is run with only 1 replica, this does not happen (of course it's not definitive, but my single-node test has been running for almost 3 hours, much longer than a 3-replica test has ever lasted).
The containers are not running out of disk.
When it happens, the test application from franz-go/examples/bench run at debug log level produces the following output:
A check with
rpkshows this:It looks kind of like maybe the container restarts shuffled the leader around, but something didn't get updated with the new leader info. (At this point, all the containers have multiple restarts: 2, 3, 3.) In fact, taking a look at the still-running single-node one, it has 4 restarts, but the bench utility has never reported less than 500Krps--if the problem is propagating new leader on crash, that wouldn't affect a single node cluster, right?
I've attached redpanda logs; the numbers 0-2 are the node numbers; "a" is the prior log of the container, in other words the "crashed" one; "b" is the log of the currently running one when the above was happening.
0a.gz 0b.gz 1a.gz 1b.gz 2a.gz 2b.gz