search

lni / dragonboat

A feature complete and high performance multi-group Raft library in Go.
Apache License 2.0
4.99k stars 534 forks source link

SendQueue: I | server: rate limited, v: 18446744073709551408, maxSize 134217728 #221

Closed xkeyideal closed 2 years ago

xkeyideal commented 2 years ago

Note: for reported bugs, please fill in the following details. bug reports without detailed steps on how to reproduce will be automatically closed.

Dragonboat version

v3.3.2

it's seems call Decrease first, when r.size == 0

// Decrease decreases the recorded in memory log size by sz bytes. func (r *RateLimiter) Decrease(sz uint64) { atomic.AddUint64(&r.size, ^(sz - 1)) }

MaxSendQueueSize: 128 1024 1024 MaxReceiveQueueSize: 0

lni commented 2 years ago

Thanks for reporting the bug, I will have a look soon.

In the mean time you can probably just disable the rate limiter in case it is causing your system problem.

lni commented 2 years ago

@xkeyideal

This race is caused by the fact that sq.increase(req) is not called before sending the req to the worker goroutine, it is thus possible for the worker goroutine to call decrease() before increase() is called.

The fix is pretty simple, just need to call sq.increase(req) before the select https://github.com/lni/dragonboat/blob/b8a6bb8561483b53bd1eea49cf8c2a159c265437/internal/transport/transport.go#L404

then call decrease() here to undo the increase() in case the queue if full.

@xkeyideal would you like to send in a PR to become a dragonboat contributor?