storage: raft replays on idle single-node cluster

[tbg]

I'm seeing what I think are Raft replays on LeaderLease requests every couple of minutes or so. While Raft replays are expected generally, it would seem that we shouldn't be seeing any in an idling single-node setup.

Log from some WIP on leader leases below. Essentially these say that we see LeaderLease requests for the same timestamp in quick succession, which is likely a replay at work.

I160404 09:13:08.637844 storage/replica_command.go:1384  range 63: normal leader lease replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 405493h13m9.636936635s [lowWater=wallTime+-15.734878ms] following replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 405493h13m8.622054486s
I160404 09:13:08.638141 storage/replica_command.go:1379  range 63: irregular leader lease replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 405493h13m9.636936635s [lowWater=wallTime+998.881652ms] following replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 405493h13m9.636936635s

cc @andreimatei: didn't you see unexpected replays during on of your SQL tests a while ago? I looked for an issue which I thought you filed but to no avail.

[andreimatei]

I haven't filed yet. It was on my list, but I was hoping to figure out some specific repro info first. In any case, simply finding a replay is pretty easy. Just put a panic here: https://github.com/cockroachdb/cockroach/blob/379c2a48f7af601931c80ba94c2c9354acc916e5/testutils/storageutils/mocking.go#L97 and

make stress PKG=./sql TESTS=TestTxnRestart

[bdarnell]

When GRPC went in (#4427) we got a lot more aggressive about retries because we started running into a case when messages could get dropped on the floor. As a result, if a proposal races with a Tick event, commands can get replayed even in a single-node cluster which should theoretically never experience them. Are there problems resulting from these replays? We should definitely clean this up, it's just a question of prioritization.

@tamird Do you remember why exactly we had to add this with GRPC? I know that there's some risk because there's no built-in retries for MsgProp, but I can't remember what led us to start hitting this in practice.

[tbg]

No, I filed this because I found it surprising and wanted to make sure this is fine. As long as it isn't frequent enough to impact performance, it's actually somewhat helpful (since it exposes our single-node test cases to replays).

On Mon, Apr 4, 2016 at 11:58 AM Ben Darnell notifications@github.com wrote:

When GRPC went in (#4427 https://github.com/cockroachdb/cockroach/pull/4427) we got a lot more aggressive about retries because we started running into a case when messages could get dropped on the floor. As a result, if a proposal races with a Tick event, commands can get replayed even in a single-node cluster which should theoretically never experience them. Are there problems resulting from these replays? We should definitely clean this up, it's just a question of prioritization.

@tamird https://github.com/tamird Do you remember why exactly we had to add this with GRPC? I know that there's some risk because there's no built-in retries for MsgProp, but I can't remember what led us to start hitting this in practice.

— You are receiving this because you authored the thread.

Reply to this email directly or view it on GitHub https://github.com/cockroachdb/cockroach/issues/5835#issuecomment-205366420

[tbg]

@bdarnell with your recent changes, can this close?

[bdarnell]

Yeah, I think this can close. The replays are now much less frequent, and I don't think we're going to completely get rid of them. (we could do so by tracking the proposal tick for each command and replaying it on N ticks after its initial proposal, instead of replaying every proposal at once on multiples of N ticks, but that doesn't seem worth the trouble)