Return the `actually committed index` if it's greater than the `official committed index` for `ReadIndex`

[ahrtr]

When etcd receives committed Entries, it does the following three things concurrently,

• apply the entries and respond to client
• persist the hard state
• persist the applied data to bboltDB

It's possible that it has successfully responded to the client, but fails to persist the hardstate and data to bboltDB, such as crashes right before persisting the hardstate(including committed index) and the applied data to bboltDB.

After etcd starts again,

• The initial committed index of the raft is an out of date value;
• When a client sends a linearizable request, it gets the readIndex firstly, and get the old value; accordingly won't wait for the new entries being applied, and respond the client with an stale data.
  • Note that etcd replays WAL entries on bootstrap, but this step may be executed before the replaying finishes.

[serathius]

Not yet convinced that the problem is with readIndex. I would be interested more into persistence of HardState. HardState should be also persisted to WAL, isn't it guaranteed to be written?

[ahrtr]

See https://github.com/etcd-io/etcd/pull/16675.

This PR is a standalone change for raft.

[ahrtr]

Won't continue to work on this.

https://github.com/etcd-io/etcd/issues/16666#issuecomment-1747077487