Derecho-Project / derecho

The main code repository for the Derecho project.
BSD 3-Clause "New" or "Revised" License
182 stars 46 forks source link

Timestamp order is guaranteed in a shard, even with loosely synchronized clock. #267

Closed songweijia closed 5 months ago

songweijia commented 5 months ago

Previously, the timestamp for each message in a shard is assigned by the sending member. If the members in a shard have drifting clocks, a message with a higher version might have a lower timestamp than another message with a lower version. We call it out-of-order timestamps. Although we manage the timestamp index independently, it would be nice if timestamp and version orders are consistent. Here we borrow the idea of the HLC clock: on message delivery, the current timestamp to assign is checked against the previous one. If the timestamp goes back, we enforce it to be the previous timestamp + 1 using HLC rules. This operation is consistent among all replicas as the message order is guaranteed among them.