As is both stated in the NSDI'19 paper and implemented in the code, a response packet will be dropped if the corresponding client packet (i.e., request packet) is still in the wheel due to retransmission:
// Ignore spurious packets received as a consequence of rollback:
// 1. We've only sent pkts up to (ci.num_tx - 1). Ignore later packets.
// 2. Ignore if the corresponding client packet for pkthdr is still in wheel
if (unlikely(pkthdr->pkt_num_ >= ci.num_tx_)) return false;
While this is undoubtfully a correct behavior, I still wonder why the client packet cannot be directly discarded from the wheel (or placing a tombstone). The NSDI'19 paper states in its Appendix C that the reason is because "Carousel requires a bounded difference between the current time and a packet's scheduled transmission time for correctness", and rolling back the state in Timely would be too complex.
However, after reading the SIGCOMM'17 Carousel paper, I see no connection between the necessity to roll back Timely's state and the correctness of Carousel's rate limiter (the timing wheel, if I understand it correctly). It seems that placing a tombstone and ignoring the retransmission packet when transferring entries in wheel slots to the ready queue is an easy task and causes no correctness issues. Timely's internal state need not be reverted -- this should only affect performance, not correctness.
For datacenter networks, it should be rare for a client to receive a response packet after a 5ms timeout and when the retransmission packet is still in the timing wheel. Since eRPC does not optimize much for the uncommon case, I understand the design choice that we directly drop the current response packet and wait for the retransmission response. But I cannot understand the reasoning in the NSDI'19 paper for this design, and if possible I would like a bit more explanation.
Please correct me if I have misunderstood eRPC's design or Carousel's rate limiter.
As is both stated in the NSDI'19 paper and implemented in the code, a response packet will be dropped if the corresponding client packet (i.e., request packet) is still in the wheel due to retransmission:
While this is undoubtfully a correct behavior, I still wonder why the client packet cannot be directly discarded from the wheel (or placing a tombstone). The NSDI'19 paper states in its Appendix C that the reason is because "Carousel requires a bounded difference between the current time and a packet's scheduled transmission time for correctness", and rolling back the state in Timely would be too complex.
However, after reading the SIGCOMM'17 Carousel paper, I see no connection between the necessity to roll back Timely's state and the correctness of Carousel's rate limiter (the timing wheel, if I understand it correctly). It seems that placing a tombstone and ignoring the retransmission packet when transferring entries in wheel slots to the ready queue is an easy task and causes no correctness issues. Timely's internal state need not be reverted -- this should only affect performance, not correctness.
For datacenter networks, it should be rare for a client to receive a response packet after a 5ms timeout and when the retransmission packet is still in the timing wheel. Since eRPC does not optimize much for the uncommon case, I understand the design choice that we directly drop the current response packet and wait for the retransmission response. But I cannot understand the reasoning in the NSDI'19 paper for this design, and if possible I would like a bit more explanation.
Please correct me if I have misunderstood eRPC's design or Carousel's rate limiter.