udpm: Queue is "first-one-wins", but some applications may want "last-one-wins" (as they may see delay)

[EricCousineau-TRI]

TL;DR Current setup for udpm:// stuff may cause new messages to get dropped, and applications will see what looks like delays.

Suggested steps:

• [ ] Update user-facing documentation to denote what the queue behavior is for each of the LCM implementations (e.g. memq, udpm, etc.)
  • Ideally, suggest workarounds when "last-one-wins" is expected, but not directly achievable.
• [ ] Add a configurable option (either globally, in an LCM object, or in the subscriber object) to dictate what the queue overflow behavior is, and somehow reflect that in the lcm_try_enqueue_message-centered logic.

---

We are using a mix of the C++ and Python interfaces for LCM, and sometimes dip into the C layer.

When perusing the user-facing documentation for LCM (the website, and the C and Python user docs), there are notes about queue sizes, but I could easily find anything that states how LCM reacts when the queue is full. Since it's FIFO, I expected "last-one-wins" (newer messages will always be available).

With @jwnimmer-tri's help, it looks like it's "first-one-wins", meaning newer messages will get discarded when the buffer is full. In lcm_udpm.c:

• _recv_short_message - uses lcm_try_enqueue_message to see if it should copy bytes from packet
• lcm_try_enqueue_message - ignores new messages if queue is full

I have modified this repository to show the "first-one-wins" queue dropping: #346 https://github.com/lcm-proj/lcm/blob/f254ee913e0d0f64d2a8f139099dd0b087e3b358/test/python/lcm_udpm_queue_issue_test.py#L31-L77

In some of the applications / libs I wrote, I (wrongly) expected it to be "last-one-wins"; it is fixable, but it's a bit burdensome to work around this. It would be nice to make this configurable. (FWIW Jeremy also explained the rationale for "first-one-wins", which I understand, but I believe in most of our applications, we're not super compute- or memory-bounded acutely in terms of the LCM UDPM recv_thread.)

[ggould-tri]

FYI at this point both @calderpg-tri and I have run into this (him for super-high-frequency-send, low-frequency-read messages, me for messages sent prior to the start of a fast polling loop). So there is still interest in this issue, even if it remains idle.

[achilsh]

ye, can try using priority queue to store, then consumer discard timeout data compared with recevied data timestamp?

[lauralindzey]

I have also run into this; I have a driver that can only run at ~10 Hz, but commands come at 50 Hz. First-one-wins means that if I try to use LCM's set_queue_capacity, I'll introduce an unnecessary ~100ms delay into my control chain.