MAVLink packet buffering improvements

[LorenzMeier]

The current MAVLink UART link implementation is optimized for low-bandwidth links and telemetry and hasn't been originally architected for log streaming or other high-throughput applications. This introduces a number of artefacts when running a fast link:

• The FIFO buffer sizes of the UARTs are inadequate for high throughput. Due to the nature of an asynchronous system, there will be a lot of telemetry AND logging data at the same time as the main system states (attitude and position) update. This means that 50% or so of the whole data is being generated at almost the same instant at 250 Hz
  • Example calculation: Configured bandwidth: 80 KByte/s. Then the transmit buffer size required as the absolute minimum when log streaming is enabled is one 80K over 250 times two (mavlink and log streaming update at the same time) = (80000/250)*2 = 640. However, this assumes zero CPU load. In reality, a significant multiple will be needed (3-10K).
  • Keeping MAVLink packets in a UART FIFO is suboptimal from the perspective of overall RAM usage. It means that very high buffer capacity has to be held for each UART, independent of whether it is in use or not.
  • There is a separate but related problem with respect to priorities: For links that carry command & control MAVLink data (which inherently should be real-time) and at the same time carry log and other data streams we should establish priority queues to not delay time-critical messages

Instead of jumping on all this and never get out of WIP state, I want to address this in a structured fashion:

• Fix the configuration of the link based on some basic calculations in terms of bandwidth. For UARTs that transmit at 80K, the TX buffer should be 5-10K
• Instrument the link and make sure nothing fails that is supposed to not fail. This is particularly relevant for transmit and receive counters. We need to be sure that we get into a state where nothing is dropped and if things drop, we can infer from local counters at which state in the pipeline.
• Look only then at the overall design and work out a new queuing structure. For this, priorities play a role as well as DMA and we may opt to queue packetized instead of the UART FIFO.

[LorenzMeier]

@baumanta I would suggest a review session with @bkueng and @julianoes of the current instrumentation and to add mavlink drop counts (when the MAVLink sequence number has holes) to the telemetry_status log message.

@bkueng @julianoes We need something like RADIO_STATUS as a new MAVLink packet that is sent back to the autopilot by mavlink_router which contains the instrumentation data of how much packet drop it sees and which RX and TX rates it sees. That would allow us to see what both sides of the link see and infer where issues are. QGC should also send this message back, which would then allow us to locate breakages.

[LorenzMeier]

@catch-twenty-two I'm assigning you to this. Please re-assign if you switch context to something else.

[dagar]

@bkueng @julianoes We need something like RADIO_STATUS as a new MAVLink packet that is sent back to the autopilot by mavlink_router which contains the instrumentation data of how much packet drop it sees and which RX and TX rates it sees. That would allow us to see what both sides of the link see and infer where issues are. QGC should also send this message back, which would then allow us to locate breakages.

This would also enable us to implement congestion control at the Mavlink level for things like UDP or serial without CTS/RTS or RADIO_STATUS. https://github.com/PX4/Firmware/issues/10618#issue-365452413

@dogmaphobic FYI

[dogmaphobic]

@dagar I won't be able to get to this until next week or so. What would be the ideal information you want back from the (any) router/bridge? And QGC while at it...

[dagar]

@dagar I won't be able to get to this until next week or so. What would be the ideal information you want back from the (any) router/bridge? And QGC while at it...

The data in the QGC connection status would be a good start.

Where this potentially gets a bit messy is routing. I'm also wondering if the uint8_t seq is insufficient beyond the simple point to point slow serial radio case.

[dogmaphobic]

The data in the QGC connection status would be a good start.

I assume a new set of message to convey this data?

Where this potentially gets a bit messy is routing. I'm also wondering if the uint8_t seq is insufficient beyond the simple point to point slow serial radio case.

It is an issue if more than 255 messages are lost in one go. The code detects wrap-arounds and take them into account, which is ok if only < 255 sequential messages are lost. Making it 16 bits would decrease the possible wrap around errors considerably but that involves changing a very basic counter in all MAVLink messages. A bit too much of a change, no?

[dagar]

Where this potentially gets a bit messy is routing. I'm also wondering if the uint8_t seq is insufficient beyond the simple point to point slow serial radio case.

It is an issue if more than 255 messages are lost in one go. The code detects wrap-arounds and take them into account, which is ok if only < 255 sequential messages are lost. Making it 16 bits would decrease the possible wrap around errors considerably but that involves changing a very basic counter in all MAVLink messages. A bit too much of a change, no?

Yes that would be a difficult change, I'm just concerned that's actually not that many messages to miss on these fast connections. Obviously this needs to be tested one way or the other, but maybe sending the overall sent and received count in a mavlink (or link?) status message would help fill the gap (if any).

MAVLINK_STATUS

• uint32_t messages_sent
• uint32_t messages_received (estimated from counting seq)
• uint32_t messages_lost (estimated from gaps in seq)
• uint32_t rx_rate (bytes/s)
• uint32_t tx_rate (bytes/s)

Update: WIP branch https://github.com/mavlink/mavlink/tree/pr-mavlink_link_status

[nbelanger99]

Have you found any workarounds for this issue? I'm currently facing a similar issue when running Mavros. The issue happens consistently when the Pixhawk is rebooted, and sometimes it happens after just running for some time. We see "TM : RTT too high for timesync" on the Mavros side, and invoking mavlink status in NSH, we see that packets are being dropped and the "tx rate mult" coefficient decreases to less than 1.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. Thank you for your contributions.