Log download throughput well beyond the maximum

[AndreasAntener]

@dogmaphobic @LorenzMeier I'm really trying to get log downloading working over Wifi but I'm hitting low limits. It may be that the ESP can't parse the messages any faster, but I don't know how to measure that yet. The throughput I currently achieve is 100 to 200 log data messages per second (depending on rate settings on the PX4 side and what other messages go through). This results in an effective download speed of 20kB/s max.

My basic question is: can we do better? Theoretically with the 921600 serial speed 100kB/s should be possible or am I mistaken?

Btw, I'm already buffering the data on the ESP side and sending it down to the client over HTTP. The numbers I mentioned before resulted from timing how fast the buffer fills on the ESP side, so no Wifi involved there.

[LorenzMeier]

How about the serial buffer on that UART? It should be 1.5-2KB in the defconfig (search for that FileMaker for FMUv4)

[AndreasAntener]

So I assume USART2?

#
# USART1 Configuration
#
CONFIG_USART1_RXBUFSIZE=600
CONFIG_USART1_TXBUFSIZE=4000
CONFIG_USART1_BAUD=115200
CONFIG_USART1_BITS=8
CONFIG_USART1_PARITY=0
CONFIG_USART1_2STOP=0
# CONFIG_USART1_IFLOWCONTROL is not set
# CONFIG_USART1_OFLOWCONTROL is not set

#
# USART2 Configuration
#
CONFIG_USART2_RXBUFSIZE=600
CONFIG_USART2_TXBUFSIZE=1100
CONFIG_USART2_BAUD=57600
CONFIG_USART2_BITS=8
CONFIG_USART2_PARITY=0
CONFIG_USART2_2STOP=0
CONFIG_USART2_IFLOWCONTROL=y
CONFIG_USART2_OFLOWCONTROL=y

#
# USART3 Configuration
#
CONFIG_USART3_RXBUFSIZE=300
CONFIG_USART3_TXBUFSIZE=300
CONFIG_USART3_BAUD=57600
CONFIG_USART3_BITS=8
CONFIG_USART3_PARITY=0
CONFIG_USART3_2STOP=0
CONFIG_USART3_IFLOWCONTROL=y
CONFIG_USART3_OFLOWCONTROL=y

#
# UART4 Configuration
#
CONFIG_UART4_RXBUFSIZE=300
CONFIG_UART4_TXBUFSIZE=300
CONFIG_UART4_BAUD=57600
CONFIG_UART4_BITS=8
CONFIG_UART4_PARITY=0
CONFIG_UART4_2STOP=0
# CONFIG_UART4_IFLOWCONTROL is not set
# CONFIG_UART4_OFLOWCONTROL is not set

#
# USART6 Configuration
#
CONFIG_USART6_RXBUFSIZE=300
CONFIG_USART6_TXBUFSIZE=300
CONFIG_USART6_BAUD=57600
CONFIG_USART6_BITS=8
CONFIG_USART6_PARITY=0
CONFIG_USART6_2STOP=0
# CONFIG_USART6_IFLOWCONTROL is not set
# CONFIG_USART6_OFLOWCONTROL is not set

[AndreasAntener]

Only had time to do a quick test so far, but it helped :). The UART buffer is now as big as the buffer I use on the ESP side before sending it down. I get the speed up to 50kB/s end to end. I realized too now that if the UART TX buffer is smaller than the one on the ESP side I'm starting to get errors (either out of order packets or losses).

#
# USART3 Configuration
#
CONFIG_USART3_RXBUFSIZE=600
CONFIG_USART3_TXBUFSIZE=2500

[AndreasAntener]

What are other impacts of increasing the TX buffer?

[LorenzMeier]

Generally low memory in the system. I think to get faster what you now would potentially need is TX DMA.

[dogmaphobic]

The "store & forward" (buffering) logic within the mavesp could use some tweaking as well. The idea is to try to fill a UDP frame with as much MAVLink messages it can. Right now it simply counts how many messages, regardless of their sizes using a quite conservative limit before deciding to flush the UDP packet. Maximizing the use of the 1500-byte frame would also help the throughput.

[AndreasAntener]

@dogmaphobic I'm actually buffering the log data separately and making it available over HTTP at the moment. I thought that this would get around the initial problems I had but I guess I'll try revert again see how it looks now, maybe optimizing framing. It seems the bottleneck is anyway the serial port at the moment.