UDPSocket::try_recv_buf_from returns truncated datagrams under high load.

[dvolodin7]

Version tokio v1.2.0

Platform Linux f4d7b7fa38aa 4.19.121-linuxkit #1 SMP Tue Dec 1 17:50:32 UTC 2020 x86_64 GNU/Linux

Description We're working on universal agent for our monitoring system (NOC). We've developed TWAMP sender and reflector collectors to measure network channel's SLA characteristics under various traffic models. TWAMP test-session relies on UDP and we use following code to receive and parse probe packets.

    pub async fn recv_from<T: FrameReader>(&mut self) -> Result<(T, SocketAddr), Box<dyn Error>> {
        self.buffer.clear();
        loop {
            self.socket.readable().await?;
            match self.socket.try_recv_buf_from(&mut self.buffer) {
                Ok((_, addr)) => {
                    let r = T::parse(&mut self.buffer)?;
                    return Ok((r, addr));
                }
                Err(ref e) if e.kind() == std::io::ErrorKind::WouldBlock => {
                    continue;
                }
                Err(e) => {
                    return Err(Box::new(e));
                }
            }
        }
    }

Starting from certain load, 20kpps approx, UDPSocket::try_recv_buf_from sometimes begins to return incomplete frames, causing parsing error. With increasing load up to 45kpps, almost every call returns broken frame.

I expect recvfrom-style functions should either return an error, block call or return full datagram, as long as it can be stored in provided buffer.

We suspect that MSG_WAITALL option should be set to underlying recvfrom call to meet the expected behavior.

TWAMP test session uses fixed-size header (14 octets for request, 41 octet for reply), padded by random data up to size of packet modelled. So we have not an option to issue saveral try_recv_buffer_from until T::parse is satisfied.

NB: #3486 may have same root cause.

[Arnavion]

(This isn't related to #3486 - #3486 is about having sufficient buffer space but it's split over multiple buffers, and would require vectored recv to solve. In your case your program has a single buffer that's not large enough.)

[dvolodin7]

I doubt the buffer space is the issue here. It is simple ping-pong cause. TWAMP sender sends the test packet and waits for response. TWAMP responder receives the test packet and reflects response back. So buffer usage should not depend upon packets rate.

[dvolodin7]

Regarding the buffer size, I've opened #3542, as the UDPSocket has no appropriate API for buffer size manipulation.