UDP streaming dropping samples in OS X

[a1k0n]

This is weird. I kept losing sync on a DMR stream... well, long story short, it appears the UDP socket is dropping samples. I became suspicious as DSD kept de-synchronizing, but if I saved a demodulated .WAV out and ran that through DSD, it worked fine. It also works fine to play back a raw I/Q stream -- this only seems to happen during live capture from my RTL-SDR, which is even more weird.

I don't think it's losing entire UDP packets. After a bit of investigation, it turned out to only be multiples of 16 samples. Here's what I did:

I started gqrx, tuned to a frequency with a narrowband FM demodulator, started nc -u -l localhost 7355 >blah in a terminal, and quickly hit the UDP and Rec buttons. I saved out about 20 seconds of 48kHz samples. I then converted the .WAV back to 16-bit raw and killed netcat, so I had two 16-bit raw samples which should be identical, but unaligned. The UDP stream was missing about a half second of samples at this point.

I then ran a correlation to align them at the beginning, and found this:

After 10485 samples, the next 16 samples are missing from the UDP stream but present in the WAV. And after another 527 samples, it skipped 224 samples. I haven't discerned any particular pattern.

Also, it seems the rounding is slightly different between the two samples; sometimes the 16-bit values are off by one.

Config: OS X 10.10.5, gqrx 2.5.3 installed from website, RTL2838UHIDIR.

[csete]

Thanks for reporting this. This feature has existed for a long time but there hasn't been much testing or feedback, so it is not unlikely that there are bugs.

The wav writer is a gnuradio block and has it's own float to signed integer converter, so that probably explains the rounding error. I may have forgotten to add 0.5 in the UDP sample converter or something like that.

As for the dropped samples, the only thing I can think of here and now is perhaps high CPU load prevents the UDP reader to get all samples from the receiver output, or perhaps the read buffer is just too small. I'm pretty sure that code could use a thorough review but unfortunately I don't know when I will get some time to do that.

[a1k0n]

Hmm. It seems to me that the wave recorder is also a gnu radio block, and that has no such issue.

It must be some sort of buffer overrun, since the raw IQ doesn't have the issue, but I'm not sure exactly what's going on; anything written to the UDP socket should have been buffered by the OS and immediately dumped to the pipe. I don't see any accumulation of bytes in the socket buffers in netstat. I'll poke into the gnuradio implementation.

[a1k0n]

This is interesting. The UDP packets being sent have wildly varying sizes; I was expecting to see some fixed buffer chunk size being decimated to 1472-byte packets plus leftovers, but instead it's all over the map:

11:03:00.109246 IP6 localhost.65449 > localhost.7355: UDP, length 398
11:03:00.109528 IP6 localhost.65449 > localhost.7355: UDP, length 128
11:03:00.109763 IP6 localhost.65449 > localhost.7355: UDP, length 288
11:03:00.110032 IP6 localhost.65449 > localhost.7355: UDP, length 32
11:03:00.110321 IP6 localhost.65449 > localhost.7355: UDP, length 320
11:03:00.110543 IP6 localhost.65449 > localhost.7355: UDP, length 512
11:03:00.110729 IP6 localhost.65449 > localhost.7355: UDP, length 256
11:03:00.110915 IP6 localhost.65449 > localhost.7355: UDP, length 128
11:03:00.111100 IP6 localhost.65449 > localhost.7355: UDP, length 576
11:03:00.111252 IP6 localhost.65449 > localhost.7355: UDP, length 512
11:03:00.111406 IP6 localhost.65449 > localhost.7355: UDP, length 32
11:03:00.111556 IP6 localhost.65449 > localhost.7355: UDP, length 14
11:03:00.111659 IP6 localhost.65449 > localhost.7355: UDP, length 1026
11:03:00.111832 IP6 localhost.65449 > localhost.7355: UDP, length 64
11:03:00.111962 IP6 localhost.65449 > localhost.7355: UDP, length 32
11:03:00.112070 IP6 localhost.65449 > localhost.7355: UDP, length 12
11:03:00.112700 IP6 localhost.65449 > localhost.7355: UDP, length 1012
11:03:00.112879 IP6 localhost.65449 > localhost.7355: UDP, length 768
11:03:00.112968 IP6 localhost.65449 > localhost.7355: UDP, length 512
11:03:00.113070 IP6 localhost.65449 > localhost.7355: UDP, length 256
11:03:00.113239 IP6 localhost.65449 > localhost.7355: UDP, length 144
11:03:00.113343 IP6 localhost.65449 > localhost.7355: UDP, length 6
11:03:00.181058 IP6 localhost.65449 > localhost.7355: UDP, length 58
11:03:00.181079 IP6 localhost.65449 > localhost.7355: UDP, length 6
11:03:00.181367 IP6 localhost.65449 > localhost.7355: UDP, length 234
11:03:00.181466 IP6 localhost.65449 > localhost.7355: UDP, length 14
11:03:00.181597 IP6 localhost.65449 > localhost.7355: UDP, length 128
11:03:00.181882 IP6 localhost.65449 > localhost.7355: UDP, length 258
11:03:00.182143 IP6 localhost.65449 > localhost.7355: UDP, length 256
11:03:00.182291 IP6 localhost.65449 > localhost.7355: UDP, length 256
11:03:00.182449 IP6 localhost.65449 > localhost.7355: UDP, length 512
11:03:00.182532 IP6 localhost.65449 > localhost.7355: UDP, length 16
11:03:00.182727 IP6 localhost.65449 > localhost.7355: UDP, length 640
11:03:00.183201 IP6 localhost.65449 > localhost.7355: UDP, length 512
11:03:00.183295 IP6 localhost.65449 > localhost.7355: UDP, length 256
11:03:00.183685 IP6 localhost.65449 > localhost.7355: UDP, length 512
11:03:00.183792 IP6 localhost.65449 > localhost.7355: UDP, length 384
11:03:00.183902 IP6 localhost.65449 > localhost.7355: UDP, length 4
11:03:00.183988 IP6 localhost.65449 > localhost.7355: UDP, length 2
11:03:00.184226 IP6 localhost.65449 > localhost.7355: UDP, length 1338
11:03:00.184336 IP6 localhost.65449 > localhost.7355: UDP, length 6
11:03:00.184468 IP6 localhost.65449 > localhost.7355: UDP, length 42
11:03:00.184573 IP6 localhost.65449 > localhost.7355: UDP, length 14
11:03:00.184647 IP6 localhost.65449 > localhost.7355: UDP, length 2
11:03:00.184809 IP6 localhost.65449 > localhost.7355: UDP, length 1408
11:03:00.184893 IP6 localhost.65449 > localhost.7355: UDP, length 64
11:03:00.185064 IP6 localhost.65449 > localhost.7355: UDP, length 32
11:03:00.185175 IP6 localhost.65449 > localhost.7355: UDP, length 16
11:03:00.185221 IP6 localhost.65449 > localhost.7355: UDP, length 4

If I play back from an IQ recording, the same thing happens but with overall smaller packets throughout; none are larger than 256 bytes (at least, when eyeballing it).

Any idea what controls the buffer sizes being propagated through the gnu radio system?

[a1k0n]

BTW, it might be nicer overall to just use a TCP stream instead of UDP for this. I understand the choice to use UDP, and data loss might be OK, but the loss of timing information is annoying.

A smarter UDP protocol which included a sample sequence number header would be nice if you care about unreliable delivery; then a remote demodulator could at least deal with missing segments without messing up its clock recovery completely.

[csete]

It seems to me that the wave recorder is also a gnu radio block, and that has no such issue

Yes,I meant that the UDP streamer is something that I have implemented, but now I look and can see that it is also a gnuradio block.

Any idea what controls the buffer sizes being propagated through the gnu radio system?

It's the gnuradio runtime, but the default payload size for the UDP sink block should be 1472 c.f. https://gnuradio.org/doc/doxygen/classgr_1_1blocks_1_1udp__sink.html#aa321c327a5031e6587b58d346420d61f

[a1k0n]

It's the gnuradio runtime, but the default payload size for the UDP sink block should be 1472 c.f.

Right, so why am I seeing such odd packet sizes coming through, is what I'm wondering. Occasionally I do see 1472-byte packets, but also 1398, 1276, 1204, 1018, 1012, all the way down to 2 bytes.

[a1k0n]

Well... I recompiled gqrx myself with a debug build, and the problem went away. The UDP packets are smaller, and nothing is getting dropped, everything is perfectly in sync now.

Do you know what version of GNU radio was used when compiling the official release?

[csete]

This is useful info, thanks.

I haven't updated gnuradio in the app bundle for a while because it's a painful process. The app is shipping with version 3.7.8.1 (File manager -> right click -> show package contents -> Contents -> libs). I guess now I have a good reason for upgrading.

Which version are you using?

[a1k0n]

I'm using 3.7.9.1 via homebrew. But I wouldn't bother upgrading until I (or someone) can find a root cause here. This is still unexplained, unless there was a bug in the udp sender in 3.7.8.1, but that hasn't changed in years. It could also be a buffer size change in gr-osmosdr or something. Let me try downgrading and see if I can reproduce this with my own build.

On Fri, Apr 15, 2016 at 10:12 AM Alexandru Csete notifications@github.com wrote:

That useful info, thanks.

I haven't updated gnuradio in the app bundle for a while because it's a painful process. The app is shipping with version 3.7.8.1 (File manager -> right click -> show package contents -> Contents -> libs). I guess now I have a good reason for upgrading.

Which version are you using?

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/csete/gqrx/issues/363#issuecomment-210550768

[a1k0n]

Well, it seems that downgrading gnuradio to 3.7.8.1 and recompiling reproduces lost UDP packets (or at least lost UDP data) again. Haven't figured out exactly what's wrong, but upgrading seems to cure it, so I'm happy with that now.

[a1k0n]

Hm, I may have spoke too soon. Why was it working fine before?!?!

[a1k0n]

I've narrowed it down -- it's definitely dropped UDP packets at the OS level. I just did a test in which gqrx sent 1153484 bytes over the udp socket (tallied up in the return value of sendto calls) and at the receiver I got 1146532 of them. However, every single packet showed up in a local tcpdump, and adding up the packet sizes yields 1153484. So the receiver didn't get them, but the sender definitely sent them. So they were dropped by the OS, even though the receive buffer size is large enough for two seconds of 48kHz 16-bit samples.

If I lower the IQ sample rate of my SDR, it loses more packets. And if I raise the sample rate, it loses fewer. And it's because the packet sizes get bigger with lower sample rates, since you get more 48kHz samples in a buffer at lower input sample rates. And at higher sample rates, you get smaller packet sizes.

It turns out OS X is dropping the packets, even to localhost, above a certain size. If I change the UDP buffer size to 512 (down from the default of 1472), everything comes in perfectly.

Anyway, this is not really a gqrx bug; it's not really even a GNU radio bug.

But a workaround is to lower the default max UDP packet size, like this:

diff --git a/src/interfaces/udp_sink_f.cpp b/src/interfaces/udp_sink_f.cpp
index adda667..5538e9a 100644
--- a/src/interfaces/udp_sink_f.cpp
+++ b/src/interfaces/udp_sink_f.cpp
@@ -50,7 +50,7 @@ udp_sink_f::udp_sink_f()
 {

     d_f2s = gr::blocks::float_to_short::make(1, 32767);
-    d_sink = gr::blocks::udp_sink::make(sizeof(short), "localhost", 7355);
+    d_sink = gr::blocks::udp_sink::make(sizeof(short), "localhost", 7355, 512);
     d_sink->disconnect();

     connect(self(), 0, d_f2s, 0);

I'd rather send the data stream over a unix domain socket or TCP, but this makes UDP work more reliably for me.

[csete]

Thanks for investigating this. I think this is a acceptable solution to the problem.

I don't disagree with respect to using TCP, but I think that is a different matter. We can't just replace the UDP sink with a TCP sink, it would have to be added as an option.

[a1k0n]

Fixed by #367

[argilo]

As noted in https://github.com/gqrx-sdr/gqrx/pull/1347#issuecomment-2062945379, the problem is not the OS, but rather that some versions of netcat truncate UDP packets to at most 1024 bytes.