5G-MAG / rt-mbms-modem

MBMS - Modem
https://www.5g-mag.com/5gbroadcast
GNU Affero General Public License v3.0
13 stars 12 forks source link

Missing packets on the output of rp / modem process #32

Open m-bures opened 2 years ago

m-bures commented 2 years ago

Since we have started testing Obeca receiver we see occasional glitches in the video. Sometimes these glitches are more frequent and sometimes they are quite hard to observe at all but we cannot get rid of them completely.

At first we didn’t know where they are coming from / where in the whole broadcast – reception chain they originate.

We use Ts over IP with HEVC encoding. We suspected video player vlc that there might be a problem with HEVC decoding so we tried many different settings and releases, we even tried to use MPEG-4, reduce number of services in the TS but any settings or change in configuration didn’t help.

If MPEG-4 or 25 frame rate is used then the visibility (manifestation) of glitches can be (quite significantly) reduced but this does not solve the nature of the problem.

We tested UDP and UDP/RTP. In the latter case we could see that during playback, packets missing are sometimes reported by vlc ( rtp demux warning: x packets(s) lost) which exactly corresponds to glitches in the video.

We identified that there are missing packets on the output from (rp / modem process) which is what is causing the glitches in the video. The missing packets are there even though no error „PMCH in TTI xx failed with CRC error” is reported and even with perfect signal.

We recorded RF signal so that we can play back again and again the same signal over and over.


One specific example using Wireshark to demonstrate the issue:

There is packet

2375 (Wireshark Nr.) with rtp seq 63794 and -ip-id-48519

so you would expect that the next received packet will be

2376 (Wireshark) with rtp seq 63795 and -ip-id-48520

but instead of above mentioned we got packet

2376 (Wireshark) with rtp seq 63798 and -ip-id-48523

Which means that packets with rtp seq 63795, 63796, 63797 are missing.

The number of missing consecutive packets is typically 1 – 3.

If we replay the RF recording the missing packets are different and the packets that were missing in the previous play-back we could find – see packet 6468 (Wireshark) with rtp-seq 63795-ip-id-48520.

We do not see any dropped packets by Kernel, for example using „netstat –ic“. The packets seems to be missing on the output.

We have two BladeRF xA4 and one HackRF. There are not many missing packets, the rate compared to total number is of received packets is usually very low – see attached pictures so sometimes it is quite difficult to spot the glitch in video.

With HackRF we have experience that if we increase signal level above level which corresponds to maximal values of CINR then the number of glitches is lowered.

In other words maximum value of CINR doesn’t correspond to minimal occurrence of glitches (missing Packets) with HackRF. With BladeRF the increase of signal level doesn’t seem to lead to reduction of glitches.

Does anybody else encounter similar behaviour @dsilhavy @ben-EDT ? Is there any solution to this @kuehnhammer ?

It might be related to SDR used, but we see similar behavior with all three different SDRs that we have. We still do not have Lime SDR with which I understand the most of the testing was performed.

Thanks for any ideas or suggestions. Michal


vlc-missing-packets-Screenshot from 2022-03-18 09-29-49 Obeca_Missing_packets_on_the_output_overview_(2nd-playback-different-packets-missing) Obeca_Missing_packets_on_the_output_overview Wireshark-packet-6468(rtp-seq63795-ip-id-48520)-2nd-playback(in-the-1st-playback-this-packet-is-missing) Wireshark-packet-2376(rtp-seq63798-ip-id-48523) Wireshark-packet-2375(rtp-seq63794-ip-id-48519)

kuehnhammer commented 2 years ago

image

The deltas are interesting here as well.. packets arrive quite regularly before the red line, then there's nothing at all for almost 18ms, three packets are lost, and the next 15 or so arrive with 0ms delta. Looks like something somewhere in the chain gets blocked for a while.

m-bures commented 2 years ago

We made a few captures directly on mbms_modem_tun interface and packets are missing already there which corresponds to our previous findings that we do not see any dropped packets on interfaces.

We observe the same behavior with bladeRF and HackRF. With HackRF we can usually see lower number of missing packets (and subsequent glitches in the video).

The packets are missing on the output of the rp / modem process.

Obeca_Missing_packets_on_the_output_TUN_interface_overview5_hackRF Obeca_Missing_packets_on_the_output_TUN_interface_overview2_bladeRF Obeca_Missing_packets_on_the_output_TUN_interface_overview3_bladeRF Obeca_Missing_packets_on_the_output_TUN_interface_overview3_bladeRF(2) Obeca_Missing_packets_on_the_output_TUN_interface_overview5_hackRF(2)