Streaming 4K H264 video through Wireguard VPN

[RobertWn16]

Hello,

I have been trying for some time to transmit H264 packets via uvgRTP. I don't need RTP_FORMAT_H264 because NVENC gives me the packets with all the necessary information, that's why I use the Generic interface. I have problems with packet loss even if the size offered by uvgRTP corresponds to that of the sender. But when I save the same scene in a file, for example, the final sizes of the two files differ, which makes me think that there is a limitation that I don't understand yet. I also set a longer length of the UDP buffer, but without success.

[jrsnen]

Hello.

What do you mean by packet loss in this case?

You should use RTP_FORMAT_H264, since that determines the packetization format inside uvgRTP. If NVENC outputs packets without start codes (based on other issues I've read, it probably does), you should add RTP_NO_H26X_SCL flag to push_frame RTP flags.

Does this help?

BR, Joni

[RobertWn16]

Hello. I initially used RTP_NO_H26X_SCL but NVENC puts the start codes in each packet, the same problem anyway. Don't get me wrong, in the local network everything goes perfectly 4k 60 without delays, but when I tried to use, for example, an ssh tunnel like Wireguard to connect to another computer and capture the packets, I have a loss of about 80%. For example, I tried a small test on the host computer and save the packets on a file (10 MB) and at the same time I sent them to another computer, the total file on remote measures around 2 MB. I thought that it could be from which band the video consumes per second, but without success, at 10 Mbps, the loss is still about 80%. At the beginning, a larger package is indeed sent that contains information about the codec, etc., but otherwise the average size of a package measures a maximum of 10,000 bytes in the rarest cases (very fast movements). This problem of packet loss leads to the appearance of artifacts because if packets are unknowingly lost, the following I-frames lose their reference. I tested today and maybe Wireguard has a problem, but everything seems to be fine, the measured latency is 6 ms to the remote computer. I will test today in more detail to see if Wireguard is actually a problem. I followed both transmission and reception examples, I don't think there is an implementation error since everything works perfectly in the local network. I just wanted to let you know that maybe someone else has encountered the same problem.

With respects, Gheorghita Robert

[jrsnen]

Ok, if NVENC puts the start codes (001 o r 0001) in front of the NAL units, then the flag should not be used as the default configuration looks for start codes and removes them. At the receiving end, uvgRTP adds the start codes back in and if this is undesirable for some reason, this functionality can be disabled with RCE_NO_H26X_PREPEND_SC flag at the receiving end.

RTP_FORMAT_H264 flag also includes fragmentation, which is needed for most transmission over the internet (a similar functionality is achieved by RCE_FRAGMENT_GENERIC, but RTP_FORMAT_H264 is the correct solution from specification perspective).

Currently, transmitting data over a VPN requires lowering the MTU size, because the default is too large. Does calling sender->configure_ctx(RCC_MTU_SIZE, 1400); fix the packet loss?

BR, Joni Räsänen

[RobertWn16]

Changing MTU_SIZE to a lower value (1350 - 1400) brought significant improvements, the artifacts not being as aggressive, for that a big thank you.

Compare with Parsec, uvgRTP is an aproppiate solution for Remote Desktop (Wireguard both).

Instead, according to the rules, I used RTP_FORMAT_H264, but at 40 - 50 frames uvgRTP displays a warning ([uvgRTP][WARNING][::garbage_collect_lost_frames] Found an old frame that has not been completed. Ts: 687483914, Seq: 53151 < -> 53158, received/expected: 7/8), which leads me to think of a possible mistake in my implementation.

Sender side:

uvgrtp::context ctx; uvgrtp::session sess = ctx.create_session("10.8.0.7"); uvgrtp::media_stream stream = sess->create_stream(8081, RTP_FORMAT_H264, RCE_SEND_ONLY); stream->configure_ctx(RCC_FPS_NUMERATOR, 60); stream->configure_ctx(RCC_UDP_SND_BUF_SIZE, 40 1024 1024) stream->configure_ctx(RCC_MTU_SIZE, 1400); //stream->configure_ctx(RCC_PKT_MAX_DELAY, 16); //stream->configure_ctx(RCE_PACE_FRAGMENT_SENDING, 60); //stream->configure_ctx(RCC_SESSION_BANDWIDTH, 40000);

__stream->push_frame((uint8_t*)pck_desc.data, pck_desc.size, RTP_NO_H26X_SCL);__ Where pck_desc is a structure where I got the packets once every 16 ms. I suspect that this is the problem, maybe the packets cannot be sent in a time less than or equal to 16 ms, probably in the next cycle I overwrite pck_desc.data over something that is already being processed. In the reception part, as in the example, I use a hook. The question is, there any method in which I can be notified back if the frame was sent successfully?

With respects, Gheorghita Robert

[jrsnen]

I'm glad to hear it helped.

That warning is an indication that a packet was lost somewhere. This can be due to the network losing it, but it can also be something that can be improved with some changes to uvgRTP usage. The sending side is very straightforward and is unlikely to be the cause of frame loss, but the receiving side is not and can sometimes lose packets in high performance situations.

• You can try to add RCE_PACE_FRAGMENT_SENDING to sending side RCE flags with | (not configure_ctx), which may improve the receiver performance
• I recommend not setting RCC_UDP_SND_BUF_SIZE unless everything else has been tried, since it is only needed in extreme cases (if at all).
• You can try increasing the RCC_RING_BUFFER_SIZE at receiver (default is 4 MB, set as number of bytes, try 40 MB)
• You can also try without RCC_FPS_NUMERATOR (and RCE_PACE_FRAGMENT_SENDING), since in its current state it may not always be reliable and cause problems
• You can also try setting the MTU size slightly higher, as this will improve the performance of receiver (fewer packets)

In my own tests, a packet is lost occasionally, but nothing close to every 50 frames.

The push_frame does not utilize threads, so it has sent everything by the time the function returns.

BR, Joni Räsänen

[RobertWn16]

The changes above brought a small improvement, however the major problem was with Wireguard, because when it received larger buffers (P-frames) it ignored them and the artifacts comes from here. The best solution would be for 2 remote computers to connect to each other directly without VPN, probably through a STUN server. For those who want to use uvgRTP via VPN, change the MTU to a lower value solves many of the problems. Thanks again for the advice.

With respects, Gheorghita Robert

[jrsnen]

The VPN situation is documented in docs folder readme.

STUN server is used to figure out your address translation in NAT (for example, google offers free servers for this). You can also try to guess your outer address/port if you know your routers address, because usually the router keeps the port the same if possible.

In other words, you should be able to stream video directly even from behind a NAT if you just use your router address in the sending end as destination and send data in the other direction as well from the receiver. For sending dummy data in the other direction, uvgRTP has a keep-alive flag (use the API which has both send and receive addresses).

BR, Joni

[wowaser]

@RobertWn16 Just out of curiosity, what is the encoding/decoding latency that you get for a 4K video with NVENC?

[RobertWn16]

@wowaser In RTX 3060 scenario:

• 7-8 ms encoding 8 bit-depth, 10-12 ms encoding 10 bit-depth(HDR10).
• 3-4 ms decoding 8 bit-depth, 11-13 ms decoding 10 bit-depth including PQ transfer function which increase the delay with 5-6 ms.

In Qudaro P620 scenario:

• encoding latency is high(around 20 - 25 ms).
• 7-8 ms decoding 8 bit-depth, 15-19 ms decoding 10 bit-depth(HDR10).

I've tested with Desktop Duplication API , 3840x2160 at 60 fps. I dont know what will happens for 120 Hz, 165Hz etc.