search

webbbn / wifibroadcast_bridge

A UDP bridge over a wifibroadcast link
GNU Lesser General Public License v2.1
6 stars 4 forks source link

Tipp #2

Open Consti10 opened 3 years ago

Consti10 commented 3 years ago

Hi, Just wanted to let you know that I've been busy writing a new version of svpcom/wifibroadcast for OpenHD. https://github.com/Consti10/wifibroadcast One really easy step to reduce the latency I found when doing so is to enable pcap immediate mode. Pcap sneaked in "not using immediate mode by default" with some release ( i think it was 1.5).

Consti10 commented 3 years ago

I am not sure how you decided to reduce FEC latency - is it by using a dynamic block size and transmitting the fec k,n parameters with each block ?

webbbn commented 3 years ago

That's great! I had heard there were some issues with using pcap on the tx side, which is why I didn't use it. It would make the code much cleaner to use pcap.

The FEC latency should only be the cost of sending the FEC packets over the air. You can transmit the data blocks immediately while you're calculating the FEC blocks in parallel. On the rx side you can also pass on data blocks as soon as they arrive until you have a dropped packet, in which case you need to wait for an FEC packet to arrive before you can decode the dropped packet.

FWIW, I recently switched my external interface away from being UDP-based to using a TUN interface. Both sides of the link are simply a network that you can send UDP over, or TCP, etc. You can also use standard routing to route packets around. Internally the code parses the IP header to get the IP port and the type of packet, so you can say, send all UDP packets on port 5600 over the link with these FEC/packet type/data rate, etc parameters.

It really presents a clean interface and you can ssh over the link, scp updates, etc. I haven't tried implementing it yet, but I think it would also be pretty easy to support multiple hosts connected over the WFB link by simply brining up another pair of TUN interfaces. The only drawback that I know of is that there is currently more overhead because it has the extra IP header. I'm pretty sure that can be at least mostly eliminated by combining that with the ieee header, but I haven't tried it yet.

It also highlighted where I had latency caused by packets getting stuck in buffers. At first, pinging the other side would soon block because packets were queued up waiting to be FEC encoded, which required me to tune the flushing of packets.

Consti10 commented 3 years ago

Immediate mode only has an effect on the rx side as far as I l know. On the tx side, I tried both pcap and the "raw" approach https://github.com/Consti10/wifibroadcast/blob/master/src/RawTransmitter.hpp I couldn't measure any difference there. Regarding latency, my steps were: 1) don't use FEC and optimize until the average and max are close to 1ms. ( in the end I found out that it was only the immedate mode on rx that caused delays but took me a lot of testing to get there) 2) Go into svpcom's rx ring and optimize there, too. With svpcom's help I found a couple of bugs / not ideal design pitfals) I also looked at your fec.cc file a couple of times for inspiration.

Consti10 commented 3 years ago

That's why in the end I decided to add a flush after a timeout on both the tx and rx, too.

Consti10 commented 3 years ago

The advantage of the svpcom "udp approach" is that people can build their own stuff by just running another tx / rx instance. This way I hope to trigger more contributions to OpenHD by different people :)

Consti10 commented 3 years ago

Btw, stuff you might also find usefull: https://github.com/Consti10/wifibroadcast/blob/master/src/RadiotapHeader.hpp My c++ approach to the radiotap header

webbbn commented 3 years ago

Immediate mode only has an effect on the rx side as far as I l know. On the tx side, I tried both pcap and the "raw" approach https://github.com/Consti10/wifibroadcast/blob/master/src/RawTransmitter.hpp I couldn't measure any difference there. Regarding latency, my steps were:

  1. don't use FEC and optimize until the average and max are close to 1ms. ( in the end I found out that it was only the immedate mode on rx that caused delays but took me a lot of testing to get there)
  2. Go into svpcom's rx ring and optimize there, too. With svpcom's help I found a couple of bugs / not ideal design pitfals) I also looked at your fec.cc file a couple of times for inspiration.

I just tried your changes, and it does appear to reduce the latency by about half in my test case. I don't have my GPS disciplined ntp server running anymore, and my test case it not very controlled, but It was showing a latency of 10-20ms, and now it's showing closer to 5-10ms. That's with a data rate of 15Mbps, so it might do even better with a lower data rate.

I'm pretty sure the stated reason for not using svpcom WFB on OpenHd was due to latency in pcap on the tx side. Since my code is based on the OpenHD code, I assume that code has (had?) the same issue. I'm glad you found the real reason. Good job!

webbbn commented 3 years ago

That's why in the end I decided to add a flush after a timeout on both the tx and rx, too.

Oh, I had the timeout in there, but the flush wasn't quite making it all the way through all the pipes! :-)

webbbn commented 3 years ago

The advantage of the svpcom "udp approach" is that people can build their own stuff by just running another tx / rx instance. This way I hope to trigger more contributions to OpenHD by different people :)

Is there some way to aggregate all the statistics? That was one of the biggest drawbacks to multiple programs that I saw. I thought I remember that svpcom did aggregate them somehow, but I cant remember for sure.

In any case, adding a TUN interface to a UDP link is not hard. In fact, I think you can do that with socat out of the box. In that case, though, you lose the ability to route the packets differently.

Consti10 commented 3 years ago

Yeah, aggregating the statistics sucks. You can obviolsy use shm for that with multiple instances, but I'm hoping that Stephen from OpenHD comes up with something. Regarding the latency: 5-10ms with FEC is in line with my testing, too. What I got down with the flushing was the max / 10% maximum. Before that it was pretty bad, up to 50ms due to the big rx ring. My test utility https://github.com/Consti10/wifibroadcast/tree/master/latencyTesting gives you stuff like that in the end.

Consti10 commented 3 years ago

Note that for building it you gotta clone the whole repo though.

Consti10 commented 3 years ago

Immediate mode only has an effect on the rx side as far as I l know. On the tx side, I tried both pcap and the "raw" approach https://github.com/Consti10/wifibroadcast/blob/master/src/RawTransmitter.hpp I couldn't measure any difference there. Regarding latency, my steps were:

  1. don't use FEC and optimize until the average and max are close to 1ms. ( in the end I found out that it was only the immedate mode on rx that caused delays but took me a lot of testing to get there)
  2. Go into svpcom's rx ring and optimize there, too. With svpcom's help I found a couple of bugs / not ideal design pitfals) I also looked at your fec.cc file a couple of times for inspiration.

I just tried your changes, and it does appear to reduce the latency by about half in my test case. I don't have my GPS disciplined ntp server running anymore, and my test case it not very controlled, but It was showing a latency of 10-20ms, and now it's showing closer to 5-10ms. That's with a data rate of 15Mbps, so it might do even better with a lower data rate.

I'm pretty sure the stated reason for not using svpcom WFB on OpenHd was due to latency in pcap on the tx side. Since my code is based on the OpenHD code, I assume that code has (had?) the same issue. I'm glad you found the real reason. Good job!

Yeah were running wifibroadcast for 2 years with this unneeded latency on RX. I assume no one tested since that, and befinitiv himself probably was running a pcap version pre 1.5 .