Linux network is unstable.

JoonikChoi commented 1 year ago

Hello. My program send the cv::Matrix data from the camera through the datachannel.

I set up dual boot (window 10,ubuntu22.04) on one laptop. The results of running my program on Windows 10 and measuring it with 'chrome://webrtc-internals' are as follows.

The results of the boot to uuntu22.04 are as follows.

The message cannot be transmitted stably. I tried adjusting kernel parameters related to UDP, but it didn't do much good.

I thought Ubuntu was a problem, so when I tested it on Fedora, the results are as follows.

Fedora is not much different from Ubuntu. In Web page(recevier), frames reception stops periodically, as in the graph above.

The results of the test using iperf3 are fine. Senders and Receivers use the same WiFi.

Why is it stable on Windows and unstable on Linux?

paullouisageneau commented 1 year ago

Could you please be more specific? What kernel parameters did you try to adjust? What are the iperf3 results? Also, have you tried over ethernet?

JoonikChoi commented 1 year ago

oh, I said it wrong. I'm using Ethernet named 'enp2s0'.

Kernel parameters that I set are follow as :

ifconfig enp2s0 mtu 9000
sudo sysctl -w net.core.wmem_max=16777216

In addition to this, I applied various values to net.core.wmem_max.

I test network using iperf3. iperf3 server and client is different pc I used this command.

# server
iperf3 -s 192.168.0.35
# client
iperf3 -b0 -c 192.168.0.35 -t 30

test result is this.

[  5] local 192.168.0.70 port 54564 connected to 192.168.0.35 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  50.1 MBytes   420 Mbits/sec    0   3.26 MBytes       
[  5]   1.00-2.00   sec  42.5 MBytes   357 Mbits/sec    1   3.79 MBytes       
[  5]   2.00-3.00   sec  37.5 MBytes   315 Mbits/sec    0   3.79 MBytes       
[  5]   3.00-4.00   sec  36.2 MBytes   304 Mbits/sec    0   3.98 MBytes       
[  5]   4.00-5.00   sec  33.8 MBytes   283 Mbits/sec    0   3.98 MBytes       
[  5]   5.00-6.00   sec  35.0 MBytes   294 Mbits/sec    0   3.98 MBytes       
[  5]   6.00-7.00   sec  36.2 MBytes   304 Mbits/sec    0   3.98 MBytes       
[  5]   7.00-8.00   sec  35.0 MBytes   294 Mbits/sec    1   3.98 MBytes       
[  5]   8.00-9.00   sec  35.0 MBytes   294 Mbits/sec    1   2.96 MBytes       
[  5]   9.00-10.00  sec  36.2 MBytes   304 Mbits/sec    0   3.19 MBytes       
[  5]  10.00-11.00  sec  35.0 MBytes   294 Mbits/sec    0   3.36 MBytes       
[  5]  11.00-12.00  sec  37.5 MBytes   315 Mbits/sec    0   3.54 MBytes       
[  5]  12.00-13.00  sec  35.0 MBytes   294 Mbits/sec    0   3.65 MBytes       
[  5]  13.00-14.00  sec  38.8 MBytes   325 Mbits/sec    0   3.75 MBytes       
[  5]  14.00-15.00  sec  36.2 MBytes   304 Mbits/sec    0   3.80 MBytes       
[  5]  15.00-16.00  sec  35.0 MBytes   294 Mbits/sec    1   3.98 MBytes       
[  5]  16.00-17.00  sec  35.0 MBytes   294 Mbits/sec    0   3.98 MBytes       
[  5]  17.00-18.00  sec  31.2 MBytes   262 Mbits/sec    0   3.98 MBytes       
[  5]  18.00-19.00  sec  33.8 MBytes   283 Mbits/sec    0   3.98 MBytes       
[  5]  19.00-20.00  sec  33.8 MBytes   283 Mbits/sec    0   3.98 MBytes       
[  5]  20.00-21.00  sec  40.0 MBytes   336 Mbits/sec    0   3.98 MBytes       
[  5]  21.00-22.00  sec  36.2 MBytes   304 Mbits/sec    0   3.98 MBytes       
[  5]  22.00-23.00  sec  37.5 MBytes   315 Mbits/sec    0   3.98 MBytes       
[  5]  23.00-24.00  sec  36.2 MBytes   304 Mbits/sec    0   3.98 MBytes       
[  5]  24.00-25.00  sec  38.8 MBytes   325 Mbits/sec    0   3.98 MBytes       
[  5]  25.00-26.00  sec  33.8 MBytes   283 Mbits/sec    1   3.98 MBytes       
[  5]  26.00-27.00  sec  38.8 MBytes   325 Mbits/sec    0   3.98 MBytes       
[  5]  27.00-28.00  sec  35.0 MBytes   294 Mbits/sec    0   3.98 MBytes       
[  5]  28.00-29.00  sec  36.2 MBytes   304 Mbits/sec    0   3.98 MBytes       
[  5]  29.00-30.00  sec  35.0 MBytes   294 Mbits/sec    0   3.98 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  1.07 GBytes   307 Mbits/sec    5             sender
[  5]   0.00-30.05  sec  1.07 GBytes   306 Mbits/sec                  receiver

JoonikChoi commented 1 year ago

This is rough code about main routine of my program.

As I said, program that compiled below code is works well in windows. But on linux based PC, throughtput is unstable.


cv::Mat greenMatrix(720,1280,CV_8UC3, cv::Scalar(0,255,0));
std::vector<unsigned char> byte_greenMat;
cv::imencode(".jpg", greenMatrix, byte_greenMat);

while (true)
{
if (!dc->isOpen()) continue;
if (dc->bufferedAmount() != 0) {
     std::cout << "bufferedAmount : " << dc->bufferedAmount() << std::endl;
     continue;
};

dc->send((std::byte*)(byte_greenMat.data()), byte_greenMat.size());
usleep(20 * 1000);
}

If you test this rough code, you'll get the results.

I want to use this program in linux based PC.

(Note. Throughput is unstable even if the bufferedAmount message does not occur.)

paullouisageneau commented 1 year ago

How is this different from the very similar issue you already reported in https://github.com/paullouisageneau/libdatachannel/issues/940 ?

oh, I said it wrong. I'm using Ethernet named 'enp2s0'.

So is the link between devices Wifi or Ethernet?

Kernel parameters that I set are follow as :
ifconfig enp2s0 mtu 9000
sudo sysctl -w net.core.wmem_max=16777216
Don't change the link MTU except if you really know what you are doing, this is likely to cause issues with some applications.

Setting the send buffer to 1MB should be sufficient, if it doesn't change the behavior then the issue is likely unrelated:

$ sudo sysctl -w net.core.wmem_max=1048576

This is rough code about main routine of my program.

As I said, program that compiled below code is works well in windows. But on linux based PC, throughtput is unstable.

cv::Mat greenMatrix(720,1280,CV_8UC3, cv::Scalar(0,255,0));
std::vector<unsigned char> byte_greenMat;
cv::imencode(".jpg", greenMatrix, byte_greenMat);

while (true)
{
if (!dc->isOpen()) continue;
if (dc->bufferedAmount() != 0) {
     std::cout << "bufferedAmount : " << dc->bufferedAmount() << std::endl;
     continue;
};

dc->send((std::byte*)(byte_greenMat.data()), byte_greenMat.size());
usleep(20 * 1000);
}

Even if it might be unrelated, there are issues with this code:

It looks like it encodes a solid green image as JPEG, which must therefore be quite small, and sends the same image every 20ms. What is the resulting message size?
The code spins in an infinite loop while the data channel is not open or messages are buffered. You should not do that and instead drop or don't generate the message.

As a side note, it looks like you are trying to send real-time video to a browser in a clumsy manner. While this should work passably on a local network, it won't perform well in practice. You should use the media transport by encoding the video properly and sending it over a track.

(Note. Throughput is unstable even if the bufferedAmount message does not occur.)

Does bufferedAmount stay at 0? If so it would mean the data channel throughput is not the limiting factor.

JoonikChoi commented 1 year ago

How is this different from the very similar issue you already reported in https://github.com/paullouisageneau/libdatachannel/issues/940 ?

I was mistaken that #940 was solved. It is true that there was a issue with the Wi-Fi driver, but the throughput is still unstable.

So is the link between devices Wifi or Ethernet?

They are connected by Wifi.

Don't change the link MTU except if you really know what you are doing, this is likely to cause issues with some applications. Setting the send buffer to 1MB should be sufficient, if it doesn't change the behavior then the issue is likely unrelated:

I understood, but it doesn't change. As you said, maybe it is unrelated about kernel parameters.

It looks like it encodes a solid green image as JPEG, which must therefore be quite small, and sends the same image every 20ms. What is the resulting message size?

message size is 15,000 Byte. Originally, my code is to send an image from a camera, but I thought the size of the message was the cause, so I did this to test it by dramatically reducing the size.

The code spins in an infinite loop while the data channel is not open or messages are buffered. You should not do that and instead drop or don't generate the message.

How do I drop messages that are already in the sending buffer via dc->send()?

As a side note, it looks like you are trying to send real-time video to a browser in a clumsy manner. While this should work passably on a local network, it won't perform well in practice. You should use the media transport by encoding the video properly and sending it over a track. Does bufferedAmount stay at 0? If so it would mean the data channel throughput is not the limiting factor.

Yes, that is stay at 0. I have confirmed that it works normally when tested with two devices in different LAN areas(only if the transmitter side is Windows).

I know that it is strange and clumsy manner. I already use the media transport by track and it's works well. but I want to know why Windows and Linux of the same PC(laptop) have this difference in performance.

(Note. Additionally, I wanted to test if the lossless compressed(ex. png) image could be transmitted to the data channel. As I said, on Windows PC, it works well.)

paullouisageneau commented 11 months ago

How do I drop messages that are already in the sending buffer via dc->send()?

You can't explicitly drop messages once they are in the buffer, but you can set the unreliable mode with maxPacketLifeTime (or the legacy Reliability::Type::Timed in previous versions) so the SCTP stack drops packets after the given duration.

paullouisageneau / libdatachannel

Linux network is unstable. #989