Closed OrvilleQ closed 5 months ago
Hi @OrvilleQ
I am currently not at home which has 80 MB/s (rx) and 30 MB/s (tx), a big difference if we compare the ADSL link where I am.
Excerpt from net/speedtest-go:
# speedtest-go
...
✓ Latency: 42.665045ms Jitter: 117.528µs Min: 42.406088ms Max: 42.883802ms
✓ Download: 4.51Mbps (used: 5.38MB) (latency: 61ms jitter: 37ms min: 48ms max: 205ms)
✓ Upload: 0.76Mbps (used: 0.91MB) (latency: 73ms jitter: 21ms min: 47ms max: 139ms)
But if we compare the result of speedtest-go executed in a jail:
# appjail jail list -j speedtest-go
STATUS NAME TYPE VERSION PORTS NETWORK_IP4
UP speedtest-go thin 13.2-RELEASE - 10.0.0.5
# appjail cmd jexec speedtest-go speedtest-go
...
✓ Latency: 31.867667ms Jitter: 845.172µs Min: 31.07015ms Max: 33.496017ms
✓ Download: 4.87Mbps (used: 5.80MB) (latency: 87ms jitter: 111ms min: 35ms max: 451ms)
✓ Upload: 0.76Mbps (used: 0.91MB) (latency: 91ms jitter: 63ms min: 51ms max: 250ms)
It's even better, but this doesn't mean that the performance in jails is better, it's probably just better because speedtest-go
used two different servers. Although there is an overhead when using virtual networks, it is actually so small that you should not notice any difference. Of course, in your case there is a problem since you notice a performance loss. Check the bridge's MTU to confirm if it matches the host's MTU currently used:
# appjail network list
NAME NETWORK CIDR BROADCAST GATEWAY MINADDR MAXADDR ADDRESSES DESCRIPTION MTU
ajnet 10.0.0.0 10 10.63.255.255 10.0.0.1 10.0.0.1 10.63.255.254 4194302 AppJail network 576
# ifconfig jext | grep mtu
jext: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 576
# grep MTU /usr/local/etc/appjail/appjail.conf
DEFAULT_VIRTUALNET_MTU=576
# appjail version
3.2.0+15f7db9c07e005d852119b2346b93b9900d4e023
Another recommendation is to try to minimize configuration overhead:
GENERIC
kernel instead of a custom one as it may or may not affect network performance in jails, but if this happens it is probably a bug. Note: I'm just trying to deduce the cause, not conclude that a custom kernel can affect its performance.# cat /etc/pf.conf
nat-anchor 'appjail-nat/jail/*'
nat-anchor "appjail-nat/network/*"
rdr-anchor "appjail-rdr/*"
This is what AppJail requires to work when using Virtual Networks, no more, no less.
Bridges and Virtual Networks use if_bridge(4) and if_epair(4) drivers, and most jail frameworks and users use the same drivers for jail networking, so if you notice a performance overhead, this will affect to many people unless it is just a problem in your system.
I hope the above can help you, tell me any results you find.
@DtxdF Thank you for your detailed instructions. I'll log the things I did down below in case I miss something.
appjail network list
did not show MTU, maybe because I'm using the release version. However, the configuration file says MTU defaults to 1500, which is the same as my EX_IF interface vtnet0. ifconfig ajnet
also shows the MTU is 1500, so I guess MTU is not the problem.
root@fsn00:~/Kernel # ifconfig vtnet0 | grep mtu
vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
root@fsn00:~ # ifconfig ajnet
ajnet: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
The pf.conf file is as minimal as possible already.
root@fsn00:~ # cat /etc/pf.conf
nat-anchor "appjail-nat/jail/*"
nat-anchor "appjail-nat/network/*"
rdr-anchor "appjail-rdr/*"
Test the network speed using speedtest-go inside and outside the jail. The upload performance inside the jail is 40% slower compared to outside the jail, but it is still fast enough.
Inside the jail:
✓ Test Server: [41175] 96.60km Dresden (Germany) by CITUX GmbH
✓ Latency: 663.682µs Jitter: 177.098µs Min: 444.743µs Max: 1.066988ms
✓ Download: 19.49Mbps (used: 23.23MB) (latency: 0ms jitter: 1ms min: 0ms max: 5ms)
✓ Upload: 1659.30Mbps (used: 1978.04MB) (latency: 2ms jitter: 2ms min: 0ms max: 11ms)
Outide the jail:
✓ Test Server: [41175] 96.60km Dresden (Germany) by CITUX GmbH
✓ Latency: 923.864µs Jitter: 1.263447ms Min: 439.323µs Max: 4.709956ms
✓ Download: 6298.97Mbps (used: 7508.96MB) (latency: 1ms jitter: 0ms min: 0ms max: 3ms)
✓ Upload: 2853.11Mbps (used: 3401.18MB) (latency: 1ms jitter: 1ms min: 0ms max: 8ms)
The inner connection inside ajnet seems to be good, get a reasonable result using iperf3.
root@packager:~ # iperf3 -c test
Connecting to host test, port 5201
[ 5] local 10.0.0.2 port 56325 connected to 10.0.0.3 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 505 MBytes 4.24 Gbits/sec 0 1.61 MBytes
[ 5] 1.00-2.00 sec 511 MBytes 4.28 Gbits/sec 0 1.61 MBytes
[ 5] 2.00-3.05 sec 538 MBytes 4.30 Gbits/sec 1 864 KBytes
[ 5] 3.05-4.04 sec 508 MBytes 4.33 Gbits/sec 0 864 KBytes
[ 5] 4.04-5.00 sec 488 MBytes 4.24 Gbits/sec 0 864 KBytes
[ 5] 5.00-6.01 sec 499 MBytes 4.14 Gbits/sec 0 1.05 MBytes
[ 5] 6.01-7.00 sec 500 MBytes 4.23 Gbits/sec 0 1.05 MBytes
[ 5] 7.00-8.00 sec 520 MBytes 4.37 Gbits/sec 0 1.05 MBytes
[ 5] 8.00-9.00 sec 530 MBytes 4.44 Gbits/sec 0 1.05 MBytes
[ 5] 9.00-10.01 sec 528 MBytes 4.40 Gbits/sec 0 1.05 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 5.01 GBytes 4.30 Gbits/sec 1 sender
[ 5] 0.00-10.01 sec 5.01 GBytes 4.30 Gbits/sec receiver
iperf Done.
After compiling the Kernel to GENERIC with BBR enabled, nothing changed. I'll try disabling BBR next.
root@fsn00:~ # sysctl net.inet.tcp.functions_default
net.inet.tcp.functions_default: freebsd
✓ Test Server: [41175] 96.60km Dresden (Germany) by CITUX GmbH
✓ Latency: 696.362µs Jitter: 239.203µs Min: 398.222µs Max: 1.077307ms
✓ Download: 19.88Mbps (used: 23.70MB) (latency: 0ms jitter: 0ms min: 0ms max: 2ms)
✓ Upload: 1634.93Mbps (used: 1948.99MB) (latency: 2ms jitter: 1ms min: 0ms max: 8ms)
After BBR disabled still nothing changed.
Edit: I was using appjail quick packager virtualnet=":packager default"
to create the jail.
On a fresh Arm64 FreeBSD environment update to the latest patch (14.0-RELEASE-p6) with only appjail installed, using ZFS, EXT_IF and ON_IF set to vtnet0. pf configured.
set up a test jail with appjail quick test virtualnet=":test default" nat overwrite start
, the speedtest-go result is still extremely slow on download speed.
✓ Test Server: [41175] 60.00km Dresden (Germany) by CITUX GmbH
✓ Latency: 767.364µs Jitter: 442.745µs Min: 390.143µs Max: 2.030396ms
✓ Download: 19.89Mbps (used: 23.71MB) (latency: 1ms jitter: 1ms min: 0ms max: 4ms)
✓ Upload: 1310.75Mbps (used: 1562.54MB) (latency: 3ms jitter: 1ms min: 0ms max: 7ms)
Thanks for telling me about the driver you use, vtnet, this helped me dig deeper into your problem, at least to investigate further on Bugzilla:
The above is the most interesting, as it seems to be related to your problem, but look at the full search list:
The inner connection inside ajnet seems to be good, get a reasonable result using iperf3.
Yes, this appears to be what was reported in the issue, since you are testing on the local system, packets don't travel outside of your interface.
0.3 Mbit/s download while upload seems to be unaffected
This is a quote from the bug, comment 0. As speedtest-go informs you, Download seems to be affected, while Upload is only affected a bit. Of course, see the full bug, as the developer says some things that are relevant to investigate further.
You haven't done anything wrong, this seems to be a bug between vtnet and the NAT implementation or something related. Of course this is not an AppJail issue, but I suggest the following:
You are right.
After read all the comments and found a post back in 2016 described similar issue, I set hw.vtnet.csum_disable="1"
in /boot/loader.conf, and everything works as expected now.
inside the jail:
✓ Test Server: [41175] 96.60km Dresden (Germany) by CITUX GmbH
✓ Latency: 578.274µs Jitter: 410.754µs Min: 364.603µs Max: 1.790574ms
✓ Download: 3212.81Mbps (used: 3829.97MB) (latency: 2ms jitter: 1ms min: 0ms max: 4ms)
✓ Upload: 2108.45Mbps (used: 2513.46MB) (latency: 1ms jitter: 0ms min: 0ms max: 3ms)
outside the jail:
✓ Test Server: [41175] 96.60km Dresden (Germany) by CITUX GmbH
✓ Latency: 555.28µs Jitter: 78.684µs Min: 418.683µs Max: 690.805µs
✓ Download: 3489.01Mbps (used: 4159.22MB) (latency: 2ms jitter: 1ms min: 1ms max: 5ms)
✓ Upload: 1755.26Mbps (used: 2092.44MB) (latency: 2ms jitter: 1ms min: 1ms max: 6ms)
But that means CPU now has to do all the checksum job and It could easily use around 50% CPU resource with maximum speed. Not a perfect solution but enough for now. I'll keep my eye on the bug fix in the future.
Once again, thank you sincerely for the help.
I've create a pull request to warn this issue in document. https://github.com/DtxdF/AppJail.docs/pull/2
Thanks for the workaround and for the PR on the other repo! I'll merge it as soon as possible.
I'll keep this in mind when writing the appjail-network(1)
man page.
Just created a server on my Hetzner server located at Germany. I self built the kernel to enable bbr and it's now on 14.0-RELEASE-p6.
I followed your document and set pf, virtual networks and dns up, then I create a jail with
appjail quick packager virtualnet=":packager"
(nat is enabled on the virtualnet), but when I download ports using gitup I notice extremely slow network speed.Here's the result if I fetch a 10gb test file outside of jail:
And here's the result inside the jail:
It's 1500x times slower.
I'm new to FreeBSD, so I'm not sure where to start with debugging and digging useful information. If there are some specific details required, please inform me, and I will provide them as promptly as possible.