Seemingly poor performance over bridged LAN using two ZTe units

[cferrey]

I have two ZT Edge units running on remote LANs in a layer 2 bridged setup, with the same subnet scheme applied to the LAN at each site. Site A's WAN connection is 940Mbps down / 880Mbps up, and Site B's is 370Mbps down / 330 Mbps up. Both ZTe devices are hardwired to their respective routers via gigabit ethernet over Cat6e cable.

Using iperf3, I'm getting an average transfer speed of ~20Mbps between the two units. Is this in line with internal tests by ZeroTier? I understand that the encryption has some CPU performance overhead, but I was expecting to see better performance than this given that my limiting WAN speed on the slowest connection is 330Mbps (upload from Site B). Even this most limited speed is almost 20x faster than what I'm getting via the bridged ZTe link.

[genieinfo]

I'm in the same situation here : i was never able to make my 2 edges run their bridge over 30Mb/s, where x86 multicore based machines with latest ZT binaries installed would perform 10x better on average (250 to 300Mb/s) on the same WAN links.

I understand you stopped development for these devices. As it has never been able to update by itself, saying 0.5.0 is the most recent version even though it seems it was released in nov. 2018, would you please allow sudo and/or permit installing other distro on these quite capable Marvell-based devices ? Many thanks in advance if you do so

[cferrey]

@genieinfo -- thanks for checking in; glad to hear it's not just my setup. @adamierymenko did comment on this in a different thread (unfortunately couldn't find it), noting that this performance gap may be related to the ARM processor architecture in the Marvell ESPRESSObin SBCs they used for the ZT Edge production units . Something about instruction set translation from the processor types ZT was written for (ARM vs. x86? 32bit vs. 64bit? Can't remember...). Despite occasionally pretending I know stuff about networking, I'm super uninformed on the hardware front -- so you'd need to follow up with them directly for more info there.

Agree though that it would be great to hear about a possible OS update for the Edge units and whether it'd have any potential to improve performance -- though sounds like it may've been something fundamentally limited by the hardware choice. Annoying with a paid crowdfunding hardware effort -- for sure -- but honestly, given the EXTREMELY generous free tier that ZT offers (and knowing they run a relatively small shop), we'd be remiss to complain here. I was happy to support them regardless.

All that said -- throwing up a gentle bat-signal reminder for @adamierymenko here, in case there are any updates on this front (potentially related to v2.0?). Many thanks!

[genieinfo]

I totally agree : ZT changed my conception of SDN and remote work in so many ways, and mostly for FREE, that i would be ashamed to blame @adamierymenko and yes, i am proud too, having backed them in the first place to support the project. In the meantime, i found this on their Atlassian wiki https://zerotier.atlassian.net/wiki/spaces/SD/pages/193134593/One+Port+Linux+Bridge The perfect hitch-hiker guide to the perfect bridged setup for a Pi ! I tried/setup/bench-marked so many different configs with so many devices to achieve a stable Zerotier-based bridging solution with a good balance between price / performance and came to the conclusion that the most adequate device was the RPi4. So happy that @Travis LaDuke wrote this guide, compiling all the useful infos we all tried to gather and re-assemble for months :)

No worries about an hypothetical Edge software upgrade, it's just me trying to re-purpose 2 tiny ARM devices :) Keep on going Zerotier gentlemen, impatiently waiting for v2.0 too !

P.S.: i'm french, so any typos corrections are welcome P.S.2: https://github.com/zerotier/ZeroTierOne/issues/914#issuecomment-591630324

[cferrey]

@genieinfo -- Super, je parle un petit peu de français, mais ton anglais est vraiment bon! Glad to hear these devices found their way to France. And yes, I also used that guide as a basis to later implement rPi 4 ZT layer-2 bridges and can get a fairly consistent 70Mbps avg between remote sites using that approach. Seems the rPis are ARM chips, so scratch my above comment about ARM CPU architecture being the (potential) hardware issue. Might've been the 32/64 angle...

[genieinfo]

With RPI4B, i get around 125-130Mb/s bi-drirectional and noticed only 2 cores were really stressed out (/htop /shows 100 + 79% CPU load) on two different WAN links measured as 1Gb/s down - 400Mb/s up in real-world scenario. There are 4 cores available on the RPI4. I don't understand why 2 cores are always unused. Anyway, for the price and extremely low power consumption of the RPI's + such simplicity of management from ZT, an /always-on/ 130Mb/s L2 bridge is still great !

Cheers

Sebastien

Le 19/09/2020 à 03:11, cferrey a écrit :

@genieinfo https://github.com/genieinfo -- Super, je parle un petit peu de français, mais ton anglais est vraiment bon! Glad to hear these devices found their way to France. And yes, I also used that guide as a basis to later implement rPi 4 ZT layer-2 bridges and can get a fairly consistent 70Mbps avg between remote sites using that approach. Seems the rPis are ARM chips, so scratch my above comment about ARM CPU architecture being the (potential) hardware issue. Might've been the 32/64 angle...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/zerotier/edge/issues/2#issuecomment-695143210, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK7VSNTYJCFFWGUQFVCCZ6LSGQAKRANCNFSM4IYHXZ5A.