search

geerlingguy / raspberry-pi-pcie-devices

Raspberry Pi PCI Express device compatibility database
http://pipci.jeffgeerling.com
GNU General Public License v3.0
1.61k stars 145 forks source link

Test MZHOU WiFi/Bluetooth Adapter M.2 NGFF to PCI-E 1x #38

Closed geerlingguy closed 3 years ago

geerlingguy commented 3 years ago

At @Shokman's suggestion, I ordered a MZHOU WiFi/Bluetooth Adapter M.2 NGFF to PCI-E 1x. My intention is to see if the desktop Intel AX200 adapter I bought to upgrade my HP PC's WiFi works with the Pi through this adapter board. Could be a good option instead of a dedicated PCIe card if someone already has an old M.2 A or E key card from an old laptop or desktop.

m2-mzhou-wifi-bt-ngff-to-pcie

I'll also use this card to test a Coral edge TPU in A type M.2; see https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/44

geerlingguy commented 3 years ago

Popped an Intel AX200 Desktop Kit M.2 A/E socket chip into the board. Going to see if the same patched driver I used with the EDUP adapter (see https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/22) will work.

geerlingguy commented 3 years ago 
01:00.0 Network controller: Intel Corporation Device 2723 (rev 1a)
    Subsystem: Intel Corporation Device 0080
    Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 0
    Region 0: Memory at 600000000 (64-bit, non-prefetchable) [disabled] [size=16K]
    Capabilities: [c8] Power Management version 3
        Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [40] Express (v2) Endpoint, MSI 00
        DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 unlimited
            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
        DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
            RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+ FLReset-
            MaxPayload 128 bytes, MaxReadReq 128 bytes
        DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
        LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L1, Exit Latency L0s <4us, L1 <8us
            ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR+, OBFF Via WAKE#
        DevCtl2: Completion Timeout: 16ms to 55ms, TimeoutDis-, LTR-, OBFF Disabled
        LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
             EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
    Capabilities: [80] MSI-X: Enable- Count=16 Masked-
        Vector table: BAR=0 offset=00002000
        PBA: BAR=0 offset=00003000
    Capabilities: [100 v1] Advanced Error Reporting
        UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
        CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
        AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
    Capabilities: [14c v1] Latency Tolerance Reporting
        Max snoop latency: 0ns
        Max no snoop latency: 0ns
    Capabilities: [154 v1] L1 PM Substates
        L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
              PortCommonModeRestoreTime=30us PortTPowerOnTime=18us
        L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
               T_CommonMode=0us LTR1.2_Threshold=0ns
        L1SubCtl2: T_PwrOn=10us

I can see the card. Let's see if I can get it to work!

geerlingguy commented 3 years ago

Trying to cross-compile on 5.10.y, the patch in #22 didn't apply, so I updated it... but then I ran into this issue: https://github.com/raspberrypi/linux/issues/4036 — so I'm going to stick to 5.4.y for now.

geerlingguy commented 3 years ago

I applied the patch from https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/22#issuecomment-727087296

Then I ran menuconfig and enabled:

Device Drivers
  > Network device support
    > Wireless LAN
      > Intel devices
        > Intel Wireless WiFi Next Gen AGN - Wireless-N/Advanced-N/Ultimate-N (iwlwifi) (set to 'M')
          > Intel Wireless WiFi MVM Firmware support (set to 'M')

Then I cross compiled and copied the kernel/modules. Then I installed the proper firmware from the kernel wifi site:

wget https://wireless.wiki.kernel.org/_media/en/users/drivers/iwlwifi/iwlwifi-cc-46.3cfab8da.0.tgz
tar -xvzf iwlwifi-cc-46.3cfab8da.0.tgz
cd iwlwifi-cc-46.3cfab8da.0/
sudo cp iwlwifi-*.ucode /lib/firmware
sudo reboot

The card comes up with ip address:

$ ip address
...
4: wlan1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether e0:d4:64:6b:20:7e brd ff:ff:ff:ff:ff:ff

And I created a file /etc/wpa_supplicant/wpa_supplicant.conf:

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=us

network={
  scan_ssid=1
  ssid="geerling-acn"
  psk="my-password-here"
}

(Note that geerling-acn is the name of my 5 GHz network—it's now running through a WiFi 6 router though so it's a bit of a misnomer... I should probably update the name to geerling-6 or something. It's not 6E though.)

geerlingguy commented 3 years ago

I'd like to do two performance tests:

  1. With the antennas built into this little MZHOU adapter card.
  2. With the external antenna bracket and much beefier-looking antennas Intel included in the Desktop kit.

It would be interesting to see if I can get more bandwidth (and better results) out of Intel's antenna. I know antenna design can do a lot to help with WiFi performance (as with all RF)...

geerlingguy commented 3 years ago

Hmm... after doing some preliminary testing of the 10 Gbps Ethernet adapter, I am beginning to question the results I found in #22 — Just with the built-in antennae, if I make sure to do the following, I end up getting around 500 Mbps (one-way), direct to my router:

# On my Mac SSH'ed into the router:
ssh admin@10.0.100.1  # using Merlin WRT Firmware
iperf3 -s

# On my Mac SSH'ed into the Raspberry Pi:
ssh pi@10.0.100.168  # SSH'ed into the WiFi interface (since I'm about to disable eth0).
sudo ip link set eth0 down  # Make sure the eth0 built-in interface is disabled completely
iperf3 --bind 10.0.100.168 -c 10.0.100.1

What I found with the 10G card was iperf3's --bind option doesn't seem to actually select the IP/interface you choose if it is on the same network as another interface higher in the stack, like eth0.

So my preliminary testing was giving me around 930 Mbps, which was suspicious, because I hadn't done any antenna placement optimization yet.

So when I disabled eth0 and ran the test again, I was getting more what I'd expect:

$ iperf3 --bind 10.0.100.168 -c 10.0.100.1
Connecting to host 10.0.100.1, port 5201
[  5] local 10.0.100.168 port 54059 connected to 10.0.100.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  59.4 MBytes   499 Mbits/sec    0   1.85 MBytes       
[  5]   1.00-2.00   sec  70.0 MBytes   587 Mbits/sec  1214    419 KBytes       
[  5]   2.00-3.00   sec  67.5 MBytes   566 Mbits/sec   40    433 KBytes       
[  5]   3.00-4.00   sec  66.2 MBytes   556 Mbits/sec  147    334 KBytes       
[  5]   4.00-5.00   sec  63.8 MBytes   535 Mbits/sec   33    382 KBytes       
[  5]   5.00-6.00   sec  68.8 MBytes   577 Mbits/sec   24    373 KBytes       
[  5]   6.00-7.00   sec  65.0 MBytes   545 Mbits/sec   30    393 KBytes       
[  5]   7.00-8.00   sec  65.0 MBytes   545 Mbits/sec   37    283 KBytes       
[  5]   8.00-9.00   sec  51.2 MBytes   430 Mbits/sec    7    239 KBytes       
[  5]   9.00-10.00  sec  55.0 MBytes   461 Mbits/sec   12    291 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   632 MBytes   530 Mbits/sec  1544             sender
[  5]   0.00-10.00  sec   629 MBytes   527 Mbits/sec                  receiver

I think I'm going to have to post an apology / correction video if it turns out my numbers were all inaccurate in that original WiFi 6 video :-/

And to confirm this annoying behavior, I'll probably set up a separate direct connection between the eth0 interface and my Mac. Still need to do more testing though!

geerlingguy commented 3 years ago

One other note: I would like to figure out how wpa_supplicant chooses which wireless interface to use... why does it pick wlan1 automatically? How could I pin the connection on wlan0 instead?

I know the easiest way to disable internal WiFi and Bluetooth (on CM4 models that have it) is to add the following in /boot/config.txt and reboot:

# Disable built-in wireless capabilities.
dtoverlay=disable-wifi
dtoverlay=disable-bt
geerlingguy commented 3 years ago

Just to confirm my suspicions:

$ iperf -c 10.0.100.1 --bind 10.0.100.168 --dualtest
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address 10.0.100.168
TCP window size:  128 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 10.0.100.1, TCP port 5001
Binding to local address 10.0.100.168
TCP window size:  153 KByte (default)
------------------------------------------------------------
[  5] local 10.0.100.168 port 54095 connected with 10.0.100.1 port 5001
[  4] local 10.0.100.168 port 5001 connected with 10.0.100.1 port 51833
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec   116 MBytes  96.9 Mbits/sec
[  4]  0.0-10.0 sec  1.22 GBytes  1.04 Gbits/sec

Compare to this result: https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/22#issuecomment-748551946

So then I ran it again:

$ iperf -c 10.0.100.1 --bind 10.0.100.168 --dualtest
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address 10.0.100.168
TCP window size:  128 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 10.0.100.1, TCP port 5001
Binding to local address 10.0.100.168
TCP window size:  144 KByte (default)
------------------------------------------------------------
[  5] local 10.0.100.168 port 41729 connected with 10.0.100.1 port 5001
[  4] local 10.0.100.168 port 5001 connected with 10.0.100.1 port 51872
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec   130 MBytes   108 Mbits/sec
[  4]  0.0-10.0 sec  1.20 GBytes  1.02 Gbits/sec

Hmm.

This is strange. Needs more testing.

geerlingguy commented 3 years ago

All right, did some more testing, and basically what I suspected (and what @6by9 mentioned elsewhere) seems to be true:

Linux's networking stack will route packets to whatever interface it wants if it's on the same subnet.

Aside: I'm not 100% sure the way it determines which interface to use—it seems like it could be either 'which one is higher in the list', or 'which one is fastest', or 'which one has lowest latency'... but networking is kind of voodoo magic to me still, and I've already dug further into the kernel's network packet handling than I could care to for years, and I'm going to give up trying to figure out the logic behind where packets are routed.

It looks like you can disable this feature, though, with arp_filter:

arp_filter - BOOLEAN
  1 - Allows you to have multiple network interfaces on the same
  subnet, and have the ARPs for each interface be answered
  based on whether or not the kernel would route a packet from
  the ARP'd IP out that interface (therefore you must use source
  based routing for this to work). In other words it allows control
  of which cards (usually 1) will respond to an arp request.

  0 - (default) The kernel can respond to arp requests with addresses
  from other interfaces. This may seem wrong but it usually makes
  sense, because it increases the chance of successful communication.
  IP addresses are owned by the complete host on Linux, not by
  particular interfaces. Only for more complex setups like load-
  balancing, does this behaviour cause problems.

  arp_filter for the interface will be enabled if at least one of
  conf/{all,interface}/arp_filter is set to TRUE,
  it will be disabled otherwise

From https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt, which was referenced in this post on the iperf mailing list.

Additional confirmation from @bmah888 in this comment on the iperf project issue queue.

geerlingguy commented 3 years ago

It is interesting, though, to see the WiFi interface seems to get better speeds than built-in Ethernet for receive... I wonder if the more powerful radio on the ASUS router is able to pump more packets through to the AX200's little antennae.

And I wonder if the beefier Intel antennae might make a difference? I think that's the next thing I'll test out.

acasadoalonso commented 3 years ago

Did you tried the M2 NGFF adapter with an SSD card ??? Thx.

geerlingguy commented 3 years ago

Rebooted with the Intel-supplied antennae attached. Testing again, first with iperf and --dualtest:

$ iperf -c 10.0.100.1 --bind 10.0.100.168 --dualtest
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address 10.0.100.168
TCP window size:  128 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 10.0.100.1, TCP port 5001
Binding to local address 10.0.100.168
TCP window size:  153 KByte (default)
------------------------------------------------------------
[  5] local 10.0.100.168 port 56705 connected with 10.0.100.1 port 5001
[  4] local 10.0.100.168 port 5001 connected with 10.0.100.1 port 60892
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec   368 MBytes   307 Mbits/sec
[  4]  0.0-10.0 sec  1.33 GBytes  1.14 Gbits/sec

Then with iperf3:

geerlingguy commented 3 years ago

Also noticed this in the logs:

[   10.250357] wlan0: Limiting TX power to 24 (24 - 0) dBm as advertised by 3c:7c:3f:6a:fa:c4

Checking on it with iwconfig:

pi@raspberrypi:~ $ iwconfig
...
wlan0     IEEE 802.11  ESSID:"geerling-acn"  
          Mode:Managed  Frequency:5.22 GHz  Access Point: 3C:7C:3F:6A:FA:C4   
          Bit Rate=6 Mb/s   Tx-Power=22 dBm   
          Retry short limit:7   RTS thr:off   Fragment thr:off
          Power Management:on
          Link Quality=69/70  Signal level=-41 dBm  
          Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
          Tx excessive retries:0  Invalid misc:16   Missed beacon:0

I looked into manually adjusting the TX power, but two issues:

  1. In many countries, manually boosting the power is technically illegal (beyond a certain limit), and so it's a good idea to stick to your defaults unless you know something fishy is going on (e.g. < 10 dBm power with a base station far away and a terrible link quality).
  2. Some devices can't handle higher power levels well, and could overheat or have other strange (and hard-to-debug) issues.
geerlingguy commented 3 years ago

Fun fact / point of comparison—my MacBook Pro 16" is about 20' further away from the AP than the Pi, and only has 802.11ac connectivity (80 Mhz, -56 dBm RSSI), and it's also getting around 460 Mbps :D

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   551 MBytes   462 Mbits/sec                  sender
[  5]   0.00-10.01  sec   550 MBytes   461 Mbits/sec                  receiver

Again proving the point that WiFi performance has a lot to do with good hardware and antenna design.

geerlingguy commented 3 years ago

An iperf3 maintainer mentioned the iperf3 issue may be fixed in https://github.com/esnet/iperf/issues/865#issuecomment-754214845 using the latest version from source with --bind-dev, so I'm going to try that.

First, using the iperf3 version installed via apt:

$ iperf3 --bind 10.0.100.169 -c 10.0.100.144
...
[  5]   0.00-10.01  sec  1.09 GBytes   933 Mbits/sec                  receiver

Definitely using Ethernet NIC, not WiFi (I was watching the blinkenlights).

So then I compiled from source like I did on the Rosewill:

# Compile iperf3 from source:
wget https://github.com/esnet/iperf/archive/master.zip
unzip master.zip
cd iperf-master/
./configure
make
cd src/
./iperf3 --help

Then I ran sudo ./iperf3 --bind 10.0.100.169 --bind-dev wlan0 -c 10.0.100.144 to connect via the wlan0 interface (IP address .169) to my Mac... but it's still showing 933 Mbps and the lights still blink only on the wired jack.

geerlingguy commented 3 years ago

Also doing a bunch more testing today, and it seems the AX200 Desktop Kit is performing more admirably, due to who-knows-what. It's very consistently hitting in the 800 Mbps range:

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1002 MBytes   841 Mbits/sec   50             sender
[  5]   0.00-10.01  sec  1001 MBytes   838 Mbits/sec                  receiver
geerlingguy commented 3 years ago

I opened https://github.com/esnet/iperf/issues/1099 to try to dig deeper into the new functionality in iperf3.

geerlingguy commented 3 years ago

For the AX200 performance, the plot thickens: https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/22#issuecomment-756873790

Seems like I could be able to get more performance and maybe marvell marvell's comment is incorrect.

geerlingguy commented 3 years ago

Closing issues where testing is at least mostly complete, to keep the issue queue tidy.

Moving further follow-up work to #56 .