Network stack issues when using Mediatek based chips on arm

[kasinjsh]

Dunno if this is the right place, but Im trying to debug network issues. I have multiple SBCs (all arm) and two Mediatek based usb adapters (MT7921au and MT7612U based), idea is to learn about network penetration testing whit these cards, more specific wifi attacks. Issue I face is hanging, both adapters hang on these type of tasks (so it seems). From Orange Pi I have zero2, zero3, also I have Radxa rockpi e, rockpi s - all show the same behavior. I have not seen issues whit x86 based PCs. For example when I do airodump-ng ( full command - sudo airodump-ng wlan1mon --manufacturer --channel 1-173 --wps --uptime --showack ) to scan, SBC becomes unresponsive on network, cant ping it. This can happen a few mins into scanning or hold for tens of minutes, but usually no more than hour. Tried also multiple operating systems like armbian,orangepi/radxa original builds, custom kali builds on zero3 etc, Ubuntu, Debian.... Kernel version always over 6.x. At this point I have made serial connection and I do see that it is not wifi stack issue I thought of, its network itself. wifi interface do not crash, its wired connection loosing connection. There is no errors in dmesg or syslog about what happens. It is not dhcp related issue and have tried on multiple networks connecting SBCs. In orangepi build there is orangepi-config manager (just like armbian) and there under network is option to reset network stack, after pressing that wired connection recovers.

I can recompile kernel and turn on more verbose debugging, but what flag will be useful? Or there is a better way to do this or do to resolve this?

[morrownr]

Hi @kasinjsh

Dunno if this is the right place...

If there is a better place, I am not aware of it.

I can recompile kernel and turn on more verbose debugging, but what flag will be useful?

Hold off on that.

To narrow the focus, maybe it is a good idea to concentrate on one adapter (mt7612u), one sbc (rockpi 5) and one os (Armbian). Is this doable?

That will give me the best chance to help you. I have adapters with both chipsets you have running on my RasPi4B as a dual band AP right now. I also maintain 6 Realtek drivers here and what you are seeing is not unique to Mediatek based adapters.

@morrownr

[bjlockie]

Maybe disable scatter gather.

[morrownr]

Maybe disable scatter gather.

That would be the first thing I would suggest as well but my head is still spinning from the complexity caused by so many distros, sbc's and adapters.

[kasinjsh]

Ok, then lets concentrate on OrangePi Zero3 and Fenvi ax1800 adopter (MT7921au), as I bought those for this type of testing and will be keeping in my bag. In troubleshooting stage will use original Debian 12 version (self compiled) from OrangePi.

I have managed to gather more information. When SBC looses internet connectivity it changes eth ports name to end0. That gives some new clues whats happening. I still dont know/understand why it happens, but have tried adding " extraargs=net.ifnames=0" as suggested by someone on armbian forums, that faced similar issue, but didn't helped. interface names was changed after, so env var was working. I also tried this whit my rpi4b, whit newest lite rpi os build (kernel 6.1.x) - the same issue. I have also tried your scripts for monitoring, also the same happens. Scatter gather was disabled on multiple OSes, but as I sad, its does not seem to be wifi interface related.

[kasinjsh]

Looks like I need to learn how to use journalctl, there is more info under NetworkManager log: https://cloud.harijs.id.lv/index.php/s/jSAxBCwxjebzHZ3 . wlx90de80d697f7 is the Mediatek interface, at Nov 19 08:22:51 you can see that NetworkManager got SIGTERM. Before that "device (wlx90de80d697f7): supplicant interface state: disconnected -> interface_disabled".

[kasinjsh]

Sorry about many posts but I do understand now whats happening.... When I prepare OS for monitoring I do following commands:

sudo ifconfig wlan1 down

sudo airmon-ng check kill

sudo airmon-ng start wlan1

sudo airodump-ng wlan1mon --manufacturer --channel 1-173 --wps --uptime --showack

Interface names changes, but the commands are the same. Now I closely monitored NetworkManagers logs and saw what I had suspicion about... "sudo airmon-ng check kill" does not kill just wpa_supplicant as it say, but also NetworkManager. As result, dhcp service is stopped and IP is not renewed. That's why I could get at max 1h, because home router gives out leases to 1h (this raised suspicion before). I will not use "airmon-ng check kill" when I'm connected like this and I should be fine. Will test this right away, and hope that wpa_supplicant in not causing issues to scans.

[ZerBea]

Usually MediaTek chipsets are running fine, because the drivers are well maintained. Tested drivers running arm (RaspBerry Pi) and x86_64 systems: mt76x0u https://github.com/ZerBea/hcxdumptool/discussions/361#discussioncomment-7550957 https://github.com/ZerBea/hcxdumptool/discussions/361#discussioncomment-7551660 https://github.com/ZerBea/hcxdumptool/discussions/361#discussioncomment-7553512

mt76x2u https://github.com/ZerBea/hcxdumptool/discussions/361#discussioncomment-7550759 https://github.com/ZerBea/hcxdumptool/discussions/361#discussioncomment-7550919

mt7601u https://github.com/ZerBea/hcxdumptool/discussions/361#discussioncomment-7551778 https://github.com/ZerBea/hcxdumptool/discussions/361#discussioncomment-7551969 https://github.com/ZerBea/hcxdumptool/discussions/361#discussioncomment-7552012

On all chipsets, it took only a few seconds to retrieve the frames which are mandatory to successfully recover a PSK.

mt7921u https://github.com/ZerBea/hcxdumptool/discussions/361#discussioncomment-7567045 This chipset is relatively new and there are some driver problems. We should give the developers enough time to fix it.

BTW: ifconfig is deprecated and it has been replaced by ip link and the same applies to iwconfig which has been replaced by iw, because wireless extensions (WEXT) are deprecated on new hardware: https://wireless.wiki.kernel.org/wext-statement

Also you should know that airodump-ng (Airodump-ng 1.7 rev 9be2f3e1) still uses wireless extensions:

$ sudo dmesg
[ 6499.926734] warning: `airodump-ng' uses wireless extensions which will stop working for Wi-Fi 7 hardware; use nl80211

[kasinjsh]

Usually MediaTek chipsets are running fine, because the drivers are well maintained. Tested drivers running arm (RaspBerry Pi) and x86_64 systems: mt76x0u ZerBea/hcxdumptool#361 (comment) ZerBea/hcxdumptool#361 (comment) ZerBea/hcxdumptool#361 (comment)

mt76x2u ZerBea/hcxdumptool#361 (comment) ZerBea/hcxdumptool#361 (comment)

mt7601u ZerBea/hcxdumptool#361 (comment) ZerBea/hcxdumptool#361 (comment) ZerBea/hcxdumptool#361 (comment)

On all chipsets, it took only a few seconds to retrieve the frames which are mandatory to successfully recover a PSK.

mt7921u ZerBea/hcxdumptool#361 (comment) This chipset is relatively new and there are some driver problems. We should give the developers enough time to fix it.

BTW: ifconfig is deprecated and it has been replaced by ip link and the same applies to iwconfig which has been replaced by iw, because wireless extensions (WEXT) are deprecated on new hardware: https://wireless.wiki.kernel.org/wext-statement

Also you should know that airodump-ng (Airodump-ng 1.7 rev 9be2f3e1) still uses wireless extensions:

$ sudo dmesg
[ 6499.926734] warning: `airodump-ng' uses wireless extensions which will stop working for Wi-Fi 7 hardware; use nl80211

I did a few long tests on two OSses and now I do see just driver issues, network it self is stable. I did long kismet scans and mt7921u do crash (actual crash in logs), but my MT7612U base Alfa runs fine. I do understand that this chip is new and practically I don't need 24/7/365 stability for my use case, so for now I'm fine. Will follow this chips development and I hope my SBCs kernel will be updated to something newer at some point (currently 6.1.x).

[ZerBea]

I agree, the "base" ALFAs (ACM and ACHM) are working fine (and out of the box). I think the AXML will get this driver support sooner or later too. But I'm sure the driver will not get WEXT. This warning appeared for the first time on kernel 6.3:

xxxx uses wireless extensions which will stop working for Wi-Fi 7 hardware; use nl80211

BTW: KALI is not easy to customize/configure and several supplied drivers are not Linux stock kernel drivers. If KALI is not well configured, a lot of "unwanted" services (which interfere e.g. with a WiFi device) are started by default. https://www.techtarget.com/searchsecurity/feature/How-to-configure-and-customize-Kali-Linux And that's just the tip of the iceberg. You discovered this already: As result, dhcp service is stopped and IP is not renewed.

[kasinjsh]

I agree, the "base" ALFAs (ACM and ACHM) are working fine (and out of the box). I think the AXML will get this driver support sooner or later too. But I'm sure the driver will not get WEXT. This warning appeared for the first time on kernel 6.3:

xxxx uses wireless extensions which will stop working for Wi-Fi 7 hardware; use nl80211

BTW: KALI is not easy to customize/configure and several supplied drivers are not Linux stock kernel drivers. If KALI is not well configured, a lot of "unwanted" services (which interfere e.g. with a WiFi device) are started by default. https://www.techtarget.com/searchsecurity/feature/How-to-configure-and-customize-Kali-Linux And that's just the tip of the iceberg. You discovered this already: As result, dhcp service is stopped and IP is not renewed.

Yes, but airmon-ng did not notify me on other OSes too. I have seen in screenshots and remember it seeing it too, that "check kill" do notify you if NetworkManager is about to be killed. In my case it just says it will kill wpa_supplicant, nothing more. Maybe its my bad and I dont understand how Linux network stack is build, meaning that killing wpa_supplicant means also killing NetworkManager.

[ZerBea]

Basically it is always the same procedure

get information about the active services

$ sudo systemctl --type=service --state=running

It is always good idea to take a look at this list of running services (instead of being dependent on a script).

stop them if they are unwanted (e.g. if Networkmanger is running)

$ sudo systemctl stop NetworkManager.service
$ sudo systemctl stop wpa_supplicant.service

Or configure them so that there is no need to stop them. NetworkManager is configured by "NetworkManager.conf". The configuration on your distribution. The basics are explained here: https://www.networkmanager.dev/docs/api/latest/NetworkManager.conf.html

set interface down

$ sudo ip link set INTERFACE_NAME down

set monitor mode

$ sudo iw dev INTERFACE_NAME set monitor none

if you're running a MediaTek chipset you should consider to set active monitor mode (which is useful when running a PMKID attack)

$ sudo iw dev INTERFACE_NAME set monitor active

set interface up

$ sudo ip link set INTERFACE_NAME up

Don't forget to set the correct wireless regulatory domain. Otherwise the kernel does not allow to use all frequencies:

$ sudo iw reg set US

The impact of the wireless regulatory domain setting is huge!

[ZerBea]

BTW: The frequency range of the AXML is really huge and I'm very excited to see a working monitor mode and working frame injection.

As mentioned above, the regulatory domain setting has a huge impact on the capabilities of the device.

Regulatory Domain: US

$ hcxdumptool -I wlp22s0f0u9u3i3

Requesting physical interface capabilities. This may take some time.
Please be patient...

interface information:

phy idx hw-mac       virtual-mac  m ifname           driver (protocol)
---------------------------------------------------------------------------------------------
  2   5 00c0cab574e6 00c0cab574e6 * wlp22s0f0u9u3i3  mt7921u (NETLINK)

available frequencies: frequency [channel] tx-power of Regulatory Domain: US

  2412 [  1] 30.0 dBm     2417 [  2] 30.0 dBm     2422 [  3] 30.0 dBm     2427 [  4] 30.0 dBm
  2432 [  5] 30.0 dBm     2437 [  6] 30.0 dBm     2442 [  7] 30.0 dBm     2447 [  8] 30.0 dBm
  2452 [  9] 30.0 dBm     2457 [ 10] 30.0 dBm     2462 [ 11] 30.0 dBm     2467 [ 12] disabled
  2472 [ 13] disabled     2484 [ 14] disabled     5180 [ 36] 23.0 dBm     5200 [ 40] 23.0 dBm
  5220 [ 44] 23.0 dBm     5240 [ 48] 23.0 dBm     5260 [ 52] 24.0 dBm     5280 [ 56] 24.0 dBm
  5300 [ 60] 24.0 dBm     5320 [ 64] 24.0 dBm     5500 [100] 24.0 dBm     5520 [104] 24.0 dBm
  5540 [108] 24.0 dBm     5560 [112] 24.0 dBm     5580 [116] 24.0 dBm     5600 [120] 24.0 dBm
  5620 [124] 24.0 dBm     5640 [128] 24.0 dBm     5660 [132] 24.0 dBm     5680 [136] 24.0 dBm
  5700 [140] 24.0 dBm     5720 [144] 24.0 dBm     5745 [149] 30.0 dBm     5765 [153] 30.0 dBm
  5785 [157] 30.0 dBm     5805 [161] 30.0 dBm     5825 [165] 30.0 dBm     5845 [169] 27.0 dBm
  5865 [173] 27.0 dBm     5885 [177] 27.0 dBm     5955 [  1] 12.0 dBm     5975 [  5] 12.0 dBm
  5995 [  9] 12.0 dBm     6015 [ 13] 12.0 dBm     6035 [ 17] 12.0 dBm     6055 [ 21] 12.0 dBm
  6075 [ 25] 12.0 dBm     6095 [ 29] 12.0 dBm     6115 [ 33] 12.0 dBm     6135 [ 37] 12.0 dBm
  6155 [ 41] 12.0 dBm     6175 [ 45] 12.0 dBm     6195 [ 49] 12.0 dBm     6215 [ 53] 12.0 dBm
  6235 [ 57] 12.0 dBm     6255 [ 61] 12.0 dBm     6275 [ 65] 12.0 dBm     6295 [ 69] 12.0 dBm
  6315 [ 73] 12.0 dBm     6335 [ 77] 12.0 dBm     6355 [ 81] 12.0 dBm     6375 [ 85] 12.0 dBm
  6395 [ 89] 12.0 dBm     6415 [ 93] 12.0 dBm     6435 [ 97] 12.0 dBm     6455 [101] 12.0 dBm
  6475 [105] 12.0 dBm     6495 [109] 12.0 dBm     6515 [113] 12.0 dBm     6535 [117] 12.0 dBm
  6555 [121] 12.0 dBm     6575 [125] 12.0 dBm     6595 [129] 12.0 dBm     6615 [133] 12.0 dBm
  6635 [137] 12.0 dBm     6655 [141] 12.0 dBm     6675 [145] 12.0 dBm     6695 [149] 12.0 dBm
  6715 [153] 12.0 dBm     6735 [157] 12.0 dBm     6755 [161] 12.0 dBm     6775 [165] 12.0 dBm
  6795 [169] 12.0 dBm     6815 [173] 12.0 dBm     6835 [177] 12.0 dBm     6855 [181] 12.0 dBm
  6875 [185] 12.0 dBm     6895 [189] 12.0 dBm     6915 [193] 12.0 dBm     6935 [197] 12.0 dBm
  6955 [201] 12.0 dBm     6975 [205] 12.0 dBm     6995 [209] 12.0 dBm     7015 [213] 12.0 dBm
  7035 [217] 12.0 dBm     7055 [221] 12.0 dBm     7075 [225] 12.0 dBm     7095 [229] 12.0 dBm
  7115 [233] 12.0 dBm

bye-bye

Regulatory Domain: 00 (unset)

$ hcxdumptool -I wlp22s0f0u9u3i3

Requesting physical interface capabilities. This may take some time.
Please be patient...

interface information:

phy idx hw-mac       virtual-mac  m ifname           driver (protocol)
---------------------------------------------------------------------------------------------
  3   6 00c0cab574e6 00c0cab574e6 * wlp22s0f0u9u3i3  mt7921u (NETLINK)

available frequencies: frequency [channel] tx-power of Regulatory Domain: 00

  2412 [  1] 20.0 dBm     2417 [  2] 20.0 dBm     2422 [  3] 20.0 dBm     2427 [  4] 20.0 dBm
  2432 [  5] 20.0 dBm     2437 [  6] 20.0 dBm     2442 [  7] 20.0 dBm     2447 [  8] 20.0 dBm
  2452 [  9] 20.0 dBm     2457 [ 10] 20.0 dBm     2462 [ 11] 20.0 dBm     2467 [ 12] 20.0 dBm
  2472 [ 13] 20.0 dBm     2484 [ 14] 20.0 dBm     5180 [ 36] 20.0 dBm     5200 [ 40] 20.0 dBm
  5220 [ 44] 20.0 dBm     5240 [ 48] 20.0 dBm     5260 [ 52] 20.0 dBm     5280 [ 56] 20.0 dBm
  5300 [ 60] 20.0 dBm     5320 [ 64] 20.0 dBm     5500 [100] 20.0 dBm     5520 [104] 20.0 dBm
  5540 [108] 20.0 dBm     5560 [112] 20.0 dBm     5580 [116] 20.0 dBm     5600 [120] 20.0 dBm
  5620 [124] 20.0 dBm     5640 [128] 20.0 dBm     5660 [132] 20.0 dBm     5680 [136] 20.0 dBm
  5700 [140] 20.0 dBm     5720 [144] 20.0 dBm     5745 [149] 20.0 dBm     5765 [153] 20.0 dBm
  5785 [157] 20.0 dBm     5805 [161] 20.0 dBm     5825 [165] 20.0 dBm     5845 [169] disabled
  5865 [173] disabled     5885 [177] disabled     5955 [  1] disabled     5975 [  5] disabled
  5995 [  9] disabled     6015 [ 13] disabled     6035 [ 17] disabled     6055 [ 21] disabled
  6075 [ 25] disabled     6095 [ 29] disabled     6115 [ 33] disabled     6135 [ 37] disabled
  6155 [ 41] disabled     6175 [ 45] disabled     6195 [ 49] disabled     6215 [ 53] disabled
  6235 [ 57] disabled     6255 [ 61] disabled     6275 [ 65] disabled     6295 [ 69] disabled
  6315 [ 73] disabled     6335 [ 77] disabled     6355 [ 81] disabled     6375 [ 85] disabled
  6395 [ 89] disabled     6415 [ 93] disabled     6435 [ 97] disabled     6455 [101] disabled
  6475 [105] disabled     6495 [109] disabled     6515 [113] disabled     6535 [117] disabled
  6555 [121] disabled     6575 [125] disabled     6595 [129] disabled     6615 [133] disabled
  6635 [137] disabled     6655 [141] disabled     6675 [145] disabled     6695 [149] disabled
  6715 [153] disabled     6735 [157] disabled     6755 [161] disabled     6775 [165] disabled
  6795 [169] disabled     6815 [173] disabled     6835 [177] disabled     6855 [181] disabled
  6875 [185] disabled     6895 [189] disabled     6915 [193] disabled     6935 [197] disabled
  6955 [201] disabled     6975 [205] disabled     6995 [209] disabled     7015 [213] disabled
  7035 [217] disabled     7055 [221] disabled     7075 [225] disabled     7095 [229] disabled
  7115 [233] disabled

bye-bye

By default, the regulatory domain is unset!

[kasinjsh]

Basically it is always the same procedure

get information about the active services

$ sudo systemctl --type=service --state=running

It is always good idea to take a look at this list of running services (instead of being dependent on a script).

stop them if they are unwanted (e.g. if Networkmanger is running)

$ sudo systemctl stop NetworkManager.service
$ sudo systemctl stop wpa_supplicant.service

Or configure them so that there is no need to stop them. NetworkManager is configured by "NetworkManager.conf". The configuration on your distribution. The basics are explained here: https://www.networkmanager.dev/docs/api/latest/NetworkManager.conf.html

set interface down

$ sudo ip link set INTERFACE_NAME down

set monitor mode

$ sudo iw dev INTERFACE_NAME set monitor none

if you're running a MediaTek chipset you should consider to set active monitor mode (which is useful when running a PMKID attack)

$ sudo iw dev INTERFACE_NAME set monitor active

set interface up

$ sudo ip link set INTERFACE_NAME up

Don't forget to set the correct wireless regulatory domain. Otherwise the kernel does not allow to use all frequencies:

$ sudo iw reg set US

The impact of the wireless regulatory domain setting is huge!

Tnx for the manual commands, will bookmark this. True about regulatory domain, but for now I dont see any 6ghz band networks around. Just around 5% are using wifi6, 6E is a long shot.

[ZerBea]

And the range is poor (compared to 2.4 GHz).

[kasinjsh]

And the range is poor (compared to 2.4 GHz).

Range is poor, but its a advantage in hi density networks, less CCI/ACI issues.

[kasinjsh]

A follow up. Now a few of my SBC have better mainline linux kernel support and I'm able to build latest kernels, LTS and edge. Tested over night airodump-ng monitoring, it usually hanged MT7921 in about and hour max, now, on 6.7.2 kernel, whit newest fw, lasted 7,5h. I did stop the test as there is no point test it any further, looks like chip is in fine condition now for pentesting. Over time will do more tests, maybe something else will pop up. But at least now I can follow driver development.