search

evilsocket / pwnagotchi

(⌐■_■) - Deep Reinforcement Learning instrumenting bettercap for WiFi pwning.
https://pwnagotchi.ai/
Other
7.55k stars 1.14k forks source link

nexmon blindness bug (brcmf_cfg80211_nexmon_set_channel) #267

Open evilsocket opened 4 years ago

evilsocket commented 4 years ago

every once in a while, nexmon dies with:

[ 4341.527847] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4101, -110
[ 4344.327806] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4097, -110
[ 4347.127853] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4098, -110
[ 4349.927917] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4099, -110
[ 4352.728074] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4100, -110
[ 4355.527970] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4101, -110
[ 4358.328022] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4102, -110
[ 4361.208095] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4103, -110
[ 4364.008157] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4104, -110
[ 4366.808218] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4105, -110
[ 4369.608431] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4097, -110
[ 4372.408345] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4098, -110
[ 4375.288408] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4099, -110
[ 4378.088474] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4100, -110
[ 4380.891399] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4101, -110

And only a reboot can fix the wifi, this is why the mon_max_blind_epochs parameter exists, to reboot the board when this happens.

Ideally we should document this known issue, the configuration and some day maybe fix it.

evilsocket commented 4 years ago

cc @hexwaxwing

evilsocket commented 4 years ago

docs added https://pwnagotchi.ai/usage/#known-issues

fregkos commented 4 years ago

You could try reloading the driver instead of rebooting the pi, using :

modprobe -r brcmfmac
modprobe brcmfmac

If this still fails, then a reboot will fix this. There is a great discussion about that problem here : https://www.bountysource.com/issues/56252669-wlan-freezes-in-raspberry-pi-3b

evilsocket commented 4 years ago

tried that way, it doesn't always work, the only reliable way is rebooting

jetbalsa commented 4 years ago

So, The Nexmon firmware is a little picky on how its interfaced, How are your bringing up the mon interface in linux before bettercap gets to it?

evilsocket commented 4 years ago

iw phy phy0 interface add mon0 type monitor && ifconfig mon0 up

evilsocket commented 4 years ago

from https://github.com/evilsocket/pwnagotchi/blob/master/builder/pwnagotchi.yml#L333

evilsocket commented 4 years ago

(from re4son monstart script)

jetbalsa commented 4 years ago

I just noticed on the Nexmon Repo that the bcm43455c0 does not support wifi frame injection. That might be the reason for the drivers crashing as they are not handling the requests to do frame injection correctly from bettercap.

jetbalsa commented 4 years ago

Ok, So I looked, We are using the older version of the firmware provided by Nexmon, I think Re4son kernel does it as its the default for Nexmon to use that. Patches and Commits from the Nexmon Project show 7.45.189 as the latest version you can use (The base firmware comes from the OEM, not the RPI foundation) as of RC4, we are using version 7.45.154 of the bcm43455c0 firmware This issue should only affect 3B+ and 4s, the 3B and the 0W use the same wifi chip and are listed as supporting injection.

jetbalsa commented 4 years ago

More Details, I've looked into how the Re4son kernel builder pulls down its firmware, its even /worse/ Looks like it pulls from https://github.com/Re4son/re4son-nexmon as its source of nexmon patches... its years out of date. I'm doing some prototyping to update this now.

jetbalsa commented 4 years ago

Good News, The Nexmon Patches with the stock kernel works well, Here is my DMesg output after running for 10 minutes, I will be running it for the next 24 hours to see if its stable, 

[    5.128852] brcmfmac: loading out-of-tree module taints kernel.
[    5.128864] brcmfmac: loading out-of-tree module taints kernel.
[    5.181306] brcmfmac: F1 signature read @0x18000000=0x15264345
[    5.190412] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[    5.190907] usbcore: registered new interface driver brcmfmac
[    5.566003] brcmfmac: brcmf_sdio_bus_preinit: before brcmf_sdio_debugfs_create
[    5.569776] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[    5.597271] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Oct 15 2019 20:30:25 version 7.45.189 (nexmon.org: -4) FWID 01-e1db26e2
[    5.675296] brcmfmac: brcmf_bus_started: before brcmf_debugfs_add_entry
[    8.495009] brcmfmac: brcmf_vif_add_validate: Attempt to add a MONITOR interface...
[    8.495024] brcmfmac: brcmf_mon_add_vif: brcmf_mon_add_vif called
[    8.495028] brcmfmac: brcmf_mon_add_vif: Adding vif "mon0"

Here is my uname

Linux pwnagotchi 4.19.75-v7+ #1270 SMP Tue Sep 24 18:45:11 BST 2019 armv7l GNU/Linux

jetbalsa commented 4 years ago

And I failed, You get a little more debug output this time overall..

[ 5387.511962] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4097, -110
[ 5390.551934] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4098, -110
[ 5393.601927] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4099, -110
[ 5396.631949] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4101, -110
[ 5399.671984] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4102, -110
[ 5402.711975] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4103, -110
[ 5405.752020] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4104, -110
[ 5408.791991] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4105, -110
[ 5411.831985] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4106, -110
[ 5414.872013] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53284, -110
[ 5417.911966] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53288, -110
[ 5420.951985] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53296, -110
[ 5423.511967] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4099, -110
[ 5426.312006] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4100, -110
[ 5429.111938] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4103, -110
[ 5431.911970] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4105, -110
[ 5434.711974] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53308, -110
[ 5437.511963] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53348, -110
[ 5440.311927] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53356, -110
[ 5443.111922] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53360, -110
[ 5445.911981] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53368, -110
[ 5448.711973] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53372, -110
[ 5451.511925] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53380, -110
[ 5454.311928] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4099, -110
[ 5457.111929] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4100, -110
[ 5459.911983] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4103, -110
[ 5462.711974] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4105, -110
[ 5465.511924] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53308, -110
[ 5468.311926] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53348, -110
[ 5471.111925] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53356, -110
[ 5473.912031] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53360, -110
[ 5476.471932] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4102, -110
[ 5479.271932] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4103, -110
[ 5482.081987] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4104, -110
[ 5484.871994] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4105, -110
[ 5487.671927] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53292, -110
[ 5490.471930] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53300, -110
[ 5493.271977] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53304, -110
[ 5496.071923] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53348, -110
[ 5498.872004] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53364, -110
[ 5501.671976] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53372, -110
[ 5504.471928] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53376, -110
[ 5507.031989] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[ 5507.032002] brcmfmac: brcmf_cfg80211_get_channel: chanspec failed (-110)
[ 5509.591930] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[ 5509.591939] brcmfmac: brcmf_cfg80211_get_tx_power: error (-110)
[ 5512.151989] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4102, -110
[ 5514.952027] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4103, -110

Edit: Added brcmfmac.debug=30 to /boot/cmdline.txt to try and get a better look at what the driver is doing, It enables full trace mode for the drivers, I have attacked a full debug log up to where it strops responding. syslog.gz

evilsocket commented 4 years ago

yep when that happens even trying to change channel manually doesn't work, i think it's the heat

jetbalsa commented 4 years ago

Based off this photo, There is no TIM under the wifi can at all... Since its used for shielding RF, this is of no surprise. Someone should get a IR Camera on the board and take a look.

image

Also You notice that White IC in the top left, Thats the same chip they used to show people the SuperMicro Implant... lulz

jetbalsa commented 4 years ago

So, I'm looking at the datasheet, for the CYW43455 It states that the Max temp for operation is 120C while under normal loads on a 4 layer board Of course Max oper temp is +85C The thing only puts out 1.2W but I did notice that on the older RPI0W Chips it had a self limiter for overheat, This one is not stated in the data sheet as having one

evilsocket commented 4 years ago

so you're saying that it's the chinese sabotaging our wifi pwning, right?

jetbalsa commented 4 years ago

Maybe.... Until I can get a proper Temp readout of the die itself while under the heavy load, We wont know.

evilsocket commented 4 years ago

suspence music

DrSchottky commented 4 years ago

Hi, is the same issue afflicting both 43455c0 and 43430a1? Have you got a reliable way to reproduce it?

hackabean commented 4 years ago

@evilsocket hey, I am investigating "brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4101, -110" aka Nexmon blindness bug issue and have noticed something that can potentially lead to resolution of the problem. My understanding was that patched Nexmon failed to set a channel, so instead of hopping on all channels I specified the channels manually to use for sending deauth frames in config.yml. In my case most AP are sitting on 1,4,6 and 11. Gotchi has been running for 15 minutes so far without any crashes. Can somebody confirm if it can be a valid solution ?

evilsocket commented 4 years ago

@hackabean well not really, you just worked around the issue, but when that happens the whole wifi chip stops being very responsive

evilsocket commented 4 years ago

if you specify the channels manually it will still hop ... i wonder if that's an issue related to hopping too quickly ...

evilsocket commented 4 years ago

maybe increasing the wifi.hop.period

hackabean commented 4 years ago

Took the gotchi for a walk in the busy neighborhood, have set the channels beforehand and the issue still remains. Got excited prematurely, this does not work either, sorry :(

DrSchottky commented 4 years ago

@hackabean -110 means firmware hanged. When working on nexmon I experienced blindness followed by cmds timeout when I was leaking memory in rx queue. Timeout/traps are fw bugs.

hackabean commented 4 years ago

@DrSchottky thanks for stepping in, seems like you know more about this issue. I just wished I could do something to make this less problematic so I was just poking around hoping to find a workaround. I am crossing my fingers that somebody will take a look at it.

DrSchottky commented 4 years ago

@hackabean to work on the problem I need to know as much as possible about the setup (board and kernel/driver/fw versions), the environment (does it crash under particular conditions? How often?) and I need to have a reliable way to reproduce the bug on my testbed. AFAIK monitor mode should be stable (never used injection in deep so far)

andrewbeard commented 4 years ago

@DrSchottky I can reproduce it pretty reliably on my Pi4 setup. I may need a little help gathering the info you need, but I'm pretty confident it will go blind within an hour every time I power it up. What can I do to help?

DrSchottky commented 4 years ago

@andrewbeard I need to know:

andrewbeard commented 4 years ago
  • Kernel: uname -a Linux pwnpi4 4.19.66-Re4son-v7l+ #1 SMP Mon Aug 19 08:12:03 AEST 2019 armv7l GNU/Linux
  • bcm fw: dmesg | grep 7.45 [ 2.749080] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Oct 8 2018 09:18:41 version 7.45.154 (nexmon.org: 2.2.2-195-g01e0-1) FWID 01-4fbe0b04
andrewbeard commented 4 years ago

System went blind within 3 minutes of powering up:

[ 2.739241] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6 [ 2.749080] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Oct 8 2018 09:18:41 version 7.45.154 (nexmon.org: 2.2.2-195-g01e0-1) FWID 01-4fbe0b04 [ 3.185453] bcmgenet: Skipping UMAC reset [ 3.301924] bcmgenet fd580000.genet: configuring instance for external RGMII (no delay) [ 3.302331] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 3.692992] uart-pl011 fe201000.serial: no DMA platform data [ 3.826165] fuse init (API version 7.27) [ 3.892931] brcmfmac: brcmf_vif_add_validate: Attempt to add a MONITOR interface... [ 3.892940] brcmfmac: brcmf_mon_add_vif: brcmf_mon_add_vif called [ 3.892944] brcmfmac: brcmf_mon_add_vif: Adding vif "mon0" [ 4.222668] Adding 102396k swap on /var/swap. Priority:-2 extents:2 across:106496k SSFS [ 4.311539] bcmgenet fd580000.genet eth0: Link is Down [ 8.471570] bcmgenet fd580000.genet eth0: Link is Up - 1Gbps/Full - flow control rx/tx [ 8.471593] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 10.595212] Bluetooth: Core ver 2.22 [ 10.595265] NET: Registered protocol family 31 [ 10.595270] Bluetooth: HCI device and connection manager initialized [ 10.596016] Bluetooth: HCI socket layer initialized [ 10.596026] Bluetooth: L2CAP socket layer initialized [ 10.596069] Bluetooth: SCO socket layer initialized [ 10.604001] Bluetooth: HCI UART driver ver 2.3 [ 10.604008] Bluetooth: HCI UART protocol H4 registered [ 10.604064] Bluetooth: HCI UART protocol Three-wire (H5) registered [ 10.604164] Bluetooth: HCI UART protocol Broadcom registered [ 10.763404] Bluetooth: BNEP (Ethernet Emulation) ver 1.3 [ 10.763411] Bluetooth: BNEP filters: protocol multicast [ 10.763423] Bluetooth: BNEP socket layer initialized [ 156.774707] brcmfmac: brcmf_sdio_hostmail: mailbox indicates firmware halted [ 160.556985] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4099, -110 [ 163.116341] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4107, -110 [ 165.996384] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4103, -110 [ 168.796416] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4104, -110 [ 171.596449] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4109, -110 [ 174.396468] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53284, -110 [ 177.196499] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53300, -110 [ 179.996566] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53304, -110 [ 182.636544] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4102, -110 [ 209.916754] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=53284, -110 [ 247.996974] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4099, -110 [ 255.597015] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4104, -110 [ 258.397045] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4107, -110 [ 261.197034] brcmfmac: brcmf_cfg80211_nexmon_set_channel: Set Channel failed: chspec=4108, -110

... and that continues ad nauseum

DrSchottky commented 4 years ago

7.45.154 is an outdated base fw (and if it cames form re4son-nexmon it also has outdated patches. Avoid it) Broadcom's built in monitor mode for bcm43455c0(RPi 3B+/4) is broken, but we fixed it (at least the bug we found) from nexmon in 7.45.189 patchset. You should get the headers for your kernel and build fw + driver with the latest patches from nexmon official repo.

I'm not telling you that it'll solve your problem, but before we go any further I need you to have the latest nexmon running on your system.

andrewbeard commented 4 years ago

@DrSchottky I am totally willing to do that, but I may need a bit of hand holding as I don't have a dev environment for this system yet. So far I've just been writing images to SD cards and changing config files. Any chance you (or anyone else) can point me at some info for building and replacing the kernel driver on a pi?

DrSchottky commented 4 years ago

As far as I read the released pwnagotchi img includes the header package, so ssh into your device and start following the README (but do not install the raspberrypi-kernel-headers package!)

andrewbeard commented 4 years ago

I'm trying to run through that now, but I've run into a couple snags. Most of them I'm getting through (aclocal version mismatches, etc) but my end result didn't go all that well:

COLLECTING STATISTICS read /usr/src/nexmon/STATISTICS.md for more information COMPILING src/version.c => obj/version.o (details: log/compiler.log) GENERATING LINKER FILE gen/nexmon.pre => gen/nexmon.ld GENERATING LINKER FILE gen/nexmon.pre => gen/flashpatches.ld LINKING OBJECTS => gen/patch.elf (details: log/linker.log, log/linker.err) GENERATING MAKE FILE gen/nexmon.pre => gen/nexmon.mk GENERATING MAKE FILE gen/nexmon.pre => gen/flashpatches.mk APPLYING FLASHPATCHES gen/flashpatches.mk => brcmfmac43455-sdio.bin (details: log/flashpatches.log) APPLYING PATCHES gen/nexmon.mk => brcmfmac43455-sdio.bin (details: log/patches.log) BUILDING DRIVER for kernel 4.19 brcmfmac_4.19.y-nexmon/brcmfmac.ko (details: log/driver.log) COPYING brcmfmac43455-sdio.bin => /lib/firmware/brcm/brcmfmac43455-sdio.bin UNLOADING brcmfmac RELOADING brcmfmac insmod: ERROR: could not insert module brcmfmac_4.19.y-nexmon/brcmfmac.ko: Invalid module format make: *** [Makefile:192: install-firmware] Error 1

I think this might be related to a mismatch in the kernel header versions, but I'm really not familiar with ARM ABI conventions.

root@pwnpi4:/usr/src# ls -al total 16 drwxr-xr-x 4 root root 4096 Nov 11 15:04 . drwxr-xr-x 10 root root 4096 Jul 9 20:03 .. drwxr-xr-x 24 root root 4096 Nov 11 15:03 linux-headers-4.19.66-Re4son-v7+ drwxr-xr-x 10 root root 4096 Nov 11 15:32 nexmon root@pwnpi4:/usr/src# uname -a Linux pwnpi4 4.19.66-Re4son-v7l+ #1 SMP Mon Aug 19 08:12:03 AEST 2019 armv7l GNU/Linux

The first build complained I didn't have a symlink in /lib/modules/4.19.66-Re4son-v7l+/build so I added one to /usr/src/linux-headers-4.19.66-Re4son-v7+. I'm thinking that l is more important than I hoped, though...

DrSchottky commented 4 years ago

The module you're compiling is not targeting your running kernel. Post the result of modinfo brcmfmac_4.19.y-nexmon/brcmfmac.ko

andrewbeard commented 4 years ago

root@pwnpi4:/usr/src/nexmon/patches/bcm43455c0/7_45_189/nexmon# modinfo brcmfmac_4.19.y-nexmon/brcmfmac.ko filename: /usr/src/nexmon/patches/bcm43455c0/7_45_189/nexmon/brcmfmac_4.19.y-nexmon/brcmfmac.ko license: Dual BSD/GPL description: Broadcom 802.11 wireless LAN fullmac driver. author: Broadcom Corporation firmware: brcm/brcmfmac4373-sdio.bin firmware: brcm/brcmfmac4356-sdio.bin firmware: brcm/brcmfmac4354-sdio.bin firmware: brcm/brcmfmac43455-sdio.bin firmware: brcm/brcmfmac43430-sdio.bin firmware: brcm/brcmfmac43430a0-sdio.bin firmware: brcm/brcmfmac4339-sdio.bin firmware: brcm/brcmfmac43362-sdio.bin firmware: brcm/brcmfmac4335-sdio.bin firmware: brcm/brcmfmac43340-sdio.bin firmware: brcm/brcmfmac4334-sdio.bin firmware: brcm/brcmfmac4330-sdio.bin firmware: brcm/brcmfmac4329-sdio.bin firmware: brcm/brcmfmac43241b5-sdio.bin firmware: brcm/brcmfmac43241b4-sdio.bin firmware: brcm/brcmfmac43241b0-sdio.bin firmware: brcm/brcmfmac43143-sdio.bin firmware: brcm/brcmfmac4373.bin firmware: brcm/brcmfmac43569.bin firmware: brcm/brcmfmac43242a.bin firmware: brcm/brcmfmac43236b.bin firmware: brcm/brcmfmac43143.bin srcversion: A208FF46FF4FDC4A48B10E7 alias: usb:v04B4p0BDCddcdscdpiciscipin alias: usb:v0A5Cp0BDCddcdscdpiciscipin alias: usb:v043Ep3101ddcdscdpiciscipin alias: usb:v04B4pBD29ddcdscdpiciscipin alias: usb:v13B1p0039ddcdscdpiciscipin alias: usb:v0A5CpBD27ddcdscdpiciscipin alias: usb:v0A5CpBD1Fddcdscdpiciscipin alias: usb:v0A5CpBD17ddcdscdpiciscipin alias: usb:v0A5CpBD1Eddcdscdpiciscipin alias: sdio:cv02D0d4373 alias: sdio:cv02D0d4356 alias: sdio:cv02D0d4354 alias: sdio:cv02D0dA9BF alias: sdio:cv02D0d4345 alias: sdio:cv02D0dA9A6 alias: sdio:cv02D0d4339 alias: sdio:cv02D0d4335 alias: sdio:cv02D0dA9A4 alias: sdio:cv02D0dA962 alias: sdio:cv02D0dA94D alias: sdio:cv02D0dA94C alias: sdio:cv02D0d4334 alias: sdio:cv02D0d4330 alias: sdio:cv02D0d4329 alias: sdio:cv02D0d4324 alias: sdio:cv02D0dA887 depends: brcmutil,cfg80211 name: brcmfmac vermagic: 4.19.66-Re4son-v7+ SMP mod_unload modversions ARMv7 p2v8 parm: txglomsz:Maximum tx packet chain size [SDIO] (int) parm: debug:Level of debug output (int) parm: p2pon:Enable legacy p2p management functionality (int) parm: feature_disable:Disable features (int) parm: alternative_fw_path:Alternative firmware path (string) parm: fcmode:Mode of firmware signalled flow control (int) parm: roamoff:Do not use internal roaming engine (int) parm: iapp:Enable partial support for the obsoleted Inter-Access Point Protocol (int) parm: ignore_probe_fail:always succeed probe for debugging (int)

It seems like the mismatch may be that I'm building for 4.19.66-Re4son-v7+ instead of 4.19.66-Re4son-v7l+ but I don't know how to force it.

DrSchottky commented 4 years ago

Yes, you have to use 4.19.66-Re4son-v7l+ headers. Don't you have that folder under /usr/src/?

andrewbeard commented 4 years ago

Nope

root@pwnpi4:/usr/src# ls -al /usr/src total 16 drwxr-xr-x 4 root root 4096 Nov 11 15:04 . drwxr-xr-x 10 root root 4096 Jul 9 20:03 .. drwxr-xr-x 24 root root 4096 Nov 11 16:00 linux-headers-4.19.66-Re4son-v7+ drwxr-xr-x 10 root root 4096 Nov 11 15:32 nexmon

DrSchottky commented 4 years ago

Oh, that's he problem. Is kalipi-kernel-headers package installed? if so, i think you need to download re4son kernel source and make the header pkg for your arch. @evilsocket are there any other reasons beyond built-in nexmon for using kalipi-kernel?

andrewbeard commented 4 years ago

Yup, the package is installed. For some reason it sticks everything in linux-headers-4.19.66-Re4son-v7+ and not linux-headers-4.19.66-Re4son-v7l+, though. It looks like that might just be an endianess flag, so I'd be a little surprised if the source was different. I don't think I have an actual kernel config to tell it otherwise, though.

andrewbeard commented 4 years ago

Just to add another complication is looks like was just a new kalipi kernel update pushed to the apt repos. Info looks the same, so I would be shocked if the problem didn't continue:

abeard@pwnpi4:/usr/src/nexmon $ uname -a Linux pwnpi4 4.19.81-Re4son-v7l+ #1 SMP Wed Nov 6 13:10:35 AEDT 2019 armv7l GNU/Linux abeard@pwnpi4:/usr/src/nexmon $ dmesg | grep 7.45 [ 2.736258] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Oct 8 2018 09:18:41 version 7.45.154 (nexmon.org: 2.2.2-195-g01e0-1) FWID 01-4fbe0b04

DrSchottky commented 4 years ago

What happen if you try ignore vermagic by adding -f to insmod?

andrewbeard commented 4 years ago

No luck:

root@pwnpi4:/usr/src/nexmon/patches/bcm43455c0/7_45_189/nexmon# insmod -f brcmfmac_4.19.y-nexmon/brcmfmac.ko insmod: ERROR: could not insert module brcmfmac_4.19.y-nexmon/brcmfmac.ko: Invalid module format

andrewbeard commented 4 years ago

So from what I can tell the source code for the kernels should be identical. I need the kernel config for the v7l+ version, though. None of the config extraction options seem to be built in, so I'm kind of at a loss as to where I can get it from.

andrewbeard commented 4 years ago

It seems like this may be an issue specific to the pi4. From what I can tell it's the only system that uses the v7l+ kernel. The build issue is probably the same thing that happened here: https://github.com/seemoo-lab/nexmon/issues/321

I'm kind of dead in the water at the moment without being able to find the right kernel source or config. It looks like the kernel is being installed from an apt repo here: http://http.re4son-kernel.com/re4son/

But the re4son website is pretty out of date and so I'm failing miserably trying to find the source package that corresponds to the installed kernel so far. It doesn't not appear that there's a package for the kalipi full kernel source, just the headers (that I already have, and don't include v7l+).

DrSchottky commented 4 years ago

Yes, v7l+ (where l stands for lpae) targets Rpi4 beacuse of its larger RAM. The problem is that the kalipi-kernel-headers package is incomplete since it contains only headers for v7+. If you look at re4son GH repo you can find their kernel and build script, so you should (never tried) to build a preper headers deb for your arch. But imho switching to a 3rd party kernel is more a hassle than a benefit, that's why I asked Simone if it's needed.

DrSchottky commented 4 years ago

@andrewbeard I patched the buildscripts to include v7l+ headers in the package. Try this It's not tested (I don't have a kalipi setup available atm) and the builder has many bugs, so it's an attempt.

andrewbeard commented 4 years ago

COLLECTING STATISTICS read /usr/src/nexmon/STATISTICS.md for more information COMPILING src/version.c => obj/version.o (details: log/compiler.log) GENERATING LINKER FILE gen/nexmon.pre => gen/nexmon.ld GENERATING LINKER FILE gen/nexmon.pre => gen/flashpatches.ld LINKING OBJECTS => gen/patch.elf (details: log/linker.log, log/linker.err) GENERATING MAKE FILE gen/nexmon.pre => gen/nexmon.mk GENERATING MAKE FILE gen/nexmon.pre => gen/flashpatches.mk APPLYING FLASHPATCHES gen/flashpatches.mk => brcmfmac43455-sdio.bin (details: log/flashpatches.log) APPLYING PATCHES gen/nexmon.mk => brcmfmac43455-sdio.bin (details: log/patches.log) BUILDING DRIVER for kernel 4.19 brcmfmac_4.19.y-nexmon/brcmfmac.ko (details: log/driver.log) /bin/sh: 1: scripts/basic/fixdep: Exec format error make[2]: [scripts/Makefile.build:303: /usr/src/nexmon/patches/bcm43455c0/7_45_189/nexmon/brcmfmac_4.19.y-nexmon/chip.o] Error 2 make[2]: Waiting for unfinished jobs.... /bin/sh: 1: scripts/basic/fixdep: Exec format error make[2]: [scripts/Makefile.build:303: /usr/src/nexmon/patches/bcm43455c0/7_45_189/nexmon/brcmfmac_4.19.y-nexmon/cfg80211.o] Error 2 make[1]: [Makefile:1519: module/usr/src/nexmon/patches/bcm43455c0/7_45_189/nexmon/brcmfmac_4.19.y-nexmon] Error 2 make: *** [Makefile:52: brcmfmac.ko] Error 2

root@pwnpi4:/usr/src/nexmon/patches/bcm43455c0/7_45_189/nexmon# file /usr/src/linux-headers-4.19.81-Re4son-v7l+/scripts/basic/fixdep /usr/src/linux-headers-4.19.81-Re4son-v7l+/scripts/basic/fixdep: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=f96cf37e4ab3abdfa90880655b262c4ae72c937a, for GNU/Linux 3.2.0, not stripped

It looks like the scripts folder in the headers has locally compiled binaries, but since the headers were cross-compiled on an x64 system it's causing the nexmon build to shit the bed.