folder organization

[haochenku]

Dear all, Is there any documentation on each file's function? For example, where is the driver of the wifi? where is the fullmac of the wifi? I understand the csi_extractor.c is the main file for udp csi creation. But where is the driver of wifi? what are other files function?

regards

[matthiasseemoo]

The code is the documentation :-) The Wi-Fi driver is part of the Linux Kernel brcmfmac or bcmdhd (for Android).

On 7. Sep 2021, at 19:55, haochenku @.***> wrote:

Dear all, Is there any documentation on each file's function? For example, where is the driver of the wifi? where is the fullmac of the wifi? I understand the csi_extractor.c is the main file for udp csi creation. But where is the driver of wifi? what are other files function?

regards

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/seemoo-lab/nexmon_csi/issues/244, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACZ773QH7RJ2AWZPUYPWBOTUAZGZHANCNFSM5DS4SX7Q. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[haochenku]

Thank you @matthiasseemoo , I agree. Is the driver in the folder nexmon_csi/brcmfmac_4.19.y-nexmon/ for nexus 6 p? How does the UDP packet of CSI directly comes from chipset to android UDP port? Do you know which file create the UDP packet?

[zeroby0]

Please correct me if I'm wrong 🙏🏻

The wifi chip sends packets to the processor (and the kernel) as if it's an Ethernet device, so there are no "wifi drivers" for it.

There are two parts to the chip, the first is a D11 core which deals with MAC and physical layer, and runs broadcomm's proprietary firmware. I believe this firmware is shipped inside the chip's ROM, and Nexmon reads this firmware and decompiles it to assembly. One of the .patch files is applied to this decompiled assembly file. Then it's recompiled again, and applied to the D11 chip.

This modified firmware makes the D11 chip read CSI data from the registers when a packet arrives, and replace the data in the packets headers with the CSI data. Then this modified packet is sent to the second part of the wifi chip.

The second part of the chip is an ARM core that interfaces with the kernel. The code it runs is opensource, and is shipped with the kernel. The data structures it uses are in the brcmfmac_5.4.y-nexmon folder. The src/csi_extractor.c has the functions it should run on receiving a packet from the d11 chip.

When the ARM core receives the packet from d11, it reads the CSI data in the packet's headers, constructs a UDP Ethernet packet around the CSI data, and sends it to the kernel on port 5500.

This is, ofcourse, a huge over simplification of a beautiful hack. And I might be mis-interpreting things. There are more details in the paper linked in the readme, you should definitely read it.

Assuming that is correct, the modified ARM core code is loaded as a kernel module. But I don't know when the modified firmware is patched into the d11 core's memory after reboots.

[haochenku]

Thank you @zeroby0 for the long reply. I was thinking, the src/csi_extractor.c is in the arm core of the chipset. Then the folder brcmfmac_5.4.y-nexmon is the revised driver to forward the UDP packet to 5500. Otherwise, I am not sure how the arm in chipset can send a UDP packet directly to android kernel port 5500. What do you think?

[mzakharo]

@haochenku : When uploading new CSI on 5500, ARM core creates an 'skb' buffer from pkt_buf_get_skb API and fills it with UDP IP/Port information. This buffer, then is forwarded to Linux DHD drvier via wl->dev->chained->funcs->xmit function, which triggers an interrupt on Linux driver side, and DHD driver receives the buffer. Linux DHD driver then forwards the received 'skb blob' to the upper Linux network layer.

More on 'skb' buffers: http://amsekharkernel.blogspot.com/2014/08/what-is-skb-in-linux-kernel-what-are.html Great post on firmware: https://blog.quarkslab.com/reverse-engineering-broadcom-wireless-chipsets.html

.

[haochenku]

Thank you so much @mzakharo . Thank you for sharing the links. When I read the code, it is the nexmon_nl_ioctl_handler function which process the interrupt on linux driver and forward the skb blob to 5500, right?

Thanks again for your time.

[haochenku]

Also, @mzakharo I am working on the nexus 6p phones with 4358 chip. Should I look at nexmon_csi/brcmfmac_5.4.y-nexmon/ or nexmon_csi/brcmfmac_4.19.y-nexmon/ ?

[mzakharo]

neither, as per Makefile, those are only used for Raspberry pi (bcm43455c0).

For 4358, all that is done is:

a. original firmware file (arm + D11) is extracted and disassembled b. original firmware is patched c. final firmware is re-assembled (arm + D11), d. patched firmware file replaces original firmware file on device. e. on wifi initialisation, vanilla Linux DHD driver loads whatever firmware file that is loaded on the device

If you are running open-source kernel (from Lineage OS for example), you can add a few prints to help understand how things work further. Original DHD driver also has plenty of debug flags that can be enabled at runtime, I found that very helpful when debugging issues.

[haochenku]

I see. Thank you. So Nexmon did not change the DHD driver on the nexus phones. And we can directly revise the vanilla DHD drivers, right?

[mzakharo]

As you can read yourself here: https://github.com/seemoo-lab/nexmon_csi/blob/master/Makefile#L371

The only changes nexmon does to the nexus phones is replace old firmware with the new one.

Not sure what your goals for modifying DHD driver are though, for the most part, it is mostly a firmware load/reload/crash manager and hands-off data pipe.

[haochenku]

Thank you @mzakharo , it is much clear in the makefile. It only load the firmware. I'am curious how the vanilla DHD driver will be able to forward the skb to the port of 5500. My goal is to add the CPU timestamp to each CSI UDP packet. do you know which DHD file I should look at?

[mzakharo]

not sure I understand which CPU timestamp? DHD driver is under Linux, in which case, you already have this timestamp. https://stackoverflow.com/questions/41805687/linux-kernel-udp-reception-timestamp

If you want ARM microcode CPU timestamp, then you would have to include it inside skb_buffer payload, no need to modify anhything on the Linux Host.

[haochenku]

Thank you so much @mzakharo , really appreciate your time here. Right, I agree with you. For the timestamp in pcap file, you are saying it is already the time the UDP packet getting received by the Linux kernel, right? Here I am trying to make the pcap timestamp more accurate, I found the udp timestamp jitter around 900us. This may affect the performance of CSI processing. So I am wondering if there is a better way to get an accurate CPU time directly after the DHD interrupt. what do you think?

[mzakharo]

unlikely, Linux is not a real-time kernel. 900us jitter is actually really good for Linux, I have seen higher jitter (especially during system load).

Why not use firmware/microcode timestamp instead? that timestamp is generated by the hardware and is within <1uS accurate.

[haochenku]

Right. Thank you @mzakharo , I was trying to sync the CSI with other sensors. So the Linux time is needed. I guess I have to find the interrupt in the DHD driver for Nexus 6p . Do you happen to know the file ?

[mzakharo]

i dont remember, but enabling 'debug' runtime flags would help you narrow it down pretty quickly. Please update this thread if you manage to improve latency.

[haochenku]

Sure. Thank you @mzakharo , do you know where the driver is for Nexus 6p angler system? and some tutorials on the modification on this driver?

[haochenku]

If we use respberry Pi, then it is the folder nexmon_csi/brcmfmac_5.4.y-nexmon/ we should look at, right? @mzakharo

[matthiasseemoo]

I am just answering to the previous messages in general. On Android smartphones, we do not have to modify the bcmdhd driver to enable monitor mode. On vanilla Linux with brcmfmac, the Wi-Fi driver by default discards frames when not connected to a network. To receive frames in monitor mode, we had to modify the driver. Additionally, another separate monitor interface was added to the driver so that tools requireing an interface in monitor mode directly work. So in the end you only need the brcmfmac folder for the Raspberry Pi. All smartphones with bcmdhd do not require those files.

As we are generally working with FullMAC chips. All low level Wi-Fi operations are handled in the Wi-Fi firmware running on the ARM core in the Wi-Fi chip. On the interface to the host it normally exchanges Ethernet frames with the Wi-Fi driver. However, in the end all frames are just buffer of bytes that can represent any kind of frame you want. It is just a question of interpretation of the data. So to send data to a UDP port, we just create a new packet buffer and fill it with an Ethernet header, an IP header and a UDP header. The network stack in the host will interpret those headers and handle the received data as UDP datagrams.

On 9. Sep 2021, at 19:23, haochenku @.***> wrote:

If we use respberry Pi, then it is the folder nexmon_csi/brcmfmac_5.4.y-nexmon/ we should look at, right? @mzakharo https://github.com/mzakharo — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/seemoo-lab/nexmon_csi/issues/244#issuecomment-916291006, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACZ773UDNLXS6UAH7HRFN4TUBDUPNANCNFSM5DS4SX7Q. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[mzakharo]

Sure. Thank you @mzakharo , do you know where the driver is for Nexus 6p angler system? and some tutorials on the modification on this driver?

Broadcomm DHD driver is in the kernel sources. I do not know of any explicit tutorials to help you with this. It is all general linux driver/android knowledge. If you do not know how to locate, edit, and recompile Linux kernel drivers for Android, you might be better off just taking microcode timestamp and running a simple clock synchronisation algorithm between firmware and linux/UDP timestamp periodically to arrive at unified Epoch timestamp.

[yjxb1]

亲爱的大家，是否有关于每个文件功能的文档？例如，wifi的驱动程序在哪里？无线网络的全Mac在哪里？我知道 csi_extractor.c 是创建 udp csi 的主要文件。但是wifi的驱动力在哪里？其他文件功能是什么？

问候 Hello, do you understand the function of each file in this project? I now want to understand the working principle and process of the extractor filter, but I don't know which file it is in.