3 sniffers, all with -c 39, see slightly different packets

[XenoKovah]

3 sniffers all sniffing the same channel (39 in this case) should see the same packets, right? If so, then I'd like to use this ticket to track what I'm seeing.

I have 3 sniffers, two Sonoff ("fast") USB dongles and one TI CC2652RB dev board. After I started noticing the possible issue, I made sure to update all to v1.9.3 firmware. I also confirmed the Sonoff dongles were operating at 2Mbaud by connecting to /dev/ttyUSB{0,1} with screen at 2Mbaud and confirming I saw valid base64 output. I also have the sniffle_hw.py edited to support the 2Mbaud per my previous #71.

Inside a VM with all the devices connected, I run 3 instances of Sniffle, on USB0, USB1, and ACM0 respectively. I'm at hardwear.io US in a BT-busy hotel, so there are fairly frequent connection events between devices going on in the background. However, I can see instances where 2 of the 3 sniffers appear to be following the traffic after a CONNECT_IND (in the sense that I see some subsequent AA traffic) but one isn't. Or all 3 are following to some degree, but one of the Sonoffs will see the packets, but the other won't. Since they're nominally identical hardware, with the antennas literally 2 inches apart, it seems unlikely, but not entirely impossible, that this is due to RSSI.

I will send the pcaps out of band to the email in the git log, so you can see what I mean.

[sultanqasim]

I haven’t yet gone through the PCAPs you sent me, but in principle this sounds normal. There are a couple reasons for this.

First, even though the three sniffers were positioned very close to each other, the signals they picked up would be slightly different; even small differences in orientation can make a substantial difference, and the PCB antenna on the Launchpad is a bit worse than the antenna that comes with the Sonoff dongles.

Second, the sniffer can struggle in crowded RF environments even when signal strengths of the target are strong because of the way the radio works. The radio listens for a sync word (0x8E89BED6 for BLE advertising), then captures a two byte packet header, and then receives additional data based on the length field in that header. Once the full packet is received, it computes the CRC, and if it matches the receiver CRC, it passes it up to higher levels of the firmware. Now, if there’s a weak advertisement signal whose sync word is captured, and then the real advertisement you’re interested in starts while that weak advertisement transmission was in progress, it will miss the stronger advertisement, and the weak advertisement’s CRC calculation will fail so it will also be dropped.

Both the RSSI filter and MAC filters in Sniffle are implemented at the higher level of the firmware, since the low level radio firmware is in ROM (and proprietary from TI). Thus, I can’t return to sync early for weak signals (in case of RSSI filter) or MAC mismatch (in case of a MAC filter). Thus unfortunately we’re stuck with this limitation. On the bright side, purpose built Bluetooth controller chips are dealing with similar radios and similar limitations.

In the future, I plan to add support for sending BLE frames with CRC errors up to the host, so you can see these interfering packets causing collisions.

A software defined radio based sniffer could avoid this limitation by both capturing a frame and searching for sync (in the middle of the frame being received) in parallel.

[XenoKovah]

Well, let me know what you think once you take a look at the specific packets then, because I think the details here will probably matter (and FWIW I wasn't using either RSSI or MAC filters in my capture commands. All default except the channel filter and save to pcap). Because I agree that I would expect to see weaker signals on the PCB antenna TI board, but when one Sonoff and the TI board are both seeing stuff and the other Sonoff isn't, the RSSI reported on the same packets between TI & Sonoff is basically the same. And also some of these occurrences are happening at what I would consider not-particularly-border-line RSSI values, like -56 to -63 (as opposed to if we were talking like -80s and -90s where it'd be more expected.) But when looking at the data, I couldn't explain it by a "One Sonoff is consistently worse than the other 2 sniffers", because then later in the packets, the two Sonoffs swap places in terms of which one is seeing and which one is missing.

[sultanqasim]

Having a look at your PCAP, that's a little different as the example you pointed out is during a connection (with a unique access address) rather than on an advertising channel. In that instance, packet loss should be less, but sometimes radio interference could cause a single bitflip on one of the sniffers that results in CRC failure and packet drop on one of them.

[XenoKovah]

Do you know if there's any way to test the theory that packets are being dropped due to CRC failure before being passed up to the custom firmware?

E.g. is there any stats that the hardware exposes that could be streamed back to the host (and e.g. embedded into the metadata visible in wireshark) to then inspect that for sniffer 1, which missed packet 250, the CRC error count was 12 at the time that packet 249 was returned, and 13 at the time that the nominal packet 250 was returned (but that packet is actually packet 251 from the perspective of the other sniffers which didn't miss the real packet 250)?

[sultanqasim]

I don't have any reporting to the host of CRC errors implemented right now. However, I could work on a basic implementation of sending packets with CRC errors to the host right now and probably have it working in a few hours, if you're staying around the hotel today.

[XenoKovah]

I've got 3.5 hours until my late check out, and I'd be happy to run it right up to the line to see if we can suss this out :) (and then I would be able to continue from the lobby afterwards, though the traffic isn't quite the same there)

[sultanqasim]

Alright, I'm almost ready with a proof of concept to capture invalid CRC packets. Give me another 30 mins to finish it up and test. Do you need prebuilt firmware binaries or can you compile the firmware yourself?

[XenoKovah]

Well, I figured I'd try to get firmware building working for the first time in anticipation of you asking me to compile it (I usually use prebuilt), and I'm currently getting 2 types of errors: 1) the "Permission denied" errors, which I don't understand given that the test user has what look like fine permissions to me and 2) the "#error "DeviceFamily_XYZ undefined. You must define a DeviceFamily_XYZ!" which seems more code-specific (but perhaps caused by the permission errors?)

test@VM:~/Downloads/Sniffle/fw$ make
make: /home/test/ti/sysconfig_1.18.1: Permission denied
make: /home/test/ti/sysconfig_1.18.1: Permission denied
Building AuxAdvScheduler.obj
In file included from ./RadioWrapper.h:16,
                 from ./AuxAdvScheduler.h:12,
                 from AuxAdvScheduler.c:11:
/home/test/ti/simplelink_cc13xx_cc26xx_sdk_7_40_00_77/source/ti/devices/DeviceFamily.h:225:6: error: #error "DeviceFamily_XYZ undefined. You must define a DeviceFamily_XYZ!"
  225 |     #error "DeviceFamily_XYZ undefined. You must define a DeviceFamily_XYZ!"
      |      ^~~~~
/home/test/ti/simplelink_cc13xx_cc26xx_sdk_7_40_00_77/source/ti/devices/DeviceFamily.h:254:39: fatal error: ti/devices/DeviceFamily_DIRECTORY/driverlib/rf_data_entry.h: No such file or directory
  254 | #define DeviceFamily_constructPath(x) <ti/devices/DeviceFamily_DIRECTORY/x>
      |                                       ^
compilation terminated.
make: *** [makefile:185: AuxAdvScheduler.obj] Error 1
test@VM:~/Downloads/Sniffle/fw$ ls -lah /home/test/ti/
total 24K
drwxrwxr-x  6 test test 4.0K May 14 13:08 .
drwxr-x--- 30 test test 4.0K Jun  2 14:34 ..
drwxrwxr-x  9 test test 4.0K Jun  2 14:28 simplelink_cc13xx_cc26xx_sdk_7_40_00_77
drwxrwxr-x  7 test test 4.0K Apr  4 10:24 sysconfig_1.18.1
drwxrwxr-x 12 test test 4.0K May 14 13:09 uniflash_8.6.0
drwxrwxr-x 10 test test 4.0K Apr  4 10:24 xdctools_3_62_01_15_core
test@VM:~/Downloads/Sniffle/fw$ ls -lah /home/test/ti/sysconfig_1.18.1/
total 100K
drwxrwxr-x 7 test test 4.0K Apr  4 10:24 .
drwxrwxr-x 6 test test 4.0K May 14 13:08 ..
-rw-rw-r-- 1 test test  206 Mar 11 09:45 conaninfo.txt
-rw-rw-r-- 1 test test  54K Mar 11 09:45 conanmanifest.txt
drwxrwxr-x 6 test test 4.0K Apr  4 10:24 dist
drwxrwxr-x 3 test test 4.0K Apr  4 10:24 .metadata
drwxrwxr-x 2 test test 4.0K Apr  4 10:24 nodejs
drwxrwxr-x 4 test test 4.0K Apr  4 10:24 nw
-rw-rw-r-- 1 test test  145 Mar 11 09:45 package.json
-rwxrwxr-x 1 test test  637 Mar 11 09:45 sysconfig_cli.sh
-rwxrwxr-x 1 test test  220 Mar 11 09:45 sysconfig_gui.sh
drwxrwxr-x 3 test test 4.0K Apr  4 10:24 tests

[sultanqasim]

The subsequent error is because of the permission error executing sysconfig. I'm guessing file permissions don't have the sysconfig utility as executable.

[sultanqasim]

Try building and using the invalid_crc branch once you get sysconfig permissions working.

[XenoKovah]

OK, yes, I accidentally had SYSCONFIG_TOOL ?= /home/user/ti/sysconfig_1.18.1 instead of SYSCONFIG_TOOL ?= /home/user/ti/sysconfig_1.18.1/sysconfig_cli.sh

Building is working now, and I'm testing.

[XenoKovah]

Closing this just because I have no further evidence that there's anything Sniffle-specific about the differences I see between 3 instances of Sniffle monitoring the same channel but seeing different data. Also because there's now the option in the mainline to see invalid CRC packets if one is so inclined.