NimBLEDevice::setScanFilterMode dropping messages

[davewyers]

Hi All

I have a sensor taking readings, broadcasting BLE advertisements, then going back to sleep. There is a seperate ESP32 device running NimBLE (1.4.1) continually scanning and writing the incoming sensor readings to SD Card. I have noticed that the scanning device is missing some readings.

The sensor device advertisement includes a counter value that increments with each sensor 'read and sent' cycle which allows us to validate if the messages are received.

I loaded the NimBLE Continuous Scan example code onto the ESP32 for testing. https://github.com/h2zero/NimBLE-Arduino/tree/release/1.4/examples/NimBLE_Scan_Continuous

With duplicates set to True, all sensor messages were printed to the serial monitor in sequential order correctly, but with obvious multiple duplicates.

pBLEScan->setAdvertisedDeviceCallbacks(new MyAdvertisedDeviceCallbacks(), true);

When I change the duplicate requirement to False not all messages get through. It appears that the scan skips one or two cycles for every line printed.

pBLEScan->setAdvertisedDeviceCallbacks(new MyAdvertisedDeviceCallbacks(), false);

Altering the ScanFilterMode does not alter the output.

NimBLEDevice::setScanFilterMode(CONFIG_BTDM_SCAN_DUPL_TYPE_DATA);

or

NimBLEDevice::setScanFilterMode(CONFIG_BTDM_SCAN_DUPL_TYPE_DATA_DEVICE);

Altering the CacheSize does not alter the output.

NimBLEDevice::setScanDuplicateCacheSize(800);

Any suggestions to resolve this?

Example output with duplicate filtering active, first 2 bytes are the uint16_t counter value.

09:13:35.639 -> Scanning...
09:13:35.906 -> Advertised Device: Name: XIAO_BT_THC, Address: ff:cb:de:f2:d0:0a, manufacturer data: 01009c2efb1a0827d902 
09:14:35.925 -> Advertised Device: Name: XIAO_BT_THC, Address: ff:cb:de:f2:d0:0a, manufacturer data: 0300a62e48180827dd02 
09:15:41.952 -> Advertised Device: Name: XIAO_BT_THC, Address: ff:cb:de:f2:d0:0a, manufacturer data: 06009a2e52180827e202 
09:16:36.361 -> Advertised Device: Name: XIAO_BT_THC, Address: ff:cb:de:f2:d0:0a, manufacturer data: 08008f2e7f1808271f03 
09:17:35.931 -> Advertised Device: Name: XIAO_BT_THC, Address: ff:cb:de:f2:d0:0a, manufacturer data: 0a00832ea21808271803 
09:18:36.179 -> Advertised Device: Name: XIAO_BT_THC, Address: ff:cb:de:f2:d0:0a, manufacturer data: 0c007e2ea11808271303 

[davewyers]

Turning on verbose debugging and I can see some of the missing messages.

Search "ff:cb:de:f2:d0:0a" (40 hits in 1 file of 1 searched)
  new 1 (40 hits)
    Line  20: I NimBLEScan: New advertiser: ff:cb:de:f2:d0:0a
    Line  49: I NimBLEScan: Updated advertiser: ff:cb:de:f2:d0:0a
    Line  50: Advertised Device: Name: XIAO_BT_THC, Address: ff:cb:de:f2:d0:0a, manufacturer data: a919eb2d31187f268b02
    Line  54: 2023-06-29T09:17:29,6569,XIAO_BT_THC,ff:cb:de:f2:d0:0a,-43,17.55,61.93,985.50,651
    Line  55: D NimBLEScan: erase device: ff:cb:de:f2:d0:0a

Missing record here

    Line  80: I NimBLEScan: New advertiser: ff:cb:de:f2:d0:0a
    Line  81: I NimBLEScan: Updated advertiser: ff:cb:de:f2:d0:0a
    Line  82: Advertised Device: Name: XIAO_BT_THC, Address: ff:cb:de:f2:d0:0a, manufacturer data: aa19ed2d1a1880268b02 
    Line  86: 2023-06-29T09:17:56,6570,XIAO_BT_THC,ff:cb:de:f2:d0:0a,-43,17.57,61.70,985.60,651
    Line  87: D NimBLEScan: erase device: ff:cb:de:f2:d0:0a

    Line 106: I NimBLEScan: New advertiser: ff:cb:de:f2:d0:0a
    Line 107: I NimBLEScan: Updated advertiser: ff:cb:de:f2:d0:0a
    Line 108: Advertised Device: Name: XIAO_BT_THC, Address: ff:cb:de:f2:d0:0a, manufacturer data: ab19f02d47187f268602
    Line 112: 2023-06-29T09:18:39,6571,XIAO_BT_THC,ff:cb:de:f2:d0:0a,-51,17.60,62.15,985.50,646
    Line 113: D NimBLEScan: erase device: ff:cb:de:f2:d0:0a

    **Line 130: I NimBLEScan: New advertiser: ff:cb:de:f2:d0:0a**  

Missing two records here

    Line 163: I NimBLEScan: Updated advertiser: ff:cb:de:f2:d0:0a
    Line 164: Advertised Device: Name: XIAO_BT_THC, Address: ff:cb:de:f2:d0:0a, manufacturer data: ae19f52d0a1880268102
    Line 168: 2023-06-29T09:19:45,6574,XIAO_BT_THC,ff:cb:de:f2:d0:0a,-49,17.65,61.54,985.60,641
    Line 169: D NimBLEScan: erase device: ff:cb:de:f2:d0:0a

    Line 192: I NimBLEScan: New advertiser: ff:cb:de:f2:d0:0a
    Line 193: I NimBLEScan: Updated advertiser: ff:cb:de:f2:d0:0a
    Line 194: Advertised Device: Name: XIAO_BT_THC, Address: ff:cb:de:f2:d0:0a, manufacturer data: af19f82d2b1880268702 
    Line 198: 2023-06-29T09:20:12,6575,XIAO_BT_THC,ff:cb:de:f2:d0:0a,-46,17.68,61.87,985.60,647
    Line 199: D NimBLEScan: erase device: ff:cb:de:f2:d0:0a

    Line 208: I NimBLEScan: New advertiser: ff:cb:de:f2:d0:0a
    Line 209: I NimBLEScan: Updated advertiser: ff:cb:de:f2:d0:0a
    Line 210: Advertised Device: Name: XIAO_BT_THC, Address: ff:cb:de:f2:d0:0a, manufacturer data: b019f72d341880268902 
    Line 214: 2023-06-29T09:20:55,6576,XIAO_BT_THC,ff:cb:de:f2:d0:0a,-51,17.67,61.96,985.60,649
    Line 215: D NimBLEScan: erase device: ff:cb:de:f2:d0:0a

    **Line 245: I NimBLEScan: New advertiser: ff:cb:de:f2:d0:0a**

Missing record here

    Line 277: I NimBLEScan: Updated advertiser: ff:cb:de:f2:d0:0a
    Line 278: Advertised Device: Name: XIAO_BT_THC, Address: ff:cb:de:f2:d0:0a, manufacturer data: b319fb2dd41780268402
    Line 282: 2023-06-29T09:22:00,6579,XIAO_BT_THC,ff:cb:de:f2:d0:0a,-45,17.71,61.00,985.60,644
    Line 283: D NimBLEScan: erase device: ff:cb:de:f2:d0:0a

    Line 305: I NimBLEScan: New advertiser: ff:cb:de:f2:d0:0a    
    Line 306: I NimBLEScan: Updated advertiser: ff:cb:de:f2:d0:0a
    Line 307: Advertised Device: Name: XIAO_BT_THC, Address: ff:cb:de:f2:d0:0a, manufacturer data: b419012ef41780268302 
    Line 311: 2023-06-29T09:22:28,6580,XIAO_BT_THC,ff:cb:de:f2:d0:0a,-45,17.77,61.32,985.60,643
    Line 312: D NimBLEScan: erase device: ff:cb:de:f2:d0:0a

[davewyers]

I have the sender and receiver both on the desk next to each other.

The sender is printing the Cycle ID to the serial port for each sensor read.

Advertising is Running
Advertising time passed, advertising will now stop.
Advertising is Stopped
Restarting in 5 sec
**Struct Cycle: 6709**
Advertising is addManufacturerData start
Advertising is addManufacturerData end
Setup end

The reciever is displaying the Cycle ID for each message it processes.

I NimBLEScan: New advertiser: ff:cb:de:f2:d0:0a
I NimBLEScan: Updated advertiser: ff:cb:de:f2:d0:0a
Advertised Device: Name: XIAO_BT_THC, Address: ff:cb:de:f2:d0:0a, manufacturer data: 3a1a092eb21781265002
**Hash Index: 6712  - Last message processed ID
Cycle Index: 6714     - New message ID**
New Cycle for this Hash
2023-06-29T10:23:06,6714,XIAO_BT_THC,ff:cb:de:f2:d0:0a,-55,17.85,60.66,985.70,592
D NimBLEScan: erase device: ff:cb:de:f2:d0:0a

[h2zero]

Messages can be missed for many reasons, in some of these the scan response is missing so the callback is not yet triggered. Are you writing to the SD card while scanning? That may be the cause.

[davewyers]

The sensor BLE generates a new value every 30 seconds, then broadcasts for 20 seconds, then sleeps. Each cycle creates a new cycleID number.

If the system was busy, I would think that it would pick up at least one packet over the 20 second broadcast.

When I remove filtering and impliment my own version I capture all cycles, so I think there is an issue in the Nimble filtering.

[h2zero]

If the filtering is causing this issue then it would be an espressif problem. This repo does not do the filtering, only requests it from the controller which is part of esp-idf and the Arduino core.

[davewyers]

Ok, good to know thanks.