search

andrewjfreyer / monitor

Distributed advertisement-based BTLE presence detection reported via mqtt
1.66k stars 194 forks source link

Support for android phones that do not advertise #161

Closed j-klier closed 5 years ago

j-klier commented 5 years ago

I'm sorry for posting a question as an issue, but don't have a better way to contact you directly as others have mentioned. I have Samsung Galaxy S7 and S9 phones that seem to fall into the group of phones that do not randomly advertise. I've been able to use both nRF Connect and Beacon Simulator to send PUBL advertisements using random MAC addresses. For example:

Mar 17 11:32:19 pi0w-2nd-floor bash[26024]:  }
Mar 17 11:32:19 pi0w-2nd-floor bash[26024]:     "version":"0.2.053"
Mar 17 11:32:19 pi0w-2nd-floor bash[26024]:     "timestamp":"2019-03-17T11:32:18",
Mar 17 11:32:19 pi0w-2nd-floor bash[26024]:     "retained":"false",
Mar 17 11:32:19 pi0w-2nd-floor bash[26024]:     "movement":"stationary",
Mar 17 11:32:19 pi0w-2nd-floor bash[26024]:     "flags":"",
Mar 17 11:32:19 pi0w-2nd-floor bash[26024]:     "rssi":"-77",
Mar 17 11:32:19 pi0w-2nd-floor bash[26024]:     "report_delay":"0",
Mar 17 11:32:19 pi0w-2nd-floor bash[26024]:     "type":"GENERIC_BEACON_PUBLIC",
Mar 17 11:32:19 pi0w-2nd-floor bash[26024]:     "manufacturer":"Unknown",
Mar 17 11:32:19 pi0w-2nd-floor bash[26024]:     "name":"SamsungGS7",
Mar 17 11:32:19 pi0w-2nd-floor bash[26024]:     "confidence":"100",
Mar 17 11:32:19 pi0w-2nd-floor bash[26024]:     "id":"42:F9:E6:1F:D9:F5",
Mar 17 11:32:19 pi0w-2nd-floor bash[26024]:  {
Mar 17 11:32:19 pi0w-2nd-floor bash[26024]: monitor/2ndfloor/42:F9:E6:1F:D9:F5
Mar 17 11:32:18 pi0w-2nd-floor bash[26024]: 0.2.053 17-03-2019 11:32:18 AM [CMD-PUBL]        42:F9:E6:1F:D9:F5 SamsungGS7 Unknown -77 dBm

These programs to not seem to have the ability to change the advertisement type to RAND even though the MAC address is indeed random (perhaps due to android limitations). monitorseems to only use RAND advertisements to trigger an arrival scan.

The FAQ says

I'm working on a solution. Stay tuned.

Can you elaborate on your plans and any suggestions on how to work around this limitation?

andrewjfreyer commented 5 years ago

Yes, I have a few ideas that I'm testing. The only Samsung phone I have had access to recently is much older than the devices folks were having issues with, so I'm not sure my testing will be relevant. The best solution may be to re-incorporate periodic scanning, but I'm still weighing pros and cons.

I have added verbose logging that can be helpful for figuring out what monitor sees. If you notice a pattern, feel free to let me know and I'll be better able to support all phones.

I'm hoping to get a samsung on hand soon, but I don't really know.

j-klier commented 5 years ago

Thanks. After more tinkering, I was able to get both nRF Connect and Beacon Simulator to advertise in a way that monitor can see (using RAND instead of PUBL).

Beacon Simulator EddyStone UID Mar 20 20:10:59 pi0w-2nd-floor bash[2622]: 0.2.053 20-03-2019 08:10:59 PM [CMD-RAND] [passed filter] data: 67:19:AE:FA:E2:CE pdu: ADV_SCAN_IND rssi: -64 dBm flags: none man: unknown delay: 0

Beacon Simulator iBeacon Mar 20 20:18:45 pi0w-2nd-floor bash[2622]: 0.2.053 20-03-2019 08:18:45 PM [CMD-RAND] [passed filter] data: 79:F5:63:6C:5A:56 pdu: ADV_SCAN_IND rssi: -66 dBm flags: none man: Apple, Inc. delay: 0

Using nRF Connect, I was only able to get it to work by cloning an existing advertising packet, but this allowed me to change the manufacturer to something much more unique like Bytestorm which is good for filtering. Mar 20 20:16:38 pi0w-2nd-floor bash[2622]: 0.2.053 20-03-2019 08:16:38 PM [CMD-RAND] [passed filter] data: 7B:B2:14:97:02:BA pdu: ADV_IND rssi: -67 dBm flags: 0x1a man: Bytestorm Ltd. delay: 0

In short, I have 3 android phones (the other phone is a nexus 5) and a tablet, and none of them seem to randomly advertise as expected without a beacon app. Does this behavior sound like a bug? These are my preferences:

> preference: minimum time between the same type of scan = 15
> preference: regex filter for flags to accept = .*
> preference: regex filter for flags to reject = NONE
> preference: regex filter for manufacturers to accept = .*
> preference: regex filter for manufacturers to reject = Microsoft

The good news is that I can use an app as a workaround, but I'm starting to think that the RAND advertisement followed by name scanning implementation is specific to iPhones. Are you getting the same feedback from others?

With respect to periodic scanning, I eventually want to use this system as a door unlock mechanism, and waiting for a door to unlock for even 1 minute would be too long. I like the quick arrival response of the current system.

andrewjfreyer commented 5 years ago

sorry for the delay in responding; had a kid. No, the random advertisements are a part of the bluetooth specification (as far as I can tell and have read when reading the spec; I'm far from an expert) so the fact that certain android phones don't advertise is the weird part.

Have you run verbose logging while cycling your android's bluetooth radio's power? Run via command line with -F flag and report back here. They'll look like this: 

***************** ARRIVE/REJECT FILTER DEBUG ********************** 
btmon packet:
> HCI Event: LE Meta Event (0x3e) plen 43                      [hci0] 66.009822
      LE Advertising Report (0x02)
        Num reports: 1
        Event type: Connectable undirected - ADV_IND (0x00)
        Address type: Public (0x00)
        Address: EE:FE:7E:14:2F:D9 (OUI EE-FE-7E)
        Data length: 31
        Flags: 0x06
          LE General Discoverable Mode
          BR/EDR Not Supported
        Company: Digi International Inc (R) (731)
          Data: a5817d12a0c4a04a0850ea869aa21067703125f153b30c17
        RSSI: -82 dBm (0xae)
flag pass:  only (0x1a|0x1b)
mfcg pass:  only (apple)
flag fail:  not (none)
mfcg fail:  not (google)
pdu filter:     only (adv_ind|adv_scan_ind|adv_nonconn_ind|scan_rsp)
blacklist:  false

type:       PUBL
mac:        EE:FE:7E:14:2F:D9
name:       ???
pdu:        ADV_IND
type:       ???
oem:        ???
rssi:       -82 dBm
flags:      0x06
hex:        a5817d12a0c4a04a0850ea869aa21067703125f153b30c17
resolvable: ???
man:        Digi International Inc
time:       1554399731
andrewjfreyer commented 5 years ago

For clarity, the brass tacks are that a samsung phone wouldn't be able to connect with any non-samsung bluetooth devices if it implemented its own passive advertising protocol. Clearly they do, so we just have to figure out how those devices automatically connect to cars, headphones, and whatnot.

j-klier commented 5 years ago

Wow, congratulations on a new baby. This is a hobby, so don't worry about being busy.
This one seems interesting for a few reasons

***************** ARRIVE/REJECT FILTER DEBUG **********************
btmon packet:
> HCI Event: LE Meta Event (0x3e) plen 26                      [hci0] 55.177618
      LE Advertising Report (0x02)
        Num reports: 1
        Event type: Connectable undirected - ADV_IND (0x00)
        Address type: Random (0x01)
        Address: 7A:99:54:A1:F1:80 (Resolvable)
        Data length: 14
        Flags: 0x1a
          LE General Discoverable Mode
          Simultaneous LE and BR/EDR (Controller)
          Simultaneous LE and BR/EDR (Host)
        Company: Apple, Inc. (76)
          Type: Unknown (16)
          Data: 0318ab692f
        RSSI: -96 dBm (0xa0)
flag pass:      only (.*)
mfcg pass:      only (.*)
flag fail:      not (none)
mfcg fail:      not (microsoft)
pdu filter:     only (adv_ind|adv_scan_ind|adv_nonconn_ind|scan_rsp)
blacklist:      false

type:           RAND
mac:            7A:99:54:A1:F1:80
name:           ???
pdu:            ADV_IND
type:           Unknown
oem:            ???
rssi:           -96 dBm
flags:          0x1a
hex:            0318ab692f
resolvable:     RESOLVABLE
man:            Apple, Inc.
time:           1554777891

***************** ARRIVE/REJECT FILTER DEBUG **********************
btmon packet:
> HCI Event: LE Meta Event (0x3e) plen 12                      [hci0] 56.812539
      LE Advertising Report (0x02)
        Num reports: 1
        Event type: Scan response - SCAN_RSP (0x04)
        Address type: Random (0x01)
        Address: 7A:99:54:A1:F1:80 (Resolvable)
        Data length: 0
        RSSI: -95 dBm (0xa1)
flag pass:      only (.*)
mfcg pass:      only (.*)
flag fail:      not (none)
mfcg fail:      not (microsoft)
pdu filter:     only (adv_ind|adv_scan_ind|adv_nonconn_ind|scan_rsp)
blacklist:      false

type:           RAND
mac:            7A:99:54:A1:F1:80
name:           ???
pdu:            SCAN_RSP
type:           ???
oem:            ???
rssi:           -95 dBm
flags:          ???
hex:            ???
resolvable:     RESOLVABLE
man:            ???
time:           1554777892

Let me know if you see anything in the log file: monitor.log

andrewjfreyer commented 5 years ago

Thanks for the log. I do anticipate that the Apple device is a neighbor's device if you do not have any apple products (including keyboards, apple tv, mice, etc) besides the iPad.

Sorry if I've asked this already, but have any of these android devices connected to any BTLE device before? Is there any reason for these devices to be advertising at all?

Have you been able to find anything in the bluetooth settings for either samsung or LG devices (e.g., here)?

Try this:

  1. Stop monitor & service entirely

  2. disable bluetooth entirely on your Samsung phone

  3. open two shell sessions

  4. first session: sudo hcitool lescan

  5. second session: sudo btmon

  6. wait for first session to settle down

  7. turn bluetooth back on, while the first session is viewable. Did you see any new mac address pop up?

  8. If YES - copy the mac and search through the output of btmon and post the packet here.

  9. If NO - is your samsung phone connected to any other bluetooth device? (e.g., the heartrate sensor or the bike helmet?) Can you power those devices down as well so that your phone isn't connected to anything?

j-klier commented 5 years ago

Have you been able to find anything in the bluetooth settings for either samsung

There's a setting under Connections -> Phone visibility that I've toggled on and off without any difference during a hcitool lescan I can see the phone when I'm in the Connections -> Bluetooth menu using a regular scan

Scanning ...
        94:B1:0A:DC:2F:1D       SamsungGS7

btmon shows lots of info from the scan

@ RAW Open: hcitool (privileged) version 2.22                                                           {0x0002} 2100.420391
< HCI Command: Inquiry (0x01|0x0001) plen 5                                                               [hci0] 2100.421451
        Access code: 0x9e8b33 (General Inquiry)
        Length: 10.24s (0x08)
        Num responses: 0
> HCI Event: Command Status (0x0f) plen 4                                                                 [hci0] 2100.421947
      Inquiry (0x01|0x0001) ncmd 1
        Status: Success (0x00)
> HCI Event: Extended Inquiry Result (0x2f) plen 255                                                      [hci0] 2107.267983
        Num responses: 1
        Address: 94:B1:0A:DC:2F:1D (Samsung Electronics Co.,Ltd)
        Page scan repetition mode: R1 (0x01)
        Page period mode: P0 (0x00)
        Class: 0x5a020c
          Major class: Phone (cellular, cordless, payphone, modem)
          Minor class: Smart phone
          Networking (LAN, Ad hoc)
          Capturing (Scanner, Microphone)
          Object Transfer (v-Inbox, v-Folder)
          Telephony (Cordless telephony, Modem, Headset)
        Clock offset: 0x49b5
        RSSI: -70 dBm (0xba)
        Name (complete): SamsungGS7
        16-bit Service UUIDs (complete): 10 entries
          OBEX Object Push (0x1105)
          Audio Source (0x110a)
          A/V Remote Control Target (0x110c)
          A/V Remote Control (0x110e)
          Headset AG (0x1112)
          PANU (0x1115)
          Handsfree Audio Gateway (0x111f)
          Phonebook Access Server (0x112f)
          PnP Information (0x1200)
          Message Access Server (0x1132)
        Company: not assigned (29952)
          Data: 02090100000001018ef108c78401ae5f3e617d9400000000000000005b50686f6e655d2000
> HCI Event: Inquiry Complete (0x01) plen 1                                                               [hci0] 2110.664537
        Status: Success (0x00)

Using the steps above with a a different phone that's been paired to a BLE device.

The fitness tracker is FE:C7:9A:F6:CA:2C ID115U paired to my Nexus 5.

The theory is that the phone must be one of the other devices. Here's what btmon says

> HCI Event: LE Meta Event (0x3e) plen 43                                                                  [hci0] 526.361877
      LE Advertising Report (0x02)
        Num reports: 1
        Event type: Non connectable undirected - ADV_NONCONN_IND (0x03)
        Address type: Random (0x01)
        Address: 21:BB:A0:36:1E:71 (Non-Resolvable)
        Data length: 31
        Company: Microsoft (6)
          Data: 01092002239701176627299e07badd8df93b67e7c5fac0daf9db74
        RSSI: -93 dBm (0xa3)
> HCI Event: LE Meta Event (0x3e) plen 37                                                                  [hci0] 526.382093
      LE Advertising Report (0x02)
        Num reports: 1
        Event type: Connectable undirected - ADV_IND (0x00)
        Address type: Random (0x01)
        Address: F1:87:36:6A:A1:E7 (Static)
        Data length: 25
        Appearance: Unknown (0x0000)
        Flags: 0x05
          LE Limited Discoverable Mode
          BR/EDR Not Supported
        16-bit Service UUIDs (complete): 1 entry
          Unknown (0x0af0)
        Name (complete): ID107Plus HR
        RSSI: -55 dBm (0xc9)
> HCI Event: LE Meta Event (0x3e) plen 22                                                                  [hci0] 526.383287
      LE Advertising Report (0x02)
        Num reports: 1
        Event type: Scan response - SCAN_RSP (0x04)
        Address type: Random (0x01)
        Address: F1:87:36:6A:A1:E7 (Static)
        Data length: 10
        Company: Awarepoint (572)
          Data: f187366aa1e7
        RSSI: -54 dBm (0xca)
> HCI Event: LE Meta Event (0x3e) plen 26                                                                  [hci0] 526.893534
      LE Advertising Report (0x02)
        Num reports: 1
        Event type: Connectable undirected - ADV_IND (0x00)
        Address type: Random (0x01)
        Address: 6A:6C:0D:C3:21:84 (Resolvable)
        Data length: 14
        Flags: 0x1a
          LE General Discoverable Mode
          Simultaneous LE and BR/EDR (Controller)
          Simultaneous LE and BR/EDR (Host)
        Company: Apple, Inc. (76)
          Type: Unknown (16)
          Data: 0318bd6f70
        RSSI: -84 dBm (0xac)
> HCI Event: LE Meta Event (0x3e) plen 26                                                                  [hci0] 527.581315
      LE Advertising Report (0x02)
        Num reports: 1
        Event type: Connectable undirected - ADV_IND (0x00)
        Address type: Random (0x01)
        Address: 5F:52:96:61:12:C1 (Resolvable)
        Data length: 14
        Flags: 0x1a
          LE General Discoverable Mode
          Simultaneous LE and BR/EDR (Controller)
          Simultaneous LE and BR/EDR (Host)
        Company: Apple, Inc. (76)
          Type: Unknown (16)
          Data: 031863eee1
        RSSI: -93 dBm (0xa3)

There are 2 Apple devices and one Microsoft device that still advertise even when my other phones are completely powered down. It appears that these phones are all listening for advertisements from other devices and do not advertise themselves over BLE, just like the other raspberry pi's running monitor

Is there anything else you can think of trying? For example, would you expect a raspberry pi to advertise when connected to a BLE device?

Alfiegerner commented 5 years ago

Just a thought, how about an option to do an arrival scan when ANY new device arrives? For those of us with distance between us and neighbors, any new devices appearing have a good chance of being our phones.

andrewjfreyer commented 5 years ago

@Alfiegerner that is how monitor works - any new random advertisement triggers a scan. Presuming, of course, that the new devices passes the filters you have set.

andrewjfreyer commented 5 years ago

Is there anything else you can think of trying? For example, would you expect a raspberry pi to advertise when connected to a BLE device?

@j-klier, no I would not expect that to advertise. Unless there was an active service on the pi that actively looked for a connection to another device. Still trying to find a samsung phone to borrow. i didn't realize that my social group is all iOS devices :-(

j-klier commented 5 years ago

I'm seeing similar behavior on all of my android devices (2 samsung phones, a nexus 5, and a tablet), so either this issue would not be hard for others to reproduce, or there is a problem with my setup. I also have an Htc One m8 that I could send to you if it would help debug this issue.

I'm also thinking about setting up the Pi as a server and pairing the phone with it so that it would automatically connect when it's within range. This site has a few examples https://www.pcsuggest.com/linux-bluetooth-setup-hcitool-bluez/

j-klier commented 5 years ago

Accidentally hit the close button. Sorry

andrewjfreyer commented 5 years ago

I have decided to add back periodic scanning mode. This will allow android phones to be automatically detected despite that they do not advertise. Try the beta channel with the -r argument.

Have you verified that the HTC One m8 exhibits the same behavior?

andrewjfreyer commented 5 years ago

Ok, as luck would have it, an Uber driver of mine was an android developer who shed some light on the bluetooth situation with Android. Online research confirms what he said.

In short, until Android OS includes at least one service that requires bluetooth peripheral mode to be enabled, Android devices will not advertise without an application running in the background. In short, as I understand it, Android/Google has been very slow to adopt BTLE peripheral mode as an option in addition to BTLE central mode. Here is a decently comprehensive list of phones that support peripheral mode should an application choose to leverage the appropriate API. It does not appear as though the native OS has an option (outside of the file sharing option mentioned above on LG phones) to enable this mode.

Unfortunately, it seems to me that absent an application causing an advertisement to send, Android users will not be able to use monitor in the same way as iOS users or beacon users.

Thanks for all your help trying to diagnose this issue. I still have some ideas to make monitor work a little bit better for android users. For now, the periodic scan mode is likely the best option.

andrewjfreyer commented 5 years ago

Additionally, I have setup the monitor beta to trigger arrival scans on new public beacons as well. This may help for apps that only advertise PUBL packets