search

andrewjfreyer / monitor

Distributed advertisement-based BTLE presence detection reported via mqtt
1.66k stars 194 forks source link

[Ubuntu] monitor not triggering arrive scans in response to BTLE advertisements that pass all filters #168

Closed vladtepz closed 5 years ago

vladtepz commented 5 years ago

OS: Ubuntu 18.04.2 LTS

Monitor Version: 0.2.148 Command Used: sudo bash monitor.sh -a -x

My issue:

Monitor will only scan if I manually trigger a scan via "monitor/scan/arrive" or "monitor/scan/depart", and this is working as intended.

Unfortunately, it will not scan for bluetooth advertisements.

behavior_preferences:

# ---------------------------
#                               
# BEHAVIOR PREFERENCES
#                               
# ---------------------------

#MAX RETRY ATTEMPTS FOR ARRIVAL
PREF_ARRIVAL_SCAN_ATTEMPTS=1

#MAX RETRY ATTEMPTS FOR DEPART
PREF_DEPART_SCAN_ATTEMPTS=2

#SECONDS UNTIL A BEACON IS CONSIDERED EXPIRED
PREF_BEACON_EXPIRATION=240

#MINIMUM TIME BEWTEEN THE SAME TYPE OF SCAN (ARRIVE SCAN, DEPART SCAN)
PREF_MINIMUM_TIME_BETWEEN_SCANS=15

#ARRIVE TRIGGER FILTER(S)
PREF_PASS_FILTER_ADV_FLAGS_ARRIVE="0x1a|0x1b"
PREF_PASS_FILTER_MANUFACTURER_ARRIVE="Apple|Google|Samsung"

#ARRIVE TRIGGER NEGATIVE FILTER(S)
PREF_FAIL_FILTER_ADV_FLAGS_ARRIVE="NONE" 
PREF_FAIL_FILTER_MANUFACTURER_ARRIVE="NONE"

PREF_DEVICE_TRACKER_REPORT=true

PREF_MQTT_REPORT_SCAN_MESSAGES=true

I've also tried (unsuccessfully) with:

PREF_PASS_FILTER_ADV_FLAGS_ARRIVE=".*"
PREF_PASS_FILTER_MANUFACTURER_ARRIVE=".*"

Here is my output with the "-F" flag enabled:

***************** ARRIVE/REJECT FILTER DEBUG **********************
btmon packet:
> HCI Event: LE Meta Event (0x3e) plen 26                   #7 [hci0] 39.178764
      LE Advertising Report (0x02)
        Num reports: 1
        Event type: Connectable undirected - ADV_IND (0x00)
        Address type: Random (0x01)
        Address: XX:XX:XX:XX:XX (Resolvable)
        Data length: 14
        Flags: 0x1a
          LE General Discoverable Mode
          Simultaneous LE and BR/EDR (Controller)
          Simultaneous LE and BR/EDR (Host)
        Company: Apple, Inc. (76)
          Type: Unknown (16)
          Data: 0118e2b532
        RSSI: -97 dBm (0x9f)
flag pass:      only (0x1a|0x1b)
mfcg pass:      only (apple|google|samsung)
flag fail:      not (none)
mfcg fail:      not (none)
pdu filter:     only (adv_ind|adv_scan_ind|adv_nonconn_ind|scan_rsp)
blacklist:      false

type:           RAND
mac:            XX:XX:XX:XX:XX
name:           ???
pdu:            ADV_IND
type:           Unknown
oem:            ???
rssi:           -97 dBm
flags:          0x1a
hex:            xxxxxxxxxx
resolvable:     RESOLVABLE
man:            Apple, Inc.
time:           1554820224

I'm expecting the advertisement to trigger a scan, flag pass: to blacklist: are all highlighted Green, but a scan is ever attempted.

Do I not have it setup correctly?

If you need any more info from me, just let me know.

Thanks.

andrewjfreyer commented 5 years ago

Are all your devices already marked 'home'? If so, an arrive scan is not necessary and is not performed. Can you post monitor log output without the -F flag so that I can see the progression from start to when we expect an arrive scan to happen?

vladtepz commented 5 years ago

4 devices are 'home', 4 devices are 'not home'. ( 8 total )

Without the -F or -V Flags, I get a bunch of these, and then nothing ever happens unless I trigger a scan with "monitor/scan/arrive"

mike@hassio:~/.monitor$ sudo bash monitor.sh -a -x
> starting monitor.sh (v. 0.2.148)...
> retaining mqtt status reports
> warning: monitor.service not installed. Install service? (y/n)
n
##### (below x8, addresses redacted ) #####
> XX:XX:XX:XX:XX confidence topic: monitor/hassio { id: XX:XX:XX:XX:XX ... } (has not previously connected to hci0)
> XX:XX:XX:XX:XX device_tracker topic: monitor/hassio { id: XX:XX:XX:XX:XX ... }/device_tracker [not_home or home]

Then nothing.... It doesn't do an initial scan for me when starting, if that's what you were looking for.

andrewjfreyer commented 5 years ago

The issue may be with the number of devices you're trying to track. Select a single device first to see if an improvement appears.

When you run: sudo hcitool lescan does the output look like a very fast stream of data or does it settle down in maybe 15 - 30 seconds to a list?

vladtepz commented 5 years ago

sudo hcitool lescan is quiet. Nothing shows up initially, and then, only 1 device did show up after a few minutes. ( ran it for about 5 minutes )

Started Monitor up with only x2 devices. ( x1 device I knew was home, x1 device I knew was away ) Waited for the "LE Advertising Report" to show up, it passed the filters.

Results are the same. No scan was triggered.

andrewjfreyer commented 5 years ago

The hcitool output is quite surprising. Are you in a rural or otherwise radio isolated area?

vladtepz commented 5 years ago

fairly rural, nearest neighbor is around 200 feet / 60 meters away.

andrewjfreyer commented 5 years ago

Ok, that explains it. I have pushed an update in the last hour - have you updated? That corrects a minor issue with single topic mode.

The brass tacks here is that monitor uses hcitool lescan to obtain information about your bluetooth environment. You evidently only have one device in the bluetooth environment, so it's probably being processed before monitor fully boots up and is able to receive that data.

As a test, try this. Let monitor boot and then power cycle bluetooth on the iPhone (e.g., through Settings App, not through the control center button). You should see a new device and monitor will begin to scan.

Most users won't run into this problem because they have more than just a single device advertising, so monitor has plenty of devices that can trigger it. I think I have a solution for you. Let me work on it.

vladtepz commented 5 years ago

Monitor Version: 0.2.150

Updated, did a used -d to restore to default behavior_preferences, no change.

FWIW, just a little more background, I've also been testing, although not all at the same time, on a Raspberry Pi 3 Model B, & Raspberry Pi 3 Model B+ ( running raspbian )

All 3 devices are experiencing the same issue. They all update if triggered manually, none through BTLE advertisements.

I've had Monitor ( 0.1.675 ) running for about the last 5 months prior without issue. I had never updated until recently because it worked well and I kind of forgot about it.

Fast Forward about a week ago, and I am currently in the process of updating/upgrading all my local Servers/PCs/Media Boxes/etc..., hence me stumbling upon the issue. Didn't have a chance to post an issue until today.

If you can't think of anything else, perhaps I'll try to get v0.1.675 running again and slowly update every few releases and see where it breaks.

Unless you ( hopefully) have any other suggestions?

andrewjfreyer commented 5 years ago

Updated to 151, with a small change that should help your installation(s) get over the hump a bit - see if that makes a change. what about the test above?

vladtepz commented 5 years ago

Tested monitor.sh (v. 0.2.151)

Results:

a few side notes:

Another thing to note.

power cycle bluetooth on the iPhone (e.g., through Settings App, not through the control center button). You should see a new device and monitor will begin to scan.

Nothing shows up when I do this, I've tried with my both my iPhone and x2 iPads. I've even tried rebooting the devices close to the device running Monitor.

vladtepz commented 5 years ago

OK @andrewjfreyer so I finally had a full day to test and the issue may be with my specific set-up and/or hardware on one of my servers and may not be related to Monitor at all.

Here are my results:

This leads me to believe that the issue is on my side. I'll have to track it down, because prior to a reformat, Monitor was working as intended on this machine/OS. The only significant changes were Hard Drive & Memory upgrades and the addition of a HUSBZB-1 Zwave/Zigbee USB stick.

You can probably close this issue for now. Thanks for the help. 👍

andrewjfreyer commented 5 years ago

Hmm. Sorry to hear that there isn't a resolution thus far... I'll close for now. Thanks for the report!

mrand commented 5 years ago

@vladtepz I don't know if it affects what you were seeing, but I'm curious regardless for my install: what version of MQTT you are running?

Looks like the default for 18.04 / Bionic is MQTT version 1.4.15 - is that working for you? If you upgraded, what method did you use? The PPA appears to jump to 1.6.2.

vladtepz commented 5 years ago

@mrand At the time, I tested with the Docker version of Mosquitto, The Mosquitto PPA, and the Hassio Mosquitto add-on. ( Among creating my own docker images )

I believe all 3 were on version 1.5.x at the time, as I remember reading that a 1.5+ version was required.

Not sure what happened to 1.5 in PPA though...

I've stopped using Monitor for now, because of the issue, until I have more time to play around with it, so I won't be of much help. Sorry.

mrand commented 5 years ago

Hmmm the "1.5+" phrase might be key here... I was originally thinking 1.5.xx when I read it.

@andrewjfreyer, from https://github.com/andrewjfreyer/monitor/issues/175#issuecomment-488001197, it looks like you're saying 1.6.2 would likely work for the machine running monitor?

For reference, I'm hoping to try this on my Wyse 7020 thin client that has the optional wifi/bluetooth installed.

andrewjfreyer commented 5 years ago

@andrewjfreyer, from #175 (comment), it looks like you're saying 1.6.2 would likely work for the machine running monitor?

Yes, the issue was fixed with 1.6.2 of mosquitto. Error introduced in 1.6.0, so anything before or after 1.6.2 will work.