search

agittins / bermuda

Bermuda Bluetooth/BLE Triangulation / Trilateration for HomeAssistant
MIT License
642 stars 17 forks source link

General help with tracking phone(s) #157

Closed gurkburk76 closed 7 months ago

gurkburk76 commented 7 months ago

Version of the custom_component

Latest as of writing

Configuration

I was hoping my bluetooth mac UUID would show up here, but it dosen't. image

Describe the bug

I'm having trouble making my esp32 to pickup my phone, i have HA companion on the phone (android) and setup a BLE sender with with a unique ID/UUID and all that. Phone: xiaomi T11

This is the esp chip i'm using and the esphome config of it, shortened down a bit:

Chip is ESP32-D0WDQ6 (revision v1.0)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None

--------------------------------------------
esphome:
  name: esphome-web-aadec8
  friendly_name: esphome-web-aadec8

esp32:
  board: esp32dev
  framework:
    type: esp-idf
    version: recommended

# Enable logging
logger:
  level: VERY_VERBOSE
  logs:
    esp32_ble: VERY_VERBOSE

#captive_portal:

# Define a simple HTML page to display the number of cores
# web_server:
#   port: 80
bluetooth_proxy:
  active: true

I've also tried the config with "framework arduino" with the same result.

I get a lot of bluetooth trafic / mac addresses in the debug log but not the mac address from my phone and no UUID from HA companion either.

I also installed ibeacon and same thing. would not work.

I'm not sure how/if i can get IRK (resolvable keys) to test that and see if that help, please advise.

small sample of the esp32 log:

[12:32:35][VV][esp32_ble_tracker:423]: Parse Result:
[12:32:35][VV][esp32_ble_tracker:440]:   Address: 08:3A:F2:7B:79:D2 (PUBLIC)
[12:32:35][VV][esp32_ble_tracker:442]:   RSSI: -85
[12:32:35][VV][esp32_ble_tracker:443]:   Name: ''
[12:32:35][VV][esp32_ble_tracker:451]:   Ad Flag: 6
[12:32:35][VV][esp32_ble_tracker:457]:   Manufacturer data: 01.05.00.0B.05.20.0A.D0.79.7B.F2.3A.08 (13)
[12:32:35][VV][esp32_ble_tracker:473]: Adv data: 02.01.06.10.FF.A9.0B.01.05.00.0B.05.20.0A.D0.79.7B.F2.3A.08 (20)
[12:32:35][VV][esp32_ble_tracker:423]: Parse Result:
[12:32:35][VV][esp32_ble_tracker:440]:   Address: 08:3A:F2:7B:79:D2 (PUBLIC)
[12:32:35][VV][esp32_ble_tracker:442]:   RSSI: -86
[12:32:35][VV][esp32_ble_tracker:443]:   Name: 'ShellyPro4PM-083AF27B79D0'
[12:32:35][VV][esp32_ble_tracker:451]:   Ad Flag: 6
[12:32:35][VV][esp32_ble_tracker:457]:   Manufacturer data: 01.05.00.0B.05.20.0A.D0.79.7B.F2.3A.08 (13)
[12:32:35][VV][esp32_ble_tracker:473]: Adv data: 02.01.06.10.FF.A9.0B.01.05.00.0B.05.20.0A.D0.79.7B.F2.3A.08.1A.09.53.68.65.6C.6C.79.50.72.6F.34.50.4D.2D.30.38.33.41.46.32.37.42.37.39.44.30 (47)
[12:32:35][W][component:232]: Component esp32_ble_tracker took a long time for an operation (66 ms).
[12:32:35][W][component:233]: Components should block for at most 30 ms.
[12:32:36][V][esp32_ble:314]: (BLE) gap_event_handler - 3
[12:32:36][VV][esp32_ble_tracker:423]: Parse Result:
[12:32:36][VV][esp32_ble_tracker:440]:   Address: 7A:9A:CA:D4:09:AF (RANDOM)
[12:32:36][VV][esp32_ble_tracker:442]:   RSSI: -70
[12:32:36][VV][esp32_ble_tracker:443]:   Name: ''
[12:32:36][VV][esp32_ble_tracker:454]:   Service UUID: 0xFE9F
[12:32:36][VV][esp32_ble_tracker:457]:   Manufacturer data: 13.8C.CA.7C.01.E5 (6)
[12:32:36][VV][esp32_ble_tracker:468]:   Service data:
[12:32:36][VV][esp32_ble_tracker:469]:     UUID: 0xFE9F
[12:32:36][VV][esp32_ble_tracker:470]:     Data: 00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00 (20)
[12:32:36][VV][esp32_ble_tracker:473]: Adv data: 03.03.9F.FE.17.16.9F.FE.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.09.FF.E0.00.13.8C.CA.7C.01.E5 (38)
[12:32:37][V][esp32_ble:314]: (BLE) gap_event_handler - 3
gurkburk76 commented 7 months ago

After more investigation it would seem like this is my phone according to esp32;

14:23:11][VV][esp32_ble_tracker:423]: Parse Result: [14:23:11][VV][esp32_ble_tracker:440]: Address: 6C:D9:12:C4:AF:EE (RANDOM) [14:23:11][VV][esp32_ble_tracker:442]: RSSI: -43 [14:23:11][VV][esp32_ble_tracker:443]: Name: '' [14:23:11][VV][esp32_ble_tracker:457]: Manufacturer data: 02.15.6F.AF.DA.7E.F6.6E.49.D3.83.BA.5C.70.64.9B.36.66.02.9A.00.42.C5 (23) [14:23:11][VV][esp32_ble_tracker:460]: iBeacon data: [14:23:11][VV][esp32_ble_tracker:461]: UUID: 66369B64-705C-BA83-D349-6EF67EDAAF6F [14:23:11][VV][esp32_ble_tracker:462]: Major: 666 [14:23:11][VV][esp32_ble_tracker:463]: Minor: 66 [14:23:11][VV][esp32_ble_tracker:464]: TXPower: -59 [14:23:11][VV][esp32_ble_tracker:473]: Adv data: 1A.FF.4C.00.02.15.6F.AF.DA.7E.F6.6E.49.D3.83.BA.5C.70.64.9B.36.66.02.9A.00.42.C5 (27)

(the major and minor id helped me finding it as i set these in the app)

Thing is, in HA companion app the UUID is 6FAFDA7E-F66E-49D3-83BA-5C70649B3666

i don't understand, regardless i can't find this UUID in the options to select it.

agittins commented 7 months ago

Hmmm, can you post the output of the dump_devices service call? In HA, go into developer tools, services and find it on the list there. If there's too much to post (it might be huge) you can just post a chunk of it, or you can try filtering it to a few chosen addresses (click the "fill example data" button to get started).

gurkburk76 commented 7 months ago

This is strange, i tried to dump without any example data filled in, did not work, then i filled in the example data and that gav me nothing, this is part of a debug log i enabled on the component:

Note that i don't have any devices added in the config (picture in the first post) i then added a few, mostly shelly devices that act as bluetooth relays, attacking that as well in the log.

2024-04-11 15:32:28.276 ERROR (MainThread) [homeassistant.helpers.script.websocket_api_script] websocket_api script: Error executing script. Unexpected error for call_service at pos 1: too many values to unpack (expected 2)
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 485, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 723, in _async_call_service_step
    response_data = await self._async_run_long_action(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 685, in _async_run_long_action
    return long_task.result()
           ^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2319, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2356, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/bermuda/__init__.py", line 1385, in service_dump_devices
    out[address] = device.to_dict()
                   ^^^^^^^^^^^^^^^^
  File "/config/custom_components/bermuda/__init__.py", line 599, in to_dict
    scanout[address] = scanner.to_dict()
                       ^^^^^^^^^^^^^^^^^
  File "/config/custom_components/bermuda/__init__.py", line 483, in to_dict
    for uuid, thebytes in self.adverts:
        ^^^^^^^^^^^^^^
ValueError: too many values to unpack (expected 2)
2024-04-11 15:32:28.286 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140394470597824] Error handling message: Unknown error (unknown_error) admin from 192.168.20.74 (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36)
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/decorators.py", line 26, in _handle_async_response
    await func(hass, connection, msg)
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 795, in handle_execute_script
    script_result = await script_obj.async_run(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 1600, in async_run
    return await asyncio.shield(run.async_run())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 435, in async_run
    await self._async_step(log_exceptions=False)
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 487, in _async_step
    self._handle_exception(
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 512, in _handle_exception
    raise exception
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 485, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 723, in _async_call_service_step
    response_data = await self._async_run_long_action(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 685, in _async_run_long_action
    return long_task.result()
           ^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2319, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2356, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/bermuda/__init__.py", line 1385, in service_dump_devices
    out[address] = device.to_dict()
                   ^^^^^^^^^^^^^^^^
  File "/config/custom_components/bermuda/__init__.py", line 599, in to_dict
    scanout[address] = scanner.to_dict()
                       ^^^^^^^^^^^^^^^^^
  File "/config/custom_components/bermuda/__init__.py", line 483, in to_dict
    for uuid, thebytes in self.adverts:
        ^^^^^^^^^^^^^^
ValueError: too many values to unpack (expected 2)
2024-04-11 15:32:34.048 ERROR (MainThread) [homeassistant.helpers.script.websocket_api_script] websocket_api script: Error executing script. Unexpected error for call_service at pos 1: too many values to unpack (expected 2)
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 485, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 723, in _async_call_service_step
    response_data = await self._async_run_long_action(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 685, in _async_run_long_action
    return long_task.result()
           ^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2319, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2356, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/bermuda/__init__.py", line 1385, in service_dump_devices
    out[address] = device.to_dict()
                   ^^^^^^^^^^^^^^^^
  File "/config/custom_components/bermuda/__init__.py", line 599, in to_dict
    scanout[address] = scanner.to_dict()
                       ^^^^^^^^^^^^^^^^^
  File "/config/custom_components/bermuda/__init__.py", line 483, in to_dict
    for uuid, thebytes in self.adverts:
        ^^^^^^^^^^^^^^
ValueError: too many values to unpack (expected 2)
2024-04-11 15:32:34.057 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140394470597824] Error handling message: Unknown error (unknown_error) admin from 192.168.20.74 (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36)
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/decorators.py", line 26, in _handle_async_response
    await func(hass, connection, msg)
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 795, in handle_execute_script
    script_result = await script_obj.async_run(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 1600, in async_run
    return await asyncio.shield(run.async_run())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 435, in async_run
    await self._async_step(log_exceptions=False)
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 487, in _async_step
    self._handle_exception(
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 512, in _handle_exception
    raise exception
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 485, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 723, in _async_call_service_step
    response_data = await self._async_run_long_action(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 685, in _async_run_long_action
    return long_task.result()
           ^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2319, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2356, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/bermuda/__init__.py", line 1385, in service_dump_devices
    out[address] = device.to_dict()
                   ^^^^^^^^^^^^^^^^
  File "/config/custom_components/bermuda/__init__.py", line 599, in to_dict
    scanout[address] = scanner.to_dict()
                       ^^^^^^^^^^^^^^^^^
  File "/config/custom_components/bermuda/__init__.py", line 483, in to_dict
    for uuid, thebytes in self.adverts:
        ^^^^^^^^^^^^^^
ValueError: too many values to unpack (expected 2)

and

ents.bermuda] Could not discern area from scanner shellyplusi4-a8032ab0c8fc (A8:03:2A:B0:C8:FC): None.Please assign an area then reload this integration
2024-04-11 15:34:07.987 WARNING (MainThread) [custom_components.bermuda] Could not discern area from scanner shellyplusi4-a8032ab0c8fc (A8:03:2A:B0:C8:FC): None.Please assign an area then reload this integration
2024-04-11 15:34:07.988 WARNING (MainThread) [custom_components.bermuda] Could not discern area from scanner shellyplus1pm-441793cefa7c (44:17:93:CE:FA:7C): None.Please assign an area then reload this integration
2024-04-11 15:34:07.988 WARNING (MainThread) [custom_components.bermuda] Could not discern area from scanner shellyplus1pm-441793cefa7c (44:17:93:CE:FA:7C): None.Please assign an area then reload this integration
2024-04-11 15:34:07.988 WARNING (MainThread) [custom_components.bermuda] Could not discern area from scanner shellypro4pm-083af27b79d0 (08:3A:F2:7B:79:D0): None.Please assign an area then reload this integration
2024-04-11 15:34:07.988 WARNING (MainThread) [custom_components.bermuda] Could not discern area from scanner shellypro4pm-083af27b79d0 (08:3A:F2:7B:79:D0): None.Please assign an area then reload this integration
2024-04-11 15:34:07.989 WARNING (MainThread) [custom_components.bermuda] Could not discern area from scanner shellypro4pm-083af27b79d0 (08:3A:F2:7B:79:D0): None.Please assign an area then reload this integration
2024-04-11 15:34:07.989 WARNING (MainThread) [custom_components.bermuda] Could not discern area from scanner shellyplus1pm-441793cefa7c (44:17:93:CE:FA:7C): None.Please assign an area then reload this integration
2024-04-11 15:34:07.995 WARNING (MainThread) [custom_components.bermuda] No area name for while updating scanner Shellypro4pm-1
2024-04-11 15:34:07.995 WARNING (MainThread) [custom_components.bermuda] No area name for while updating scanner shellyplusi4-a8032ab0c8fc
2024-04-11 15:34:07.996 WARNING (MainThread) [custom_components.bermuda] No area name for while updating scanner shellyplus1pm-441793cefa7c
2024-04-11 15:34:08.008 WARNING (MainThread) [custom_components.bermuda] Could not discern area from scanner shellyplusi4-a8032ab0c8fc (A8:03:2A:B0:C8:FC): None.Please assign an area then reload this integration
2024-04-11 15:34:08.008 WARNING (MainThread) [custom_components.bermuda] Could not discern area from scanner shellyplusi4-a8032ab0c8fc (A8:03:2A:B0:C8:FC): None.Please assign an area then reload this integration
2024-04-11 15:34:08.008 WARNING (MainThread) [custom_components.bermuda] Could not discern area from scanner shellyplusi4-a8032ab0c8fc (A8:03:2A:B0:C8:FC): None.Please assign an area then reload this integration
2024-04-11 15:34:08.008 WARNING (MainThread) [custom_components.bermuda] Could not discern area from scanner shellyplusi4-a8032ab0c8fc (A8:03:2A:B0:C8:FC): None.Please assign an area then reload this integration
2024-04-11 15:34:08.009 WARNING (MainThread) [custom_components.bermuda] Could not discern area from scanner shellyplusi4-a8032ab0c8fc (A8:03:2A:B0:C8:FC): None.Please assign an area then reload this integration
2024-04-11 15:34:08.009 WARNING (MainThread) [custom_components.bermuda] Could not discern area from scanner shellyplus1pm-441793cefa7c (44:17:93:CE:FA:7C): None.Please assign an area then reload this integration
2024-04-11 15:34:08.009 WARNING (MainThread) [custom_components.bermuda] Could not discern area from scanner shellyplus1pm-441793cefa7c (44:17:93:CE:FA:7C): None.Please assign an area then reload this integration
2024-04-11 15:34:08.009 WARNING (MainThread) [custom_components.bermuda] Could not discern area from scanner shellypro4pm-083af27b79d0 (08:3A:F2:7B:79:D0): None.Please assign an area then reload this integration
agittins commented 7 months ago

D'oh! Looks like there's a bug in the dump_devices - it won't be related to your issue, so I've raised a separate issue for that.

This:

2024-04-11 15:34:07.987 WARNING (MainThread) [custom_components.bermuda] Could not discern area from scanner shellyplusi4-a8032ab0c8fc (A8:03:2A:B0:C8:FC): None.Please assign an area then reload this integration
2024-04-11 15:34:07.988 WARNING (MainThread) [custom_components.bermuda] Could not discern area from scanner shellyplus1pm-441793cefa7c (44:17:93:CE:FA:7C): None.Please assign an area then reload this integration
2024-04-11 15:34:07.988 WARNING (MainThread) [custom_components.bermuda] Could not discern area from scanner shellyplus1pm-441793cefa7c (44:17:93:CE:FA:7C): None.Please assign an area then reload this integration
2024-04-11 15:34:07.988 WARNING (MainThread) [custom_components.bermuda] Could not discern area from scanner shellypro4pm-083af27b79d0 (08:3A:F2:7B:79:D0): None.Please assign an area then reload this integration
2024-04-11 15:34:07.988 WARNING (MainThread) [custom_components.bermuda] Could not discern area from scanner shellypro4pm-083af27b79d0 (08:3A:F2:7B:79:D0): None.Please assign an area then reload this integration
2024-04-11 15:34:07.989 WARNING (MainThread) [custom_components.bermuda] Could not discern area from scanner shellypro4pm-083af27b79d0 (08:3A:F2:7B:79:D0): None.Please assign an area then reload this integration
2024-04-11 15:34:07.989 WARNING (MainThread) [custom_components.bermuda] Could not discern area from scanner shellyplus1pm-441793cefa7c (44:17:93:CE:FA:7C): None.Please assign an area then reload this integration
2024-04-11 15:34:07.995 WARNING (MainThread) [custom_components.bermuda] No area name for while updating scanner Shellypro4pm-1

Indicates you don't have your proxies allocated to Areas, which is a pre-requisite for the area-based tracking to work. I don't know if that might be causing a wider issue for you, it may be worth checking that at least a few of them have an Area defined.

agittins commented 7 months ago

Can you update to "main" by going into HACS and choosing "Redownload" from the top-right menu? It should fix the dump_devices service, so hopefully we can get some more info on what your system is seeing in the backend.

gurkburk76 commented 7 months ago

Yeah, i'll evetually get to placing my sensors in HA, but right now i'll be really happy if i could actually track my phone as a starter ;) Here is the dump, nothing i need to hide from it from what i could find you you might as well get the whole shabang :)

big_bermuda_dump.zip

agittins commented 7 months ago

The reason I suggest adding proxies to areas is that I don't know if things will work properly otherwise, it's a fault-finding step.

The dump looks good, plenty of devices showing up, and they're being tagged with their proxies (albeit most without areas). I can't identify your android phone in there, though, and I can't see any iBeacon adverts.

Are Beacon1 and Beacon3 esp32 proxies, or are they shelly's as well?

It's a bit weird that the iBeacon integration isn't seeing it either, although it can be a bit more finicky at times.

Can you screenshot the beacon settings in your companion app? Not sure I'll see anything helpful but just trying to check off all the options, it's really weird that it's not showing up. Do you know which proxies should be in range? Between the i4, 1pm and two 4pm's, plus Beacon1 and Beacon3 I'd expect something should be seeing it, if it's transmitting (the rssi in your previous esp32 log did look pretty weak though, maybe it's sending at a very low power level - still odd to not receive anything).

I have to head off to sleep, I'll see if I can think of anything else after that, but feel free to dump any more info in here in the meantime!

agittins commented 7 months ago

BTW, the easiest (only?) way I found so far to get IRK for androids is to flash this arduino project to an esp32 and monitor the serial console: https://github.com/fryefryefrye/Decoding-Random-Bluetooth-Address - then you pair your phone to it (it shows up as ESP_BLE_SECURITY) and the console will spit out exactly what you need to put in to the Private BLE Device integration.

gurkburk76 commented 7 months ago

I think i finally got it working, i removed my esp32 from ha and redid the config. i might have forgotten to add it to setting/devices & services/esp home before so that could be why it wasen't getting registered.

Now to figure out if those shellys/other esp devices also play ball or not. i should not have to add the shellies as devices to the esphome addon should i? (in order for the bluetooth relay thing in them to work with bermuda)

EDIT: Seems like i have to run this script on the shelly, not sure if it's mandatory but i don't think it hurts :-)

// aioshelly BLE script 2.0
const queueServeTimer = 100; // in ms, timer for events emitting
const burstSendCount =  5; // number if events, emitted on timer event
const maxQueue =  32; // if the queue exceeds the limit, all new events are ignored until it empties
const packetsInSingleEvent = 16; // max number of packets in single event

let queue = [];
let timerHandler = null;

function timerCallback() {
  for(let i = 0; i < burstSendCount; i++) {
    if (queue.length <= 0) {
      break;
    }

    Shelly.emitEvent(
      "ble.scan_result", [
        2,
        queue.slice(0, packetsInSingleEvent),
      ]
    );
    queue = queue.slice(packetsInSingleEvent);
  }

  timerHandler = null;
  if (queue.length > 0) {
    timerHandler = Timer.set(queueServeTimer, false, timerCallback);
  }
}

function bleCallback(event, res) {
  if (event !== BLE.Scanner.SCAN_RESULT) {
    return
  }

  if (queue.length > maxQueue) {
    return;
  }

  queue.push([
    res.addr,
    res.rssi,
    btoa(res.advData),
    btoa(res.scanRsp)
  ]);

  if(!timerHandler) {
    timerHandler = Timer.set(queueServeTimer, false, timerCallback);
  }
}

// Skip starting if scanner is active
if (!BLE.Scanner.isRunning()) {
  BLE.Scanner.Start({
    duration_ms: -1,
    active: true,
    interval_ms: 320,
    window_ms: 30,
  });
}

BLE.Scanner.Subscribe(bleCallback);
agittins commented 7 months ago

Glad you're finding some success!

Can you possibly send the full config you're using on your esp32? You're right that if the device hadn't been added into the esphome integration in HA then it wouldn't be reporting any bluetooth traffic, so it's possible that the Shellys were just not reporting iBeacons, perhaps? Not sure. I don't have any Shelly devices so I don't know how they're configured or if there's anything required at the HA end - folks seem to find they "just work" but maybe they did some level of configuration first.

gurkburk76 commented 7 months ago

looks like you need to run the script on the shellies or they don't relay traffic, that's my experience anyway. this is my full conf for esp32, it's likely enought to set it as a bt relay and it's fine.

esphome:
  name: beacon2
  friendly_name: beacon2

esp32:
  board: esp32dev
  framework:
    type: arduino

# Enable logging
logger:

# Enable Home Assistant API
api:
  encryption:
    key: REDACTED

ota:
  password: REDACTED

wifi:
  networks:
  - ssid: !secret wifi_ssid
    password: !secret wifi_password

  - ssid: !secret wifi_ssid
    password: !secret wifi_password
    hidden: true
  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "Beacon2 Fallback Hotspot"
    password: REDACTED

captive_portal:

web_server:
  port: 80

bluetooth_proxy:
  active: true

switch:
  - platform: restart
    name: beacon1_WiFi-restart
    id: restart_switch
gurkburk76 commented 7 months ago

I usually run mt iot ssid hidden just to to clog up the air so to speak, this config makes connecting to it when it's actually hidden quicker, sometimes i need to unhide it when joining with shitty devices that have bad support for hidden ssid's and that's that the other wifi config is for, might be unneccesary but it works fine so... :)

gurkburk76 commented 7 months ago

Now, distance is another issue and will take a while to get right. right now it gives me 10m when the phone is 1m away from a beacon in a clear los so... oh well :)

agittins commented 7 months ago

Yep, your esphome config looks good. You're using the defaults for the esp32_ble_tracker's scan parameters though, which means your proxy only spends about 10% of its time actually listening for broadcasts. This still "works" of course, but if you might be missing a lot of packets that you want to capture more regularly, I've found reasonable results with:

esp32_ble_tracker:
  scan_parameters:
    interval: 1000ms
    window: 900ms

Note that this isn't the officially suggested setting, and might cause issues. In particular, I'd consider disabling the ap, web_server and captive_portal components before doing this, as the less other stuff the esp has to do the less likely it is to crash with such tight scan timings (the difference between those two values is the amount of time the esp gets to do everything else, like keeping the wifi connection up, talking to HA etc).

Re distance, yep - it's all covered in the readme. Everything's relative, but you can calibrate the settings to get a reasonable approximation for a given device/proxy combination.

gurkburk76 commented 7 months ago

SO i was thinking a bit. The aproach we're taking here is having the esp's listen to broadcast from the phone. If i want i can adjust the output signal of the ble on the phone so it's more effective, my device gets "noticed" byt the esp's more, bette tracking. Bad for my phones battery.

I think we can use another aproach and that would be letting the esp's blast out ble signals and let the phone listen to them, we might possibly have the option to boost the ble signal from the ESP somehow (would need to check esphome docs), and since we're already hooked up to the powergrid the batt. time is not something we need to worry about.

Is this doable with bermuda? I've assuming we would need to pick the ble deices, esp, shelly etc. in the config and let the phones report in via ble monitoring in the companion app.

agittins commented 7 months ago

That would almost be another project entirely, and there are a few like that around (in research etc). There are a few issues with the idea generally, and some with it in Bermuda specifically. It's not a bad way to go, but it's not one that I have much interest in pursuing, but I'll explain why because it's not a bad idea on the face of it.

So, the companion app warns you about increased battery usage, which is technically not a lie. But it's a lie. Just set advertise mode to "balanced 3Hz" and transmitter power to Medium or High, and see if it does make any noticeable difference in consumption. I think the "Medium" setting on my pixel 6 makes a big difference in rssi, and negligible impact on battery life. I think I saw a bit more battery drain on high, but it didn't seem to bump the rssi by very much, so Medium is my "sweet spot" - each phone might be different on that.

I do find the default "ultra low" power setting to be too weak for my purposes, but this also depends on how many proxies one has around and what boards they are etc.

The big improvement I find with reliable detection is what I said above about fixing the window and interval settings - it doesn't matter much how many megawatts you transmit with if the receiver is only turned on for 10% of the time, in 30ms snippets!

An advantage of listening at the fixed stations and doing all the calculations centrally is that you can leverage the readings from everything in the area to get better estimates, so to still centralise you'd need the phone to continually send (via wifi) the details of what it has received (effectively becoming a proxy itself), which will drain the battery WAY more than sending the tiny broadcast packets in the current setup.

We already have the power-hungry stuff happening at the esp - it's the continuous listening and wifi retransmission of the receipts. The devices broadcasting actually have the easier (power-wise) role - they wake up, spend a couple of milliseconds and a picowatthour or whatever sending three packets (one on each bt advert channel) then goes back to sleep for 300millisends (or whatever their setup is). This takes very little current - like, a year of life for a single CR2032 cell. That same cell might power your phone for, 3 seconds maybe, then be dead. The beacon power budget for phones is, imo, a non-issue.

It might work out ok if one phone was the only thing you wanted to track, but honestly I think this is why so many of the university research papers I've seen had crappy results - I think a network of receivers is more powerful than a single receiver listening to a network of transmitters. Also, the phone will inherently be a bad receiver because it has zero stability in received signal strength - being carried about by a hairy bag of water (or in the pocket of one), and placed in random places etc, it has a continuously varying RF environment. But the static proxies have (comparatively) a much more stable environment, and benefit from being able to see other static devices so I think there are better opportunities to normalise and smooth distance readings, since we can infer more things about their environment when all signals change vs some signals changing etc.

The current esphome code doesn't support sending broadcasts at the same time as being a proxy. This is a bit of a shame, actually. The Shelly devices do however, which is a nice benefit. Hopefully this can be solved at some point, I can't remember if I saw an issue for it in esphome already, but I might raise one when I have a chance to check into it (I'm interested in this because it will give us an easier/quicker way to measure distance between each proxy). At any rate, we currently can't have (esphome) proxies acting as both beacon and scanner at the same time, and being able to deploy a large network of receivers is exactly what makes Bermuda so powerful, in my opinion - with no other requirements we can track any ble device that wanders by.

All that aside, if the companion app supported having a phone act as a bluetooth scanner and send its receipts in to HA the same way esphome bluethooth_proxy's do, that would be great, and it would just work in Bermuda. It would mean having to exclude that device from the calculation of certain things, but I am planning to support doing that anyway, since mobile proxies is a definite potential use-case. Again though, I think this would massively increase battery drain on the phone.

So in short(!)... I think doing it would use much more battery on the phone, but if the HA companion app wanted to it could certainly do it (I don't know how hard it would be for them, though - it would depend on the libs they're using) but it's not something I have a strong desire to pursue, currently.

agittins commented 7 months ago

Since I think your original issue might be resolved now I will close this ticket. If you run into anything on the same topic feel free to re-open it, or raise a fresh issue if you run into anything else.

If you haven't already, you might also want to check out the HA community thread at https://community.home-assistant.io/t/bermuda-bluetooth-ble-room-presence-and-tracking-custom-integration/625780, there's some interesting things discussed there as well.

gurkburk76 commented 7 months ago

Just gonna leave this here, it seems that changing platform can make the esp32 work "better" so that's good to know :) https://esphome.io/components/bluetooth_proxy