Implement inter-proxy measurements

[agittins]

Inter-proxy measurements are a pre-requisite for trilateration.

There are two methods we will use to make these measurements:

Triangle Inequality

• Esphome (until (checks watch) NOW) can't be a beacon and a proxy at the same time. This is now fixed, but I still want to support the case.
• If you have two sides of a triangle, you can know that the third side is no more than the sum of the two known sides.
• This means we can trawl through each pair of proxies, and successively approximate their distance apart by taking the sums of their measurement to each known device. This should be fairly quick to do, basically:

  foreach proxy1 in proxies:
  foreach proxy2 in proxies:
  foreach device in devices:
     estimate = proxy1_to_device + proxy2_to_device
     if estimate < proxy1.estimate_to_proxy2:
       proxy1.estimate_to_proxy2 = estimate

  (obviously this is not real code and lacks serious optimisation)

Direct rssi measurement

• esphome as of the last few days (as at July 24) can now act as a beacon at the same time as being a proxy. This is something that the Shelly devices have always done, it seems.
• This simplifies the inter-proxy measurement, as we can apply it directly.
• I think we still need to track it as... let's call it an "ephemeral minima".
• Still undecided if a triangle inequality measurement should be able to trump a direct, so we should collect each estimate in its own field (so we have diagnostic visibility) and apply one as the current "blessed" value.

[jleinenbach]

What do we need to upgrade - the ESPHome integration, ESPHome firmware (do we need to activate something?) or ESPHome server? How can I check if this is available - my ESPHome proxies don't have any entities, so is it "under the hood"?

After some research: You'll add this option to your ESPHome device as described here: https://esphome.io/components/esp32_ble_beacon.html And don't forget to use a different uuid for each device, you'll find a generator there, just reload the page.

Then verify your settings. You have the old version if you get this message: "Component esp32_ble_beacon cannot be used together with component esp32_ble_tracker."

BTW: Some devices transmit their own signal RSSI value at 1m for iOS – this may be potentially useful for calculations as well (if true).

[asucrews]

What do we need to upgrade - the ESPHome integration, ESPHome firmware (do we need to activate something?) or ESPHome server? How can I check if this is available - my ESPHome proxies don't have any entities, so is it "under the hood"?

After some research: You'll add this option to your ESPHome device as described here: https://esphome.io/components/esp32_ble_beacon.html And don't forget to use a different uuid for each device, you'll find a generator there, just reload the page.

Then verify your settings. You have the old version if you get this message: "Component esp32_ble_beacon cannot be used together with component esp32_ble_tracker."

BTW: Some devices transmit their own signal RSSI value at 1m for iOS – this may be potentially useful for calculations as well (if true).

The necessary pull request for esp32_ble_beacon and esp_32_tracker has been merged into the ESPHome dev branch but has not yet made its way into the official version. I am guessing it will be included in the 2024.8.0 release, but it could be sooner.

You could use the Docker image that follows the dev branch of ESPHome if you want, but I would recommend caution as it changes frequently.

You can follow ESPHome changelog here https://esphome.io/changelog/

In theory on ESPHome side you just need to add the below code to YAML, once the code is merged into official version. A Totally different story on the bermuda side of things.

# Example configuration entry
esp32_ble_beacon:
  type: iBeacon
  uuid: 'c29ce823-e67a-4e71-bff2-abaa32e77a98'

[jleinenbach]

It worked for me, but although I used ESPHome 2024.8.0-dev, I had to use an old "recommended" version of the framework. "latest" and "dev" didn't compile:

framework: type: esp-idf version: recommended

[jleinenbach]

max_interval (Optional, [Time](https://esphome.io/guides/configuration-types#config-time)): The iBeacon maximum transmit interval in milliseconds from 20 to 10240. Setting this greater than min_interval gives the BLE hardware a better chance to avoid collisions with other BLE transmissions. Defaults to the iBeacon specification’s defined interval: 100ms.

I guess we shouldn't touch these values.

Proximity beacons must broadcast the entire 30 byte advertising packet in all Advertising frequencies using a fixed 100 ms advertising interval. https://developer.apple.com/ibeacon/

[droans]

You could use the Docker image that follows the dev branch of ESPHome if you want, but I would recommend caution as it changes frequently.

Alternatively, you can use an external_component.

Assuming this is the PR, I believe you can just add this to your code:

external_components:
  - source: github://pr#6908
    components:
      - esp32_ble
      - esp32_ble_beacon
      - esp32_ble_server
      - esp32_improv
    refresh: 1d

The exact components might need to be changed - I just based it off of what's in the PR. However, I don't think there's any value for us since Bermuda doesn't support it yet.

[agittins]

since Bermuda doesn't support it yet.

Well... it probably does :-) If you deploy a proxy with the beacon enabled it should show up with distance sensors to the various other proxies.

I haven't tested it myself yet, but I haven't had reports of weird stuff hapenning with Shellys, and they've operated like that from the beginning, it seems.

Any reports of weird are welcome, and I plan to test using both the dev version and the custom-component methods once I'm back home later this week.

[agittins]

BTW: Some devices transmit their own signal RSSI value at 1m for iOS – this may be potentially useful for calculations as well (if true).

Yes, I can't remember if I already included that or not - Bermuda does track the value, but I think I didn't bother using it because most devices seem to include a not-very-useful default. I'll revisit that when we do per-device settings, as we might then want to use the advertised tx_power in the absense of a configured value (but we'd always allow overriding it in Bermuda, as some devices are not configurable, plus it's easier to do in Bermuda than on each device).

guess we shouldn't touch these values.

Proximity beacons must broadcast the entire 30 byte advertising packet in all Advertising frequencies using a fixed 100 ms advertising interval. https://developer.apple.com/ibeacon/

Luckily we don't need to follow Apple's spec for Bermuda's purposes. In fact we don't even need it to be an iBeacon at all, any advert would work. Far better is to add a small random delay between advert cycles (which is what the bluetooth spec says to do) and the interval can be quite widely varied for most systems.

For Bermuda, it won't mind at all up to the timeout values, but I'd personally go with perhaps 1 to 2 seconds, or do some testing to see how it affects receipt of adverts, and maybe increase it to 10 or 30 seconds if it causes the proxy to miss too many adverts. Bermuda will probably change to treat static proxy adverts differently to consider them 'fresh' for a much longer time than device adverts, since we expect proxies (typically) to be stationary.

[jleinenbach]

Well... it probably does :-) If you deploy a proxy with the beacon enabled it should show up with distance sensors to the various other proxies.

I haven't tested it myself yet, but I haven't had reports of weird stuff hapenning with Shellys, and they've operated like that from the beginning, it seems.

Works for me. ESP32-S3 and Shellys:

[agittins]

Heh, I'm guessing the 'Area' sensor is probably just confusing, because it says the proxy is at the same area as its nearest proxy, not the area it's actually in, yes?

[jleinenbach]

"Windfang" is the area of the nearest proxy, yes. In the "Technikraum" is another proxy, directly one floor below. The signal passes through the wooden floor, the underfloor heating, and the reinforced concrete, yet remains surprisingly strong.

[Lash-L]

Proof of concept implemented in coordinator.py:

    def _calculate_inter_proxy_distance(self):
        for scanner_1_address in self.scanner_list:
            scanner_1_device = self.devices[scanner_1_address]
            for scanner_2_address in self.scanner_list:
                real_dist = None
                if scanner_2_address in scanner_1_device.scanners:
                    real_dist = scanner_1_device.scanners[
                        scanner_2_address
                    ].rssi_distance
                lowest_estimated_dist = 99999
                if scanner_1_address != scanner_2_address:
                    for device_address, device in self.devices.items():
                        if (
                            device_address not in {scanner_1_address, scanner_2_address}
                            and scanner_1_address in device.scanners
                            and scanner_2_address in device.scanners
                        ):
                            dist_1 = device.scanners[scanner_1_address].rssi_distance
                            dist_2 = device.scanners[scanner_2_address].rssi_distance
                            if dist_1 is not None and dist_2 is not None:
                                estimated_dist = dist_1 + dist_2
                                if estimated_dist < lowest_estimated_dist:
                                    lowest_estimated_dist = estimated_dist
                print(
                    f"Distance from {scanner_1_address} to {scanner_2_address}: Real: {real_dist}, estimated: {lowest_estimated_dist}"
                )