tesla-local-control / tesla-local-control-addon

Control your Tesla locally from HomeAssistant
Apache License 2.0
21 stars 2 forks source link

[ble presence] Run if enable, BLE_MAC calculate & rm config. opt [0.0.8a][ready] #25

Open baylanger opened 3 months ago

baylanger commented 3 months ago

. Remove BLE_MAC option from configuration . Added function to calculate BLE_MAC . Added option to enable or not BLE presence detection . Added DEBUG and BLE_PRESENCE_ENABLE to standalone

Fixed #26

This is again part of splitting the big PR #16 into multiple smaller PRs.

BogdanDIA commented 3 months ago

Just one minor note, the S...C string is not a BLE MAC. In BLE it is named BLE Local Name. Hence I suggest changing it to BLE_LNAME to not be misleading.

baylanger commented 3 months ago

Just one minor note, the S...C string is not a BLE MAC. In BLE it is named BLE Local Name. Hence I suggest changing it to BLE_LNAME to not be misleading.

I didn't have the time to read the multiple Bluetooth & BLE bibles. In the current PRs the name of the var was renamed to BLE_SC_NAME. For Bluetooth newbies to know the proper name, I propose BLE_LOCAL_NAME. I struggle last night with my Search engine to figure it out... Thanx for pointing us to the right name 👍

We haven't figure out yet how to scan for LE devices yet but below some good info.

BLE Advertisement Interval

The time interval between packet set has both a fixed interval and a random delay. {Snip} fixed interval from 20ms to 10.24 seconds, in steps of 0.625ms. The random delay is a pseudo-random value from 0ms to 10ms that is automatically added.

  • The current bluetoothctl TIMEOUT is set at 5 seconds.
  • Using an app like "nRF Connect" , we can figure out the time that Tesla currently uses and adjust the TIMEOUT accordingly. With a 5s timeout, if the 1st packet isn't received, we can hope the 2nd or (3rd?) one will make it. If the vehicle is far from the BLE dongle, the TIMEOUT can have a default value 5s but we should probably allow the end user to adjust it.

BLE Advertiser Address

Mandatory identifier The advertiser address is the one and only mandatory identifier in a BLE advertising packet. It has the form 12:34:56:78:9a:bc

  • iOS nRF Connect only shows the local name. Trying to "connect" doesn't work. BLE Scanner doesn't show either the advert. addr but can show the vehicle's name and the UUID. Does this work because my phone is paired with the vehicle? I will try this later app on my iPad (not paired to the car) and see what shows up. Will also try to find an app that might show all info.
  • Android EFR Connect & BLE Scanner, can display the advert. addr. - Can it display the vehicle's name? @raphmur Anyone with an old Android phone not paired to their car that can confirm the advert. addr and perhaps the car's name can be displayed w/ a device that isn't paired to the vehicle?

Ref:

baylanger commented 3 months ago

And there's this known Bluetooth adapter bug ... perhaps fixed in latest HAOS (12.4).

Not sure if this is why the "send key" wasn't working and perhaps this is why bluetoothctl wasn't able to do any discovery... I thought it was because "bluetoothclt" and "tesla-control" ran and caused a racing condition 🤔 I did update, I'll definitely run those 2 apps and try to cause the driver to fail... hopefully it is now stable.

BogdanDIA commented 3 months ago

Just one minor note, the S...C string is not a BLE MAC. In BLE it is named BLE Local Name. Hence I suggest changing it to BLE_LNAME to not be misleading.

I didn't have the time to read the multiple Bluetooth & BLE bibles. In the current PRs the name of the var was renamed to BLE_SC_NAME. For Bluetooth newbies to know the proper name, I propose BLE_LOCAL_NAME. I struggle last night with my Search engine to figure it out... Thanx for pointing us to the right name 👍

We haven't figure out yet how to scan for LE devices yet but below some good info.

BLE Advertisement Interval

The time interval between packet set has both a fixed interval and a random delay. {Snip} fixed interval from 20ms to 10.24 seconds, in steps of 0.625ms. The random delay is a pseudo-random value from 0ms to 10ms that is automatically added.

* The current bluetoothctl TIMEOUT is set at 5 seconds.

* Using an app like "nRF Connect" , we can figure out the time that Tesla currently uses and adjust the TIMEOUT accordingly. With a 5s timeout, if the 1st packet isn't received, we can hope the 2nd or (3rd?) one will make it. If the vehicle is far from the BLE dongle, the TIMEOUT can have a default value 5s but we should probably allow the end user to adjust it.

BLE Advertiser Address

Mandatory identifier The advertiser address is the one and only mandatory identifier in a BLE advertising packet. It has the form 12:34:56:78:9a:bc

* iOS `nRF Connect` only shows the local name. Trying to "connect" doesn't work. `BLE Scanner` doesn't show either the advert. addr but can show the vehicle's name and the UUID. Does this work because my phone is paired with the vehicle? I will try this later app on my iPad (not paired to the car) and see what shows up. Will also try to find an app that might show all info.

* Android `EFR Connect` & `BLE Scanner`, can display the advert. addr. - Can it display the vehicle's name? @raphmur  Anyone with an old Android phone not paired to their car that can confirm the advert. addr and perhaps the car's name can be displayed w/ a device that isn't paired to the vehicle?

Ref:

* [Silabs ](https://docs.silabs.com/bluetooth/2.13/general/adv-and-scanning/bluetooth-adv-data-basics)

* [Argenox primer](https://www.argenox.com/library/bluetooth-low-energy/ble-advertising-primer/)

* [punchthrough](https://punchthrough.com/android-ble-guide/?utm_source=BlogEmail&utm_medium=Email&utm_campaign=BlogRoundUp&mc_cid=9113555d36&mc_eid=767bbcdc1b)

There are two advertised values of concern:

The BLE advertiser address is the MAC address of the BLE. It must be there after scan all the time. The BLE Local Name is not mandatory to be advertised (according to the standard) but Tesla chooses to advertise it.

Tesla car sends also iBeacons that contain more info (UUID, Minor no, Major no, etc) but it is not necessary to be scanned for as we just need to find only if the car is close or not.

As I said the scanning should be passive:

bluetoothctl --timeout 60 scan on > scan-results.log

Normally each scan results line should contain MAC and BLE_LOCAL_NAME but depending on the version of the bluetoothctl/OS type, etc, sometimes the BLE_LOCAL_NAME is not obtained even though it is advertised. In this case only the MAC is shown, maybe together with RSSI. This makes things a bit more complex.

So for each of the MACs in scan-results.log do:

bluetoothctl info $MAC

This will give you among other things the BLE_LOCAL_NAME on the first line and you can use it to check against the car's one:

Device 11:22:33:44:55:66 (public) Name: S11223344556677C Alias: S11223344556677C Paired: no ... RSSI: -84

The info command above should be executed while scanning otherwise it will give you unknown device (on rpi ). On some systems like Ubuntu, BlueZ will cache the MAC and BLE_LOCAL_NAME and info command will succeed all the time even when scan is off. But we need to make sure it works on all the systems so info command needs to be executed while scanning. And for that, scanning and info command should have each a separate thread/process.

If you want to go and get car's dash name, connect to the car and in this case the scan is active by interrogating the device and resolving BLE services.

bluetoothctl --timeout 5 connect $MAC

This will give to you something like this:

Device 11:22:33:44:55:66 Name: 🔑 TeslaDashName Device 11:22:33:44:55:66 Alias: 🔑 TeslaDashName Device 11:22:33:44:55:66 Appearance: 0x08c1 Device 11:22:33:44:55:66 Name: S11223344556677C Device 11:22:33:44:55:66 Alias: S11223344556677C

1st note: it will be nice to show the user the RSSI obtained form info command as it will give the user the idea of how far the car is from the BLE device and help on troubleshooting the connection.

2nd note: the bluetoothctl was not meant to be used for what we try to do here hence the inconsistencies of the output. The proper way to implement is directly with BlueZ API (thre is a python lib). But yeah it works.

baylanger commented 3 months ago

@BogdanDIA thanx for all the valuable information.

Just pushed a commit to rename the var to BLE_LOCAL_NAME - but this PR generates that value automatically.

That said, until we figure out a way to scan for the "local name" OR be able to calculate the advertised addr (MAC) using the "local name" (?) , this PR shouldn't be merged yet. If the MAC is available using bluetoothctl , then we should use that.

@raphmur I guess we should put this PR on hold. I'm going to create another PR to allow the end user to set a MAC. We can probably keep what's in place in main branch but update the var name & translation files. I'll take a look later or tomorrow.

BogdanDIA commented 3 months ago

@BogdanDIA thanx for all the valuable information.

Just pushed a commit to rename the var to BLE_LOCAL_NAME - but this PR generates that value automatically.

That said, until we figure out a way to scan for the "local name" OR be able to calculate the advertised addr (MAC) using the "local name" (?) , this PR shouldn't be merged yet. If the MAC is available using bluetoothctl , then we should use that.

@raphmur I guess we should put this PR on hold. I'm going to create another PR to allow the end user to set a MAC. We can probably keep what's in place in main branch but update the var name & translation files. I'll take a look later or tomorrow.

Probably I was not clear but is should be very simple. The idea is to find MAC<->CAR_BLE_NAME correspondence (the procedure is above). And this should be done only once at the initial stage when MAC is not known. Alternatively, MAC can be entered manually based on other tools like apps on the phone. MAC is fixed per car and will not change. After that just scan all time with the known car's MAC and detect presence:

bluetoothctl --timeout 10 scan on | grep $MAC

Can add a threshold for number of finds to get a solid result. I have this working for more than a year now.

raphmur commented 3 months ago

@baylanger please advise where you are on it. I suggest to pack all this clever MAC calculation in one dedicated sh file that will do all the setup work:

... and to be honest, this addon is already for fairly "geeky" users. Personally I really don't bother using the "BLE scanner" app to find the MAC, and that saying, this is only if I want to use the proximity sensor.

Remember what is needed to setup FleetAPI proxy? :)

What do you think? As you suggest @baylanger PR on hold

baylanger commented 3 months ago

@baylanger please advise where you are on it. I suggest to pack all this clever MAC calculation in one dedicated sh file that will do all the setup work:

👍 but. ... but but ... read below 🥇

* Define & publish new MQTT entity "setup_proximity_detection"
* Then clicks on button "setup_proximity_detection" in MQTT entity
* This calls the setup_proximity_detection.sh which calculates the LNAME from the VIN, then runs bluetoothctl to scan for all devices, then "finds" the Tesla. That way you can park all the code, independently of manually setting the MAC if you know it.

My BLE dongle was too far from my car.... until ~1hr ago finally it showed up in bluetoothctl. Then I started to code ... and once I had code to match and extract the MAC address, I realized that if "scan on" previously was enable, bluetoothctl doesn't always display the car's "MAC + LNAME". Like everyone usually do, then I decided to read the manual .... to realize the MAC address is just... useless!

With the "scan on" command:

But the "devices" command lists all the previously detected ones. The output shows up both the MAC + LNAME:

[bluetooth]# devices
Device 66:B7:E3:02:3E:C0 66-B7-E3-02-3E-C0
Device Xx:Xx:Xx:Xx:Xx:Xx S.............C                    <<<<< The Tesla
Device C0:EE:40:71:4E:C2 HB-00107804
Device 6F:7F:F2:59:19:65 6F-7F-F2-59-19-65
Device 60:74:F4:03:0D:33 ihoment_H7130_0D33
Device 7D:FA:20:FF:F1:4B 7D-FA-20-FF-F1-4B
Device 55:09:BB:D4:EF:71 55-09-BB-D4-EF-71
Device 44:67:55:1A:F4:2C bhyve_0AF42C
Device 62:D1:AD:09:F7:22 62-D1-AD-09-F7-22
Device CC:6A:10:50:16:8A MyQ-326

In the end, if we run the "scan on" following by the "devices" command; if the car isn't too far from the dongle, either it will show up immediately in the "devices" list otherwise soon or later, "scan on" will pick it up. We can do a direct match using the LNAME 🎉🎉🎉 unless there's a case I'm not aware, I see no reason for us to extract the MAC.

... and to be honest, this addon is already for fairly "geeky" users. Personally I really don't bother using the "BLE scanner" app to find the MAC, and that saying, this is only if I want to use the proximity sensor.

I think we now have a way to make this presence a lot more easier for the end user ;) We can have a toggle switch to enable the detection or not.

Remember what is needed to setup FleetAPI proxy? :)

lol

What do you think? As you suggest @baylanger PR on hold

I should finish the new PR today ... but it's late here :D

BogdanDIA commented 3 months ago

@baylanger using 'devices command' is fine but most of the data in bluetoothctl is cached in bluez. Rpi is not really affected but big OSs like Ubuntu are. Internet is full of issues coming from caching in bluez.

To solve the problem just remove the car's entry from devices (you will need the MAC): remove $MAC This will force reading again the broadcast data from the car and if the car is close the entry will be recreated.

baylanger commented 3 months ago

@baylanger using 'devices command' is fine but most of the data in bluetoothctl is cached in bluez. Rpi is not really affected but big OSs like Ubuntu are. Internet is full of issues coming from caching in bluez.

To solve the problem just remove the car's entry from devices (you will need the MAC): remove $MAC

Per the tests I did, is removing the entry from the cache still needed today with the latest bluez?

Here what I found out after writing my last comment:

  1. After lauching bluetoothctl , if no command is issued once and a while something triggers a scan and the caching list of devices gets "temporarily" populated. Once the scan is over, I do notice the entries in the caching list gets removed exactly 30 seconds later.

  2. Issuing a scan on sets discovery on and the caching list gets temporarily populated.

    [bluetooth]# scan on
    [bluetooth]# SetDiscoveryFilter success
    [bluetooth]# hci1 type 7 discovering on
    [bluetooth]# Discovery started
    [bluetooth]# [CHG] Controller 04:42:1A:5B:2B:00 Discovering: yes
    [bluetooth]# [NEW] Device CC:6A:10:50:16:8A MyQ-326
    [bluetooth]# [NEW] Device 44:67:55:1A:F4:2C bhyve_0AF42C
    ...
  3. ~6 seconds later, discovering is turned off and the caching list stops to grow.

    ...
    [bluetooth]# [NEW] Device 5A:39:E8:D7:4D:05 5A-39-E8-D7-4D-05
    [bluetooth]# [CHG] Device 5A:39:E8:D7:4D:05 RSSI: 0xffffffa8 (-88)
    [bluetooth]# [CHG] Device 61:1E:0D:F2:5A:88 RSSI: 0xffffffb6 (-74)
    [bluetooth]# [NEW] Device 70:2D:02:0F:5E:96 70-2D-02-0F-5E-96
    [bluetooth]# [NEW] Device 59:01:DB:8B:0A:75 59-01-DB-8B-0A-75
    [bluetooth]# hci1 type 7 discovering off

    Note: 6 seconds seem quite short if a vehicle signals is a weak. We should probably loop & retry a few times within HA rules (10 times within 30 minutes). @raphmur was it in HA's system logs that you saw this warning.. was it a warning or scanning wasn't possible anymore? 🤔

  4. Within the next ~90 seconds, the caching list gets slowly emptied out. Will be populated again once something turns on discovery or that our script sends a scan on command.

Each time I tried to remove the vehicle's entry using the MAC, I always get a unavailable. I don't think it's needed to have code to remove the entry from the caching. From what's happening, IF the vehicle appears in the devices list, it's because something triggered a scan and added it. If so, the vehicle most be present, right?

I'm going to make the PR on the assumption that caching works fine today because that seems to be the case.

Let me know if you have concerns.

[edit 1] Once the scan is over, I do notice the entries in the caching list gets removed exactly 30 seconds later. [edit 2] 1. After lauching bluetoothctl , if no command is issued once and a while something -> tesla-local-control-addon <- Somehow working on the BLE PR made me think there was currently no code in place now launching a scan - Lol. If that's the case than we should expect the devices list to always be empty but again, if it's not empty and the vehicle's Local Name & MAC is there, it's cuz of it's present.

raphmur commented 3 months ago

@baylanger I think it was in the Supervisor logs. I did not capture it when I saw it unfortunately, but the use of bluetoothctl more than 10 times in 30 mins caused a sort of ban. You can maybe test it at same periodicity of MQTT loop and see if it still happens? I will try tomorrow morning.

baylanger commented 3 months ago

I have run ~200 scans since yesterday. What's great, not once the driver failed. During those scans, the container was running as well a scan every ~180 seconds. What didn't run in parallel is tesla-control and bluetoothctl , coud that cause issues? It sure is easy to launch them in parallel for a while and see if anything fails.

While doing all those scans, if HA complained the container was doing too many scans, well ... it might complain but it didn't prevent them to complete.

Perhaps HAOS 12.4 did it?

baylanger commented 3 months ago

I'm not sure how to start this comment :D

The comment I wrote above is True ONLY IF the bluetooth_le_tracker integration isn't configured. If that integration is configured my vehicle is found 100% of the time in the cache using the devices command.

Do we need to be aware if bluetooth_le_tracker is configured? Would the interaction with bluetoothctl differ?

While I have this one in mind... What if someone has 2 Tesla?

I'm far to be done w/ the PR - I had too many questions on my mind today 🤔 , will continue tomorrow.

iainbullock commented 3 months ago

I will catch up with this conversation when I get home from holidays, so I have the means of testing. Note I am working on a multiple car solution in my repo. It should be ready for testing in a few days

BogdanDIA commented 3 months ago

@baylanger using 'devices command' is fine but most of the data in bluetoothctl is cached in bluez. Rpi is not really affected but big OSs like Ubuntu are. Internet is full of issues coming from caching in bluez. To solve the problem just remove the car's entry from devices (you will need the MAC): remove $MAC

Per the tests I did, is removing the entry from the cache still needed today with the latest bluez?

Here what I found out after writing my last comment:

1. After lauching bluetoothctl , if no command is issued once and a while something triggers a scan and the caching list of devices gets "temporarily" populated. Once the scan is over, I do notice the entries in the caching list gets removed exactly 30 seconds later.

2. Issuing a `scan on` sets `discovery on` and the caching list gets temporarily populated.
[bluetooth]# scan on
[bluetooth]# SetDiscoveryFilter success
[bluetooth]# hci1 type 7 discovering on
[bluetooth]# Discovery started
[bluetooth]# [CHG] Controller 04:42:1A:5B:2B:00 Discovering: yes
[bluetooth]# [NEW] Device CC:6A:10:50:16:8A MyQ-326
[bluetooth]# [NEW] Device 44:67:55:1A:F4:2C bhyve_0AF42C
...
3. ~6 seconds later, discovering is turned off and the caching list stops to grow.
...
[bluetooth]# [NEW] Device 5A:39:E8:D7:4D:05 5A-39-E8-D7-4D-05
[bluetooth]# [CHG] Device 5A:39:E8:D7:4D:05 RSSI: 0xffffffa8 (-88)
[bluetooth]# [CHG] Device 61:1E:0D:F2:5A:88 RSSI: 0xffffffb6 (-74)
[bluetooth]# [NEW] Device 70:2D:02:0F:5E:96 70-2D-02-0F-5E-96
[bluetooth]# [NEW] Device 59:01:DB:8B:0A:75 59-01-DB-8B-0A-75
[bluetooth]# hci1 type 7 discovering off

Note: 6 seconds seem quite short if a vehicle signals is a weak. We should probably loop & retry a few times within HA rules (10 times within 30 minutes). @raphmur was it in HA's system logs that you saw this warning.. was it a warning or scanning wasn't possible anymore? 🤔

4. Within the next ~90 seconds, the caching list gets slowly emptied out. Will be populated again once something turns on discovery or that our script sends a `scan on` command.

Each time I tried to remove the vehicle's entry using the MAC, I always get a unavailable. I don't think it's needed to have code to remove the entry from the caching. From what's happening, IF the vehicle appears in the devices list, it's because something triggered a scan and added it. If so, the vehicle most be present, right?

I'm going to make the PR on the assumption that caching works fine today because that seems to be the case.

Let me know if you have concerns.

[edit 1] Once the scan is over, I do notice the entries in the caching list gets removed exactly 30 seconds later. [edit 2] 1. After lauching bluetoothctl , if no command is issued once and a while something -> tesla-local-control-addon <- Somehow working on the BLE PR made me think there was currently no code in place now launching a scan - Lol. If that's the case than we should expect the devices list to always be empty but again, if it's not empty and the vehicle's Local Name & MAC is there, it's cuz of it's present.

As I said on rpi caching is minimum and it will be fine but on an Ubuntu caching will stay there for long time. On my Ubuntu reports a device that I don't have at home for 3 week now. The safest way to make sure you discover immediately the existence of the car is to remove the device. It also makes the discovery process faster. I paste a script I am using and it works for rpi, Ubuntu and CentOS.

Each system has a different caching configuration and bluetoothctl configuration so it is better to make sure the code works in any case.

#!/bin/bash
# BogdanDIA

echo Start MAC finding
BLE_LOCAL_NAME=S1122334455667788C

# print BLE Local Name for which we want the MAC
echo BLE_LOCAL_NAME: $BLE_LOCAL_NAME

# start scan
bluetoothctl --timeout 10 scan on > /dev/zero &

INFOMAC=""
for (( i=0; i<2; i++ ))
do
        DEVICES=$(bluetoothctl --timeout 1 devices | grep "$BLE_LOCAL_NAME")
        echo "$DEVICES"

        if [[ -n "$DEVICES" ]]; then
                echo Matched car\'s BLE name
                INFOMAC=$(echo "$DEVICES" | grep -o -E '([[:xdigit:]]{1,2}:){5}[[:xdigit:]]{1,2}')
                echo MAC: "$INFOMAC"

                echo removing device "$INFOMAC"
                bluetoothctl --timeout 1 remove "$INFOMAC"
        fi
done

echo MAC: "$INFOMAC"

echo End MAC finding

You may want to play with the timeouts and number of loops.

baylanger commented 3 months ago

rpi caching is minimum and it will be fine but on an Ubuntu caching will stay there for long time

I'm puzzle to understand why the RPi vs Ubuntu differ so differently in terms of caching.

What OS release and what runs on each of those boxes? Do they run have the same bluez release - and if so, what differs in bluez config?

BogdanDIA commented 3 months ago

rpi caching is minimum and it will be fine but on an Ubuntu caching will stay there for long time

I'm puzzle to understand why the RPi vs Ubuntu differ so differently in terms of caching.

What OS release and what runs on each of those boxes? Do they run have the same bluez release - and if so, what differs in bluez config?

rpi5 has bluez 5.66 and Ubuntu 20.04 has 5.33 but various params can be configured at build time. I guess it is the distribution builders decision.

BTW, the caching is in /var/lib/bluetooth/ dir and can be deleted manually but I do not recommend :)

baylanger commented 3 months ago

I'm puzzle to understand why the RPi vs Ubuntu differ so differently in terms of caching. What OS release and what runs on each of those boxes? Do they run have the same bluez release - and if so, what differs in bluez config?

rpi5 has bluez 5.66 and Ubuntu 20.04 has 5.33 but various params can be configured at build time. I guess it is the distribution builders decision.

From Home Assistant Bluetooth:

It is highly recommended to use BlueZ >= 5.63 as older versions have been reported to be unreliable. Linux kernel 5.15.62 or later.

From Bluez github Issue 191 ; a few PR were merged in relation to the issue.

I would strongly suggest you update bluez on your Ubuntu box and see if it behaves better. It's hard to imagine the hardware would make a difference on the caching, unless the config on the RPi is good and bad on Ubuntu - but if HA tells us to use 5.63 and later, I would definitely follow that guideline.

baylanger commented 3 months ago

BTW 5.33 was released 9 years ago.

BogdanDIA commented 3 months ago

BTW 5.33 was released 9 years ago.

Agree, it is an old system. However, in a standalone docker implementation (not related to HA) we'll find any kind of systems, including very old ones.

baylanger commented 3 months ago

@iainbullock couldn't tag you in #32 - see whats in there and the comment on what I was on in past days.

Hopefully we can all gather tomorrow. I'm on East coast Canada and tomorrow (Monday) not working. Raohael and I connected via WhatsApp if you have that, try to connect with him.

iainbullock commented 3 months ago

I'm back from my holidays and trying to catch up with this (and various other things), before going back to work tomorrow (Tuesday). It would be good to have a call to see where we are with the projects.

I'm in the UK, time zone is currently UTC+1 @baylanger I assume you are UTC-5 so 6 hours behind me @raphmur were are you located?

I could do a call this afternoon UK time (though have to take my motorbike for service, so it depends when). Shall we use the Discord private channel to make the arrangements?

raphmur commented 3 months ago

I'm in France. Weekend can manage the details via WhatsApp.