BLE connection "latency"

[ACrazyConcept]

Hi it's me again.

So my setup right now is a RPI Zero W in the bathroom to control the PAX Calima. Then I have created three python scripts. One to just get the fan status values, one to turn on boostmode and one to turn off boostmode. So I have totally disabled the automatic programming of the fan. (I actually don't think it works very well either) I have then installed Home Assistant on the PI Zero W and set up command_line sensor and switch to execute the python scripts and fetch the sensor values etc. (I then have a NUC running Debian and my main Home Assistant setup but thats not important here)

This is actually working pretty well. I can read the sensor value and control the fan by any intricate rules I set up using node-red flows.

I only have one thing I really wish could be improved and that is the latency when connecting to the fan. It takes around 10-15 seconds for the script to connect, execute a function and print the return value. Then I have to wait a few more seconds to be sure the connection has been terminated before I can attempt a new connection. I have tried running a script continuously to keep the connection active. But it was neither reliable nor practical. At least with my limited python skills.

So my question is. Is there any way to improve or otherwise get around this ?

[PatrickE94]

I have tried running the script continuously. This results in the fan not responding to sensors at all (noticed when the fan didn't evacuate the humid air when showering). So continuous connection is not preferable.

As for the BLE latency. It requires a bit more digging. It could be slow interactions using the bluepy library. If this is the case, the solution is to rewrite it in another language with more low-level access.

It's also possible that it's related to the bluetooth stack in linux. Being configured with a high scan interval or long timeouts.

First order of business would be to read the characteristics using an existing tool to determine a reference latency. I'm not afraid of rewriting the tool in another language (professionally working as a C/C++ programmer). But determining the issue should be done first :)

[ACrazyConcept]

I like a good detective case ^^ I just wish I knew enough to go digging myself in this case.

[PatrickE94]

It's hard to measure using the command line as most tools are interactive consoles and not easily "scriptable". But I do experience a much faster response from bluetoothctl when reading and writing characteristics. I'll look into creating a prototype in Rust using rumble. If the problem is python/bluepy related, this should make it disappear quickly.

Don't worry about rust, if you want I can provide a binary blob for RPi to replace cmdline.py.

[PatrickE94]

I've found the major speed bump in the current script. When requesting a Bluetooth characteristic, bluepy invokes a full scan if no previous scan of characteristics has been done. This is the heavier delay.

Tomorrow I might try invoking a limited discovery (only state and pincode) and see if this results in a speedier "state only script".

[PatrickE94]

I haven't had time to test this one out yet. If you're curious you might try it before I get the time!

Just insert the following into a .py file.

import bluepy.btle as ble
from struct import unpack, pack

addr = "58:2b:db:de:ad:be"
pin = "01234567"

fan = ble.Peripheral(deviceAddr=addr)

# Set Auth (You might want to try without)
fan.getCharacteristics(uuid="4cad343a-209a-40b7-b911-4d9b3df569b2")[0].write(pack("<I", int(pin)))

state_c = fan.getCharacteristics(uuid="528b80e8-c47a-4c0a-bdf1-916a7748f412")[0]
try:
    v = unpack('<4HBHB', state_c.read())
    print(v)
except Exception as e:
    print(e)

fan.disconnect()

It essentially skips all the bells and whistles and tries to assign auth without checking for a response and then fetching the state. It doesn't not print the state nicely, but the time until you get the line should indicate (at least somewhat) the latency. You might want to try without auth as well. I don't know if the fan will return any correct values if we skip auth, but hey, try it!

As I said previously, I haven't hade time to dig into BluePy to debug thoroughly. If this works well, I might not have to...

[ACrazyConcept]

Thanks Patrick! I appreciate the effort.

I did a few runs of my old code, with one and two get functions per connection, and your new code and there is not much difference. But I think I am OK with the way my setup works now actually. Even though I get:

Update of sensor.fan_getstate is taking over 10 seconds

on almost all state updates. But only actual connection errors around a dusin times a day.

pi@hassbian:~ $ sudo python3 /home/homeassistant/.homeassistant/pycalima/calima_get_state.py
FanState(Humidity=72, Temp=24.0, Light=1, RPM=0, Mode='Trickle ventilation')
--- 12.643166542053223 seconds ---
pi@hassbian:~ $ sudo python3 /home/homeassistant/.homeassistant/pycalima/no_bells_and_whistles.py
(73, 96, 1, 0, 1, 0, 2)
--- 8.390788316726685 seconds ---
pi@hassbian:~ $ sudo python3 /home/homeassistant/.homeassistant/pycalima/no_bells_and_whistles.py
(73, 96, 2, 0, 1, 0, 2)
--- 5.072439432144165 seconds ---
pi@hassbian:~ $ sudo python3 /home/homeassistant/.homeassistant/pycalima/no_bells_and_whistles.py
(73, 96, 1, 0, 1, 0, 2)
--- 5.480857849121094 seconds ---
pi@hassbian:~ $ sudo python3 /home/homeassistant/.homeassistant/pycalima/no_bells_and_whistles.py
(73, 96, 2, 0, 1, 0, 2)
--- 5.103446960449219 seconds ---
pi@hassbian:~ $ sudo python3 /home/homeassistant/.homeassistant/pycalima/calima_get_state.py
FanState(Humidity=73, Temp=24.0, Light=2, RPM=0, Mode='Trickle ventilation')
--- 5.533169269561768 seconds ---
pi@hassbian:~ $ sudo python3 /home/homeassistant/.homeassistant/pycalima/calima_get_state.py
FanState(Humidity=73, Temp=24.0, Light=2, RPM=0, Mode='Trickle ventilation')
--- 5.386448383331299 seconds ---
pi@hassbian:~ $ sudo python3 /home/homeassistant/.homeassistant/pycalima/calima_get_state.py
FanState(Humidity=73, Temp=24.0, Light=2, RPM=0, Mode='Trickle ventilation')
--- 5.301048517227173 seconds ---
pi@hassbian:~ $ sudo python3 /home/homeassistant/.homeassistant/pycalima/calima_get_state.py
FanState(Humidity=73, Temp=24.0, Light=1, RPM=0, Mode='Trickle ventilation')
BoostMode(OnOff=0, Speed=2400, Seconds=0)
--- 8.087800025939941 seconds ---
pi@hassbian:~ $ sudo python3 /home/homeassistant/.homeassistant/pycalima/calima_get_state.py
FanState(Humidity=73, Temp=24.0, Light=1, RPM=0, Mode='Trickle ventilation')
BoostMode(OnOff=0, Speed=2400, Seconds=0)
--- 7.667898178100586 seconds ---
pi@hassbian:~ $ sudo python3 /home/homeassistant/.homeassistant/pycalima/calima_get_state.py
FanState(Humidity=73, Temp=24.0, Light=1, RPM=0, Mode='Trickle ventilation')
BoostMode(OnOff=0, Speed=2400, Seconds=0)
--- 7.581124305725098 seconds ---
pi@hassbian:~ $ sudo python3 /home/homeassistant/.homeassistant/pycalima/calima_get_state.py
FanState(Humidity=72, Temp=24.0, Light=1, RPM=0, Mode='Trickle ventilation')
BoostMode(OnOff=0, Speed=2400, Seconds=0)
--- 7.553771734237671 seconds ---
pi@hassbian:~ $ sudo python3 /home/homeassistant/.homeassistant/pycalima/no_bells_and_whistles.py
(73, 96, 1, 0, 1, 0, 2)
--- 5.217174053192139 seconds ---
pi@hassbian:~ $ sudo python3 /home/homeassistant/.homeassistant/pycalima/no_bells_and_whistles.py
(73, 96, 1, 0, 1, 0, 2)
--- 5.11168360710144 seconds ---
pi@hassbian:~ $ sudo python3 /home/homeassistant/.homeassistant/pycalima/calima_get_state.py
FanState(Humidity=73, Temp=24.0, Light=2, RPM=0, Mode='Trickle ventilation')
--- 5.353248119354248 seconds ---
pi@hassbian:~ $ sudo python3 /home/homeassistant/.homeassistant/pycalima/calima_get_state.py
FanState(Humidity=72, Temp=24.0, Light=1, RPM=0, Mode='Trickle ventilation')
BoostMode(OnOff=0, Speed=2400, Seconds=0)
--- 7.431379795074463 seconds ---
pi@hassbian:~ $ sudo python3 /home/homeassistant/.homeassistant/pycalima/calima_get_state.py
FanState(Humidity=72, Temp=24.0, Light=2, RPM=0, Mode='Trickle ventilation')
BoostMode(OnOff=0, Speed=2400, Seconds=0)
--- 7.634907960891724 seconds ---
pi@hassbian:~ $ sudo python3 /home/homeassistant/.homeassistant/pycalima/calima_get_state.py
FanState(Humidity=73, Temp=24.0, Light=1, RPM=0, Mode='Trickle ventilation')
BoostMode(OnOff=0, Speed=2400, Seconds=0)
--- 7.408637285232544 seconds ---

[ACrazyConcept]

Keeping to this thread since it is not really a new issue.

I just started using ESPHome for the Mi Flora sensors. It uses the beacons the sensors send out on their own, instead of actively connecting to them. So I just began wondering if the Calima also broadcasts it state on its own and only need the pin code for changing the state? So it might be a way to go rather than a Pi.

Using a BLE scanner on iPhone I can see it broadcasts a lot of stuff. But that is as far as my skills go..

[PatrickE94]

Generally I don't think BLE broadcasts data before interaction. In general, the only message broadcasted without an explicit request is the advertisement info which is limited in size and typically holds device type, name etc. This is essentially what you see on your connect screen in the phone before connecting.

I don't think Mi Flora advertises values without GATT data request. However, I don't think we need to Auth to the device in order to read data. I don't recall to be honest, but I don't remember it being required. In the previous post I made, I provided a snippet. You could try to remove the auth line and run it without. See if it works anyway.

However, if ESP Home is performant on the Mi Flora devices. It might be a good approach to try, it might be a faster way to interact with bluetooth than via the pi's bluetooth stack. I even think I have an ESP32 lying around, just not the time to mess with it atm.

[ACrazyConcept]

Seems to get the same without auth for the no bells and whistles script. I just don't remember what the numbers besides humidity, which seems obvious, refer to. Edit: ahh:

# Hum   Temp  Light FanSpeed Mode Tbd   Tbd
return FanState(v[0], v[1]/4, v[2], v[3], trigger)

pi@hassbian:~ $ sudo python3 /home/homeassistant/.homeassistant/pycalima/calima_get_state.py
FanState(Humidity=29, Temp=23.5, Light=1, RPM=0, Mode='Trickle ventilation')
BoostMode(OnOff=0, Speed=1000, Seconds=0)
--- 10.628783941268921 seconds ---
pi@hassbian:~ $ sudo python3 /home/homeassistant/.homeassistant/pycalima/no_bells_and_whistles.py
(28, 94, 1, 0, 1, 0, 2)
--- 3.0074169635772705 seconds ---

[PatrickE94]

That's a heck of a lot faster though. I'll modify the API to allow non-authed usage. Seems appropriate!

I also notice you're running of hassbian. Have you considered trying to implement a hassio integration? Or possibly test mine... My Hass solution is a large computer in the basement an all my Pi's are busy doing other things.

[ACrazyConcept]

Ohh yeah, it's not that consistent it turns out. Goes up to between 8-10 seconds.

I am running this hassbian for the Calima. My main HA is in docker on my Intel NUC.

I don't know about making an integration myself, but I would be happy to test one.

[PatrickE94]

I'll see if I can scratch one together. However my master thesis just begun, so time is becoming more of a rarity.

I still consider having a "connect but don't auth" as a reasonable feature, albeit not actually improving the timing. I currently have a local state where I started to clean up the API and document it properly. Mostly as a hassio addon requires a published package (pip) and I feel I should do it right when publishing it to a publicly installable repo. I'll get back to you when I've drafted a prototype!

[ACrazyConcept]

A little while later.. I have been monitoring the script latency all this time and I just now noticed something interesting. When my fan is running in boost mode it seems to responds much faster. I will keep an eye on it from now to see if it is consistent.

I wonder if it can then somehow be tricked into the fast lateny mode without actually running the fan.

Stil using the script you suggested but I have upgraded to a RPi 3B+ (was a non plus before).

import bluepy.btle as ble
from struct import unpack, pack
import time

start_time = time.time()

addr = "58:2B:DB:01:D5:33"
fan = ble.Peripheral(deviceAddr=addr)

state_c = fan.getCharacteristics(uuid="528b80e8-c47a-4c0a-bdf1-916a7748f412")[0]

try:
    v = unpack('<4HBHB', state_c.read())
    print(v)
except Exception as e:
    print(e)

fan.disconnect()
print(" %s seconds" % round((time.time() - start_time),3))