Feature request - BLE scan and read values

[michapr]

Hi, I'm not sure but maybe it could be interesting to change / extend the BLE sensor reading feature. As we know every connect to sensor costs battery capacity. A lot of sensors send the data in advertising package, without encryption. Flower Care, VegTrug, Mijia LYWSDCGQ, Xiaomi Formaldehyde Monitor Honeywell JQJCY01YM, Xiaomi LYWSD02 Mijia CGG1, Xiaomi CGP1W, ClearGrass CGD1, Xiaomi Smart Weight Scale XMTZC04HM, Xiaomi Smart Weight ScaleXMTZC05HM, INKBIRD, ...

I made some experiments - and it could be possible (for you) to add it in rpieasy if modify the BLEscan modul. Taking a template like for MiFlora sensors (for selecting the needed parameters for the sensor) and the BLEscan code (adding the service data to your lib) the user could select the MAC and the needed parameters - and script could read it.

A code sample for extracting all data we can find at OpenMQTTGateway. https://github.com/1technophile/OpenMQTTGateway/blob/development/main/ZgatewayBT.ino

For new sensors with transmitting advertised data it could be easy to add the code.

I'm using the OpenMQTT Gateway for some sensors out of range and it is working fine.

What do you think?

Thanks! Michael

[enesbcs]

I made some experiments - and it could be possible (for you) to add it in rpieasy if modify the BLEscan modul. Taking a template like for MiFlora sensors (for selecting the needed parameters for the sensor) and the BLEscan code (adding the service data to your lib) the user could select the MAC and the needed parameters - and script could read it.

It is possible, sure.

A code sample for extracting all data we can find at OpenMQTTGateway. https://github.com/1technophile/OpenMQTTGateway/blob/development/main/ZgatewayBT.ino

I saw it, but i have no clue what data size these templates mean. Maybe all sensor row means a byte size variable?

I'm using the OpenMQTT Gateway for some sensors out of range and it is working fine.

What do you think?

It could be done sometime. But many experiments + time + coding needed.

[michapr]

I saw it, but i have no clue what data size these templates mean. Maybe all sensor row means a byte size variable?

What do you mean with "what data size" ?

You get in dev.getScanData() as servicedata as sample [(1, 'Flags', '06'), (2, 'Incomplete 16b Services', '0000fe95-0000-1000-8000-00805f9b34fb'), (22, '16b Service Data', '95fe712098002b85086a8d7cc40d08100135'), (9, 'Complete Local Name', 'Flower care')]

From here: 22 16b Service Data = 95fe712098002b85086a8d7cc40d08100135 can be taken the model and in result temperature, humidity etc.

[enesbcs]

Byte, Integer, etc. is a data size. Offset in an array is a location.

For example this is a usable documentation for data size and location: (but only valid for Mijia) https://www.domoticz.com/forum/viewtopic.php?t=20925&start=20#p177159

What you linked is a code, which i can not use in python, and there are no usable comments inside it.

[enesbcs]

22 16b Service Data = 95fe712098002b85086a8d7cc40d08100135 can be taken the model and in result temperature, humidity etc.

Its easy to say... the question is "how". :)

[enesbcs]

I've succed with LYWSDCGQ and Mi Flora data harvesting, they are surprisingly "chatty". I am surprised that they are transmitting very frequently. Its a miracle that my wifi can operate in the middle of this massive BLE traffic. :) I will try with LYWSD02 also. I do not have the other type of devices.

[michapr]

Nice, already began to collect my information for you.

Here is LYWSD02: LYWSD02

95fe70205b0411387c717d5b3f090610021c02
95fe70205b0474387c717d5b3f09041002ed00

70205b0411387c717d5b3f090610021c02   ->6 (25->hum), data_lenght= 2*2=4  ->1c02 = 540 - 54%
70205b0474387c717d5b3f09041002ed00   ->4 (25.>temp), data_lenght= 2*2=4  ->ed00 = 237 - 23.7°C

[michapr]

here for CGG1

CGG1
95fe5030470394505410342d580a100106
95fe50304703a3505410342d580d1004eb00f101

5030470394505410342d580a100106        -a [23] data_lenght= 1*2=2 -> 06% batt 
50304703a3505410342d580d1004eb00f101   -d [23] data_lenght= 4*2=8 , [28..31] -> eb00 -> 235 -> 23.5°C / [32..35] ->f101 -> 497 -> 49.7%
5030470389505410342d58041002eb00    -4 [23] data_lenght= 2*2=4 , [28..31] -> eb00 -> 235 -> 23.5°C 
5030470352505410342d58061002f101    -6 [23] data_lenght= 2*2=4 , [28..31] -> f101 -> 497 -> 49.7%

[enesbcs]

BLE xiaomi sniffer plugin added at commit https://github.com/enesbcs/rpieasy/commit/4809cfecc9f930a0f512121a5b4cec812f4b8fdb Tested with LYWSD02, CGQ and MiFlora.

[michapr]

Thanks! Have tested today - it is working fine for MiFlora, LYWSD02 and CGG1 (ClearGrass Temp & RH).

Would be nice if in "Devices" page could be shown the MAC address of the devices (is empty in table)

[enesbcs]

I do not have CGG1 for testing. "95fe5030470394505410342d580a100106" gives me {'batt' : 6} as result, so the decoder works, in theory.

MAC address can be added to the table for sure.

[michapr]

sorry about wrong battery info - in this moment I posted I have got the value :-D That's why have removed the line. All is working as expected!

Now we can save battery ;)

[michapr]

for CGG1: I'm not sure, but it seems that batt value = 0 will be shown if no batt value will be reported by the device. The device is reporting sometimes only temp, or only hum, or both - and not every time the batt value. Would be nice not to overwrite the last received value.

EDIT: It seems that other Bluetooth plugins will not work more? With old scanner plugin they were still working. Or is it any local problem?

[enesbcs]

for CGG1: I'm not sure, but it seems that batt value = 0 will be shown if no batt value will be reported by the device. The device is reporting sometimes only temp, or only hum, or both - and not every time the batt value. Would be nice not to overwrite the last received value.

If no battery value reported, than battery value wont be overwritten, as no data to overwrite... Battery will be 0 if no battery data receiving for (3 x interval) times. So if you set interval to 90 seconds and no data coming for 270 seconds, battery will be 0 and rssi will be -1 to show that device may be lost. Otherwise battery is 255 as default it means not supported.

EDIT: It seems that other Bluetooth plugins will not work more? With old scanner plugin they were still working.

Other BLE plugins will not work at the same time with passive BLE scanning. It is written clearly at the plugin page. Sniffing is continous, without pause, as there are no way to see when data will come. Old scanner plugin is doing "active" scanning, only at the specific interval.

[michapr]

So if you set interval to 90 seconds and no data coming for 270 seconds, battery will be 0 and rssi will be -1 to show that device may be lost. Otherwise battery is 255 as default it means not supported.

Ok, understand. But maybe better would be not to set the battery value to 0 but only the rssi? Device may report the battery value only within a large interval, but can report temp/hum/rssi.

Sniffing is continous, without pause, as there are no way to see when data will come.

Maybe to make the "continues" optional? I do not think that it is needed for sensors. During my tests I have used one scan per minute for 10 seconds - and got all needed information. Yes, maybe "lost" some data, but it is not critical for sensors, they are advertise very frequently the data. OpenMQTTGateway make the same - by default 10 seconds scan - and every minute. In such a case could the other plugins work again and get the data from other devices with needed connections. (yes, scan should pause if connection process is active and vice versa ...)

What do you think?

[enesbcs]

Ok, understand. But maybe better would be not to set the battery value to 0 but only the rssi?

It is more than possible that if a previously advertising device stops advertising than its battery is deplated.. unless it is moving. I've raised the report-as-dead interval to 1800 seconds at minimum. Although it is possible that some misidentified packet causes 0 values. If you are setting Logging to DEBUG, you will see the actual cached values and the incoming battery values on the console.

Device may report the battery value only within a large interval, but can report temp/hum/rssi.

I've checking all incoming data and the last one time is checked, not just the battery timestamp.

Maybe to make the "continues" optional? I do not think that it is needed for sensors. During my tests I have used one scan per minute for 10 seconds - and got all needed information. Yes, maybe "lost" some data, but it is not critical for sensors, they are advertise very frequently the data. OpenMQTTGateway make the same - by default 10 seconds scan - and every minute.

It sounds a really bad idea. Especially if some sensor data - like batteries - comes rarely.

What do you think?

Plugins have 1-5 second after every 10 seconds of passive scanning to grab BLE if they want to connect at commit https://github.com/enesbcs/rpieasy/commit/0c3137b1baf1869e1b74f3b9daf54eca0ff03636 I've tested with passive scanning and a LYWSD03 plugin at the same time and it is working. I think that many active BLE plugin with small intervals will make passive scanning almost unusable. Bluetooth is a bottleneck, use zigbee if you intend to use a swarm of devices.

[michapr]

If you are setting Logging to DEBUG, you will see the actual cached values and the incoming battery values on the console.

ok, I will check the battery value in log over longer time and tell you.

... by default 10 seconds scan - and every minute. It sounds a really bad idea....

Not sure bad idea or not - I'm using OMQTT Gateway since 3 weeks - and have no "lost data".

Plugins have 1-5 second after every 10 seconds of passive scanning to grab BLE if they want to connect

Maybe to make it configurable? My other sensors have still some problems to connect - will debug it later. How can I change this interval in code for testing (without looking for the right place in your code ;) )

Bluetooth is a bottleneck, use zigbee if you intend to use a swarm of devices.

Right, I'm using Z-Wave for switches, alarm sensors, thermostats and other "critical" devices. For informational devices like room temperature, green house and weather sensors use in this time 433MHz sensors (TFA) - they are cheap and good. Bluetooth sensors are cheap with good display and I think for room sensors they are fine (even the design). Using the advertising data can solve the bottleneck if connecting additional only in rare cases - as sample for not transmitted battery value once in 2 hours or some other devices once in 15minutes.

[enesbcs]

Plugins have 1-5 second after every 10 seconds of passive scanning to grab BLE if they want to connect

Maybe to make it configurable?

Maybe it will be an option later in the plugin.

My other sensors have still some problems to connect - will debug it later. How can I change this interval in code for testing (without looking for the right place in your code ;) )

Which and how many sensors? There are semaphors which most BLE plugins using to advertise if they are using the BLE line, that is checked at line 94. (most of the time it solves race conditions)

In lib/lib_blescan.py at line 102 if you change: time.sleep(uniform(1,5)) to time.sleep(30) than the process will wait 30 seconds without scanning.

Bluetooth sensors are cheap with good display and I think for room sensors they are fine (even the design). Using the advertising data can solve the bottleneck if connecting additional only in rare cases - as sample for not transmitted battery value once in 2 hours or some other devices once in 15minutes.

I agree with this, but how many BLE device do you control with one RPI? With average 5m range (best case 10m) i can not imagine scenarios where more than 4 BLE room temperature sensors seen by one RPI. Except if there are many flowers with Mi Flora next to each other.

[michapr]

Have found something interesting:

Sep  1 20:19:20 RPIEasy RPIEasy2: 58:2d:34:10:54:50 {'hum': 53.3, 'hum-t': 1598982189.593753, 'batt': 8, 'batt-t': 1598982245.5919523, 'temp': 23.0, 'temp-t': 1598982249.5884478}
Sep  1 20:20:21 RPIEasy RPIEasy2: 58:2d:34:10:54:50 {'hum': 53.3, 'hum-t': 1598982189.593753, 'batt': 8, 'batt-t': 1598982245.5919523, 'temp': 23.0, 'temp-t': 1598982249.5884478}
Sep  1 20:21:21 RPIEasy RPIEasy2: 58:2d:34:10:54:50 {'hum': 53.3, 'hum-t': 1598982189.593753, 'batt': 8, 'batt-t': 1598982245.5919523, 'temp': 23.0, 'temp-t': 1598982249.5884478}

Sep  1 20:19:20 RPIEasy RPIEasy2: Event: ClearGrassTemp&RH#Temp=23.0
Sep  1 20:19:20 RPIEasy RPIEasy2: Event: ClearGrassTemp&RH#Hum=53.3
Sep  1 20:19:20 RPIEasy RPIEasy2: Event: ClearGrassTemp&RH#Batt=0.0
Sep  1 20:20:21 RPIEasy RPIEasy2: Event: ClearGrassTemp&RH#Temp=23.0
Sep  1 20:20:21 RPIEasy RPIEasy2: Event: ClearGrassTemp&RH#Hum=53.3
Sep  1 20:20:21 RPIEasy RPIEasy2: Event: ClearGrassTemp&RH#Batt=0.0
Sep  1 20:21:21 RPIEasy RPIEasy2: Event: ClearGrassTemp&RH#Temp=23.0
Sep  1 20:21:21 RPIEasy RPIEasy2: Event: ClearGrassTemp&RH#Hum=53.3
Sep  1 20:21:21 RPIEasy RPIEasy2: Event: ClearGrassTemp&RH#Batt=0.0

I already had found the time.sleep and changed to (4,15) - so it seems to work in this time more stable ;) Will tell you later then.

Using a Pi ZeroW I have very good connection in my house to this sensors (even to other floor... but do not use it for connection - but for scanning ;) ). Using 3 flower sensors and 4 other sensors (different for testing - LYWSD02, LYWSD03, CGG1, MHO-C401) and Thermostat BLE EQ3 + other sensors in "scan mode" in other floor - if receiving advertising data ok, if not, also ok... Want to replace the old 433MHz sensors in rooms. For main floor will be used same Pi as Domoticz. (here is the "hanging" BLE adapter after some time a problem, do not want to reboot this Pi of course)

[michapr]

Hi, the _P516_BLE_EQ3 plugin seems not to be compatible at all with the new plugin.

If enable the plugin after reading thermostat state the scanning process will cancel for ever. I have added a log in decode_xiaomi() - here will not be any call more inside. But old values will be still reported if adding thermostat after some minutes running time. If restarting application null values will be received from scanner ( .. { } ...). Disable plugin does not help, restart application after disable help.

[enesbcs]

the _P516_BLE_EQ3 plugin seems not to be compatible at all with the new plugin.

This plugin also has the warning at the plugin page. I am thinking now that nobody reads these lines... webserver.addFormNote("This plugin WILL NOT work with ble scanner plugin.")

And before you ask, the eq3 plugin working with continous connection with the EQ3 device and fetches data (subscribing and waiting for notifications for device), it is working that way, at least i understand base on its code. https://github.com/rytilahti/python-eq3bt/blob/master/eq3bt/eq3btsmart.py

[michapr]

I am thinking now that nobody reads these lines...

Oh... yes,... :-D ... I really have not seen it (made a fast reinclude of the device...) That's a shame, but there is probably nothing to be done at the moment...

Checking the plugin now step by step. Have another problem with LYWSD02 Clock&Hygrometer: Device is calling once blestatus.requeststopscan - and then nothing more happen. (needed for battery value - would like to call once in a hour as sample, or less)

LYWSD03MMC seems to work fine

[enesbcs]

Have another problem with LYWSD02 Clock&Hygrometer: Device is calling once blestatus.requeststopscan - and then nothing more happen.

Yes the LYWSD02 init section need to be modified. Until it fixed, you can try to move Passive scanning task at the end of task list, definitely behind the LYWSD02 to let it to initialize.

[enesbcs]

LYWSD02 init section repositioned. Passive scan init delayed by 30 seconds now to let init tasks after it. BLE plugins modified to one device use BLE at a time. Commit https://github.com/enesbcs/rpieasy/commit/2d7d900686c6bac56b14681d2e831aa8742c0812 Please backup config before updating

[michapr]

Looks perfect in this time ! Is running stable with 4 LYWSD03/MHO-C401, LYWSD02, CGG1 , and 4 sensors in sniffer without any reboot. Values will be still reported, all seems to be ok now..

Will have a look at it next 3 days and then close if all is still working fine ;)

THANKS!

[michapr]

Sorry to say that - but I accidentally provoked the error again :(

After all was working fine I added in every task line the send to controller option (MQTT) step by step. After finishing I have seen that in log the "not free" line came up for a long time (10 minutes) - so I had to restart the application.

Reason seems to be the same device as before CCG1 (54:50) -> task [5] and not the LYWSD02 (7C:38) [4] ?:

Sep  5 16:43:53 RPIEasy RPIEasy98: BLE line not free for P518! [4]
Sep  5 16:43:53 RPIEasy RPIEasy98: BLE line not free for P518! []
Sep  5 16:43:53 RPIEasy RPIEasy98: BLE connection initiated to 58:2d:34:10:54:50
Sep  5 16:43:53 RPIEasy RPIEasy98: BLE line not free for P513! [4]
Sep  5 16:43:53 RPIEasy RPIEasy98: BLE line not free for P513! [4]
Sep  5 16:43:53 RPIEasy RPIEasy98: BLE line not free for P518! [4]
Sep  5 16:43:54 RPIEasy RPIEasy98: BLE line not free for P513! [4]
Sep  5 16:43:54 RPIEasy RPIEasy98: BLE line not free for P513! [4]
Sep  5 16:43:54 RPIEasy RPIEasy98: BLE line not free for P518! [4]
Sep  5 16:43:54 RPIEasy RPIEasy98: BLE connection failed 58:2d:34:10:54:50
Sep  5 16:43:54 RPIEasy RPIEasy98: BLE line not free for P513! []
Sep  5 16:43:54 RPIEasy RPIEasy98: BLE connected 3f:5b:7d:71:7c:38
Sep  5 16:43:54 RPIEasy RPIEasy98: BLE line not free for P513! [5]
Sep  5 16:43:54 RPIEasy RPIEasy98: BLE line not free for P518! [5]
Sep  5 16:43:55 RPIEasy RPIEasy98: BLE line not free for P513! [5]
Sep  5 16:43:55 RPIEasy RPIEasy98: BLE line not free for P518! [5]
Sep  5 16:43:55 RPIEasy RPIEasy98: Sync LWSD02 time
Sep  5 16:43:55 RPIEasy RPIEasy98: BLE line not free for P513! [5]
...
Sep  5 16:44:34 RPIEasy RPIEasy98: BLE line not free for P518! [5]
Sep  5 16:44:35 RPIEasy RPIEasy98: MiFlora v3.2.1 connected, address: c4:7c:8d:6a:08:85
Sep  5 16:44:35 RPIEasy RPIEasy98: BLE line not free for P513! [5]
...
Sep  5 16:44:39 RPIEasy RPIEasy98: BLE line not free for P518! [5]
Sep  5 16:44:39 RPIEasy RPIEasy98: MiFlora v3.2.1 connected, address: c4:7c:8d:6a:08:85
Sep  5 16:44:39 RPIEasy RPIEasy98: BLE line not free for P513! [5]
...
Sep  5 16:44:46 RPIEasy RPIEasy98: BLE line not free for P518! [5]
Sep  5 16:44:46 RPIEasy RPIEasy98: MiFlora v3.2.2 connected, address: c4:7c:8d:6b:3f:3f
Sep  5 16:44:46 RPIEasy RPIEasy98: BLE line not free for P513! [5]
...
Sep  5 16:44:51 RPIEasy RPIEasy98: BLE line not free for P518! [5]
Sep  5 16:44:51 RPIEasy RPIEasy98: MiFlora v3.2.2 connected, address: c4:7c:8d:6b:3f:3f
Sep  5 16:44:51 RPIEasy RPIEasy98: BLE line not free for P513! [5]
...
Sep  5 16:53:05 RPIEasy RPIEasy98: BLE line not free for P518! [5]
Sep  5 16:53:05 RPIEasy RPIEasy98: CMD: exit
Sep  5 16:53:05 RPIEasy RPIEasy98: Event: System#Shutdown

[enesbcs]

When you are pressing Submit at the plugin page it will try to reconnect immediately, without checking if BLE is free. An RPIEasy restart solve this. Modifing plugin settings is not an everyday task. [5] is Task 6, as Tasks are zero-based internally but 1 based at the Device page.

[michapr]

ok, understand. So it is less critical if we know about it. I have seen that it will try to connect immediately.

[michapr]

All is still working as expected. Happy to use now most sensors in scanning mode, only for battery I connect once in hour the Miflora sensors. The other sensors (not sending data in advertising) will be requested every 10 -15 minutes, it is working stable now.

Thanks!