Closed Ernst79 closed 2 years ago
In case it's interesting, I've put my Arduino IDE ESP32 code for the pulse counter in github too now: https://github.com/scrambledleek/ESP32_BLEPulseCounter
Hi! I have the same wish/need to get my home brew sensors report back to base in a good way. My sensors send all their data in the advertise message. They have multiple sensor types, and send them all through the advertise message. My coding skills are a bit limited so, for me to make this happen, i need all support i can get. I am able to get BLE data to an RPI and decode it, but my end need is a BLE sensor -> BLE to Wifi gateway -> cloud storage.
There are some important developments, we have plans to move ble_monitor to HA core, in which we will make adding sensors more flexible (e.g. via a json file). So, please allow us some time to make this switch, and we will try to take this request into account as well.
There are some important developments, we have plans to move ble_monitor to HA core, in which we will make adding sensors more flexible (e.g. via a json file). So, please allow us some time to make this switch, and we will try to take this request into account as well.
Congratulations @Ernst79 - a significant recognition of all the hard work you've been putting into this - well done!
What hardware did you use for your sensors @nikfo953? I used an ESP32 board, so I could have used the ESPHome firmware and Home Assistant plugin to send the data back from sensor to Raspberry Pi using WiFi (my ESP32 is permanently plugged in rather than battery powered). I just chose to use Bluetooth because I thought it might save a bit of energy and be an interesting challenge. Not sure my ESP32 code works very well though - the Bluetooth advertising seems to be continuous rather than turn off some of the time if there's nothing new to tell...and then work (and it being installed under the cabinets in my kitchen) got in the way of playing with it any more!
@scrambledleek, my sensor is light (lux), temp and humid sensors, that is semi self-developed. It uses the Nordic nrf-chip for BLE comms. I haven't designed the sensor myself, but I have access to the protocol.
Okay @nikfo953, if you're keen to try out using the ble_monitor to receive your sensor data before it gets integrated into core Home Assistant, we may be able to extend/adjust what I did for my pulse counters a little.
Adding support for lux, temp and %RH readings in custom_components/ble_monitor/ble_parser/homebrew.py
shouldn't be too difficult. You'd probably want 2 bytes for sending temp and %RH (although actually we could save space by using a single byte for the %RH as most sensors are only accurate to +/-3%RH or so anyway and we can fit 0 to 255 into a single byte...I'll go with 2 bytes and a decimal place below though), and lux may go over 65535 in bright sunshine, so we need 3 bytes for that, so we could add objects like this:
T_STRUCT = struct.Struct("<h") # Signed, 2 bytes
H_STRUCT = struct.Struct("<H") # Unsigned, 2 bytes
LUX_STRUCT = struct.Struct("<I") # Unsigned, 4 bytes (although only 3 are sent and we pad to allow unpacking)
...
def objHomebrew_temp(xobj):
# Temperature received will be an integer 10x the size of the temperature, so divide by 10
(value,) = T_STRUCT.unpack(xobj) / 10
return {"temp": value}
def objHomebrew_rh(xobj):
# Temperature received will be an integer 10x the size of the temperature, so divide by 10
(value,) = H_STRUCT.unpack(xobj) / 10
return {"humidity": temp}
def objHomebrew_lux(xobj):
# Lux is sent without multiplication as no decimal place is needed
(value,) = LUX_STRUCT.unpack(xobj + b'\x00')
return {"illuminance": value}
# Dataobject dictionary
# {dataObject_id: (converter, bytes in data object)}
dataobject_dict = {
0x01: (objHomebrew_count, 4),
0x02: (objHomebrew_temp, 2),
0x03: (objHomebrew_rh, 2),
0x04: (objHomebrew_lux, 3),
}
If you can get your sensor advertisement data to fit that format (described in the same file, or in my ESP32_BLEPulseCounter repository code), and turn logging on, you should then be able to see your data being received in the HA log.
HA probably won't create the sensor entities for you at this stage. To handle that some changes might be needed in custom_components/ble_monitor/sensor.py
too as the homebrew code will have added "0" to the end of temp, humidity and illuminesence just in case there are multiple of these in the same advertisement. I won't look at that just now though...
In fact, as you only have one of each per advertisement, you could just change the homebrew.py line not to add the index...from this:
newVal[key + str(resfuncCounter[resfunc])] = thisVal[key]
to this:
newVal[key] = thisVal[key]
That change would break my pulse counting homebrew sensor, so wouldn't make it into the main branch of my git repo, but would be the quickest way for you get get the sensor entities showing up in HA as a test.
Thx @scrambledleek! I will see if I can do this.
I started experimenting with the iBeacon protocol, as this is a standard BLE example on how to use BLE on an Espressif ESP32 in Arduino IDE. https://circuitdigest.com/microcontroller-projects/esp32-based-bluetooth-ibeacon
@dmamontov's work allows to send my own data (without setting up connections on wifi/zigbee/normal bluetooth) to HA on a raspberry pi just by adapting the above example and changing continuously the major/minor data. I've adapted it already not to sleep (ESP32 stopped running after some hours), just created a loop, and I've added a watchdog timer. By using multiple UUID's on the same controller/MAC I can also send alternatingly multiple sensor values with one ESP32.
The iBeacon protocol is very simple: a UUID (128 bit) that can be freely generated/"allocated" on https://www.uuidgenerator.net/ and two 16-bit values, so any sensor can be given a unique identifier (like IPv6) and broadcast 2 sensor values. The only downside is that iBeacon is proprietary to Apple.
there is altbeacon as an alternative
The altbeacon is really to be used only as a beacon, it has a 20-byte identifier, but no real data fields. For DIY projects we should have/use a format over the BLE advertising function that combines the following:
I don't know if the BLE stack automatically adds a CRC value to discard wrong received packages, otherwise a CRC check would also be valuable.
I'll leave it here as an idea:
As BLE format I suggest to use CayenneLPP it's very simple, contains well defined data (from LwM2M standard) and used in areas required low bandwidth and low power consumption like Lora WAN and NB-IoT. To get idea how it's looks, check this data types spec (look for IPSO Alliance types) and js decoder for basic types.
HA sensors should be generated on the fly according data type of received values, this will allow to avoid the need to transfer the description of the sensors by the device. In case when user will need to exclude or disable by default some types of sensors from a specific device or globally, data type ids of this sensors can be specified in the configuration (yaml or GUI). Default sensor name (suffix or prefix), unit of measurements, etc will be taken from data type definition.
I like the idea from @myhomeiot, so I've created a new issue where I keep track of the progress. The format shouldn't be that hard to implement, but the auto-generating of sensors will require some adjustments. See #689
I was thinking that we can also use the official GATT Characteristic and Object Type
e.g. 05162A1964 would than decode to
05
= length = 5 bytes
16
= Service data
2A19
= Battery
64
= value = 100 (%)
Requirements of the value byte(s) can be found here https://www.bluetooth.org/DocMan/handlers/DownloadDoc.ashx?doc_id=524815
E.g. for battery, it's this
The main advantage I see is that we stick to the official BLE GATT Characteristic and Object Type
definition, which is documented by the Bluetooth organization
@Ernst79 The data will take a bit more space compared to the CayenneLPP, but it looks great and the most important that this data packages will be standard and can be interpreted by other systems.
I doesn't found the usual button pressed/released characteristic, but I think it can be emulated with some existing in specification characteristic.
BLE monitor 7.7.0 adds initial testing support for your own Do It Yourself (DIY) sensors. The BLE format (called HA BLE
) that has to be used by your DIY sensor is following the official Bluetooth format (GATT Characteristic and Object Type). More explanation is given in the documentation.
If you need a different sensor type than the currently implemented ones, let me know!
Important note
Note that in the current release, you will have to define your sensors manually in const.py
. This file will be overwritten when updating BLE monitor, so make a note of the added sensors. In a future release, I will change it such that the sensors are added automatically, based on what type of measurement is broadcasted. Till then, you will have to work with this temporary workaround.
in the documentation.
0x2A6D | pressure | uint32 (4 bytes) | 0.001 | 07166D2A78091000 | 1051.0 ? Unit is in pascals with a resolution of 0.1 Pa 0x00100978 -> 1051000 -> 105100.0 Pa
If you want another sensor, let us know by creating a new issue on Github.
Digital signals? Switches, Keys, Relay, Counts, Times (UTC) ... ?
0x2A56 The Digital characteristic is used to expose and change the state of an IO Modules digital signals. The Digital characteristic is an array of n 2-bit values in a bit field
0x2A4D The Report characteristic is used to exchange data between a HID Device and a HID Host. The Report characteristic value contains Input Report, Output Report or Feature Report data to be transferred between the HID Device and HID Host.
How to encrypt data?
Are (BT5.0) LE Advertising Extensions supported?
This packet could contain up to 254 bytes payload. An example of an AdBLE-MQTT data package:
{"humidity":41,"battery":100,"temperature":13.3}
Are (BT5.0) LE Advertising Extensions supported?
Yes. Before starting the scan, the adapter supported features are checked, and LE extended scan is launched if supported.
0x2A6D | pressure | uint32 (4 bytes) | 0.001 | 07166D2A78091000 | 1051.0 ? Unit is in pascals with a resolution of 0.1 Pa 0x00100978 -> 1051000 -> 105100.0 Pa
Pressure sensor in Home Assistant is in hPa
, see https://www.home-assistant.io/integrations/sensor/ I'll add a note in the docs.
Encryption of data is a good idea, I'll also add the suggested sensors you mentioned.
Digital State Bits: 0x2A56 Analog Values: 0x2A58 (Have an association with "Aggregate IO" 0x2A5A, usage example: Infineon-CE217613_Bluetooth_LowEnergy(BLE)_Automation_IO )
Count (24 bits) 0x2AEB Count (16bits) 0x2AEA (Has support in "nRF Connect", shows values in advertise Count: ....)
Count sensors are both added in 7.9.1-beta. I'm first going to focus on "automatic" sensor creation now, as the manually modification of const.py
isn't the most user friendly way 😄
The modified LYWSD03MMC has the following data to transmit:
To transfer these values, at least 9 UUID identifiers are required. Each value with a UUID uses a minimum of 5 bytes for an 8-bit value. For 16-bit already 6 bytes, for 32 - 8 bytes. The maximum information in the advertising package is only 31 bytes. And in the "HA_BLE" format, you will need to transfer more than 50 bytes plus 8 bytes of the "HA_BLE" name. And no encryption!
If transmitted in parts, this requires a reduction in the transmission period, which excludes the use in stand-alone devices and is a pollution of the radio air. As a result, there is no way to use the proposed format in home devices... Using the name "HA_BLE" prevents users from setting their own names to distinguish and easily identify their devices. The advertise package already contains the MAC and is an identifier. Random MAC is not desirable for stationary devices.
Typical encryption in BLE requires 8 additional bytes (analog mija). Typical, because the BLE device already contains procedures or hardware devices to support AES CCM encryption.
@pvvx I understand the issue, but I'm not sure we can put all the data in one advertisements.
Xiaomi sensors normally also send only one or two properties in one advertisements and just send multiple advertisements, all with one or two properties. e.g. Battery data is almost always send in a separate data packet, often at a totally different interval (e.g. once per 10 minutes). The length of Xiaomi MiBeacon data is similar, also 1 length byte, 2 bytes UUID identifiers and 1 or more bytes for the actual data.
Perhaps you can split up the data that you want to send very often, and data that you don't want to send that often (like battery voltage, battery level in %).
I think you have two choices, one is to extend your own ATC format, but I guess you run into the same issue of limited length. The other is that we extend the HA_BLE format with the missing events, which should not be a problem.
Regarding encryption, I agree that encryption should be an option, but I have no experience adding encryption, so it will take me a lot of time to investigate and to add this. Moreover, next month, I'm going to move, so will have limited time to do a lot of stuff. For now, I have the following priority list
0x2A6E
0x2A6F
0x2B18
0x2A19
0x2AEA
or 0x2AEB
const.py
Regarding the two analog voltage inputs, I have to think how we can have multiple sensors of the same type (voltage
), and how to distinguish between these sensors. This isn't possible in the current set up.
The largest power consumption of a BLE device occurs during transmission. Each additional byte transferred is a reduction in battery life. The dependence of the battery life of a BLE device includes the main factor: 12/(12 + additional_data_size)
Xiaomi sensors normally also send only one or two properties in one advertisements and just send multiple advertisements, all with one or two properties.
In version 5, only one property per advertisements. All Xiaomi sensors transmit BLE advertising every 1..1.6 seconds. Some are distinguished by the absence of a connection flag and name polling. As a result, the average battery life is 8 months. It makes no sense to choose the worst of the worst.
According to the technical characteristics of modern SoCs with BLE and other conditions, there is such a dependence: An advertising transmission interval of less than 1 second does not allow reaching the level of operation from typical batteries for more than 1 year. An advertising transmission interval of more than 10 seconds does not allow connection with the device. BLE standard. Increasing the advertising interval beyond 10 seconds does not improve battery life. Does not give real more than 10% with completely disabled advertising.
Duplication of advertising is necessary, because. some use ESP as a receiver. This SoC has a lot of losses in BLE, especially when working with WiFi at the same time.
Ok, I understand. But what do you propose to make it more efficient than?
One thing I can think of is changing the shortened_local_name
to HA
, which requires only 4 bytes (07084841
).
I don't see an easy way to support a wide variety of sensor types without the need of an UUID to identify which data it is. I guess with two advertisements, you can send all the data.
Format option:
As recommended by BLE, a typical advertisement contains 'BLE Discovery Mode flags' - this is 3 bytes. Without 'flags', the device does not show up in the Windows Bluetooth system menu. Out of 31 bytes, 28 bytes remain. The UUID16 identifier takes 4 more bytes. 24 bytes remain for user data. When encrypting an AES CCM message, another 8 bytes are taken away. The remainder is only 16 bytes. Fragment length indication is limited to 5 bits (0..31).
Data Structure Format:
SizeAndType, DataID, [byte0[..byte N]]
SizeAndType: Bit0..4 - size 0..31 The high bits (bit5,6,7) can represent 7 data types. Bit5..7 - type 0..7 sample: | N | Type |
---|---|---|
0 | Binary unsigned | |
1 | Binary signed | |
2 | Float | |
3 | String | |
4..7 | Reserved |
'DataID' may describe a variable name from a user installation and/or generic device classes.
For the user data header, you must select 2 generic UUIDs. One for unencrypted data and one for encrypted data.
In the encrypted data, the last 8 bytes of the message mean: 32 bits - the counter and 32 bits "Message integrity check (MIC)". "Bindkey" is used - by analogy with the mija format.
In this format, it is possible to transfer four 16-bit values simultaneously and in encrypted form.
Chrome Bluetooth API does not return MAC address in java script. To include in the MAC message and save 1 byte, you can add to 'SizeAndType': | H | Type |
---|---|---|
0 | Binary unsigned | |
1 | Binary signed | |
2 | Floating | |
3 | String | |
4 | MAC | |
5..7 | Reserved |
Then for MAC DataID is not transmitted.
OK, I like your proposal. I also have a an issue with the GATT specifications, which seem to miss quite some types. e.g. there is nothing that even looks like an open/close sensor (either a door sensor or a switch sensor).
Can you show me an example how an data packet looks like. I'm struggling a bit with the SizeAndType byte. How do I get the length from this byte, e.g. when I have two dataIDs, e.g. temperature and humidity?
Assume that the following DataID values are defined: 0x55 - Temperature in 0.01 C 0xAA - Humidity at 0.01% Then for a temperature of + 25C there are several options: | SD | DID | Data | |
---|---|---|---|---|
03 | 55 | C409 | uint16 | |
23 | 55 | C409 | int16 | |
04 | 55 | C40900 | uint24 | |
24 | 55 | C40900 | int24 | |
05 | 55 | C4090000 | uint32 | |
25 | 55 | C4090000 | int32 | |
29 | 55 | C409000000000000 | int64 :) | |
45 | 55 | float32? | float32 | |
49 | 55 | float64? | float64 | |
63 | 55 | 3235 | string "25" | |
66 | 55 | 32352E3031 | string "25.01" |
And set UUID16 for unencrypted data = 0x1234 and humidity 50.55%
AdvUUID16 | Temp | Humi |
---|---|---|
0B163412 | 2355C409 | 03AABF13 |
(either a door sensor or a switch sensor)
If you do not feel sorry for the DID numbers, then it is possible to describe individual events without data: | SD | DID | |
---|---|---|---|
01 | 23 | door opened | |
01 | 24 | door closed |
But it is not recommended to do so... Let it be with data: | SD | DID | Data |
---|---|---|---|
02 | 23 | 01 | door opened |
02 | 23 | 00 | door closed |
If the data is not declared, then the default value is always taken - for example = 0
The problem remains in the Chrome API for javascript with ad encryption. When scanning, do not find out the MAC. But when connected, the device can use standard encryption based on a pin code and there is no need for its own encryption...
Ok, I will change this in HA BLE according to your proposal. Better to do it right now, than changing it later.
Strange that the Chrome API can’t find the MAC with scanning, the MAC is always in the first part of an BLE advertisement. But I’m not familiar with the API.
Strange that the Chrome API can’t find the MAC with scanning, the MAC is always in the first part of an BLE advertisement. But I’m not familiar with the API.
For security reasons, the user selects a device in a separate menu, and the API provides access only to the data of the selected device. And in the device selection request, you must describe all the UUIDs that you will work with.
@pvvx. I made a very first draft. I need to add more sensors, add some Error handling, examples, tests, etc. but the first draft is in the new-HA_BLE branch. It isn't beta ready yet, but if you want to try, overwrite the bleparser/__init__.py
and ha_ble.py
files and add the ha_ble_legacy.py
file.
Format is explained here (for now)
https://github.com/custom-components/ble_monitor/blob/new-HA_BLE/docs/ha_ble.md
Average consumption at 3.3V Xiaomi LYWSD03MMC B1.4 (default settings):
Custom format 13.7 uA (adv_size: 19 bytes)
HA_BLE format 14.6 uA (adv_size: 22 bytes)
'HA BLE DIY' : [["temperature", "humidity", "battery", "voltage", "rssi", "count"], [], ["switch", "opening"]]
"temperature", "humidity", "battery", "voltage", "rssi" - worked ok. "count", "switch", "opening" - not work.
typedef enum {
HaBleID_PacketId = 0, //0x00, uint8
HaBleID_battery, //0x01, uint8, %
HaBleID_temperature, //0x02, sint16, 0.01 °C
HaBleID_humidity, //0x03, uint16, 0.01 %
HaBleID_pressure, //0x04, uint24, 0.01 hPa
HaBleID_illuminance, //0x05, uint24, 0.01 lux
HaBleID_weight, //0x06, uint16, 0.01 kg
HaBleID_weight_s, //0x07, string, kg
HaBleID_dewpoint, //0x08, sint16, 0.01 °C
HaBleID_count, //0x09, uint8/16/24/32
HaBleID_energy, //0x0A, uint24, 0.001 kWh
HaBleID_power, //0x0B, uint24, 0.01 W
HaBleID_voltage, //0x0C, uint16, 0.001 V
HaBleID_pm2x5, //0x0D, uint16, kg/m3
HaBleID_pm10, //0x0E, uint16, kg/m3
HaBleID_boolean //0x0F, uint8
} HaBleIDs_e;
// Type bit 5-7
typedef enum {
HaBleType_uint = 0,
HaBleType_sint = (1<<5),
HaBleType_float = (2<<5),
HaBleType_string = (3<<5),
HaBleType_MAC = (4<<5)
} HaBleTypes_e;
typedef struct __attribute__((packed)) _adv_na_ble_ns1_t {
uint8_t size; // = 21?
uint8_t uid; // = 0x16, 16-bit UUID
uint16_t UUID; // = 0x181C, GATT Service HA_BLE
uint8_t p_st;
uint8_t p_id; // = HaBleID_PacketId
uint8_t pid; // PacketId (measurement count)
uint8_t t_st;
uint8_t t_id; // = HaBleID_temperature
int16_t temperature; // x 0.01 degree
uint8_t h_st;
uint8_t h_id; // = HaBleID_humidity
uint16_t humidity; // x 0.01 %
uint8_t b_st;
uint8_t b_id; // = HaBleID_battery
uint8_t battery_level; // 0..100 %
uint8_t v_st;
uint8_t v_id; // = HaBleID_voltage
uint16_t battery_mv; // mV
} adv_ha_ble_ns1_t, * padv_ha_ble_ns1_t;
typedef struct __attribute__((packed)) _adv_na_ble_ns2_t {
uint8_t size; // = 15?
uint8_t uid; // = 0x16, 16-bit UUID
uint16_t UUID; // = 0x181C, GATT Service 0x181C
uint8_t p_st;
uint8_t p_id; // = HaBleID_PacketId
uint8_t pid; // PacketId (!= measurement count)
uint8_t f_st;
uint8_t f_id; // = HaBleID_boolean ?
uint8_t flags;
uint8_t c_st;
uint8_t c_id; // = HaBleID_count
uint8_t counter[3]; // count
} adv_ha_ble_ns2_t, * padv_ha_ble_ns2_t;
...
padv_ha_ble_ns1_t p = (padv_ha_ble_ns1_t)&adv_buf.data;
p->size = sizeof(adv_ha_ble_ns1_t) - sizeof(p->size);
p->uid = GAP_ADTYPE_SERVICE_DATA_UUID_16BIT; // 16-bit UUID
p->UUID = ADV_HA_BLE_NS_UUID16;
p->p_st = HaBleType_uint + sizeof(p->p_id) + sizeof(p->pid);
p->p_id = HaBleID_PacketId;
p->pid = (uint8_t)cnt;
p->t_st = HaBleType_sint + sizeof(p->t_id) + sizeof(p->temperature);
p->t_id = HaBleID_temperature;
p->temperature = measured_data.temp; // x0.01 C
p->h_st = HaBleType_uint + sizeof(p->h_id) + sizeof(p->humidity);
p->h_id = HaBleID_humidity;
p->humidity = measured_data.humi; // x0.01 %
p->b_st = HaBleType_uint + sizeof(p->b_id) + sizeof(p->battery_level);
p->b_id = HaBleID_battery;
p->battery_level = measured_data.battery_level;
p->v_st = HaBleType_uint + sizeof(p->p_id) + sizeof(p->battery_mv);
p->v_id = HaBleID_voltage;
p->battery_mv = measured_data.battery_mv; // x mV
...
padv_ha_ble_ns2_t p = (padv_ha_ble_ns2_t)&adv_buf.data;
p->size = sizeof(adv_ha_ble_ns2_t) - sizeof(p->size);
p->uid = GAP_ADTYPE_SERVICE_DATA_UUID_16BIT; // 16-bit UUID
p->UUID = ADV_HA_BLE_NS_UUID16;
p->p_st = HaBleType_uint + sizeof(p->p_id) + sizeof(p->pid);
p->p_id = HaBleID_PacketId;
p->pid = (uint8_t)cnt;
p->f_st = HaBleType_uint + sizeof(p->f_id) + sizeof(p->flags);
p->f_id = HaBleID_boolean;
p->flags = trg.flg_byte;
p->c_st = HaBleType_uint + sizeof(p->c_id) + sizeof(p->counter);
p->c_id = HaBleID_count;
p->counter[0] = rds.count_byte[0];
p->counter[1] = rds.count_byte[1];
p->counter[2] = rds.count_byte[2];
adv_buf.data_size = sizeof(adv_ha_ble_ns2_t);
bls_ll_setAdvData((u8 *)&adv_buf.data, adv_buf.data_size);
Write base HA from reboot (4 GB per day):
Removing the device helped.
"Counter" appeared in the device components
At the moment there is no "switch" and "opening".
I have added the opening
and switch
binary sensors now. However, I do see an issue with these sensors, they are updated in HA on every message (as normally, you want to cache every push on a switch, even if it is the same state (e.g. On
). If you are going to send the switch state every few seconds, this would mean a state update every few seconds in HA.
There are two ways to solve this.
switch
type in HA BLE that only updates on state changes (we have something similar for non-binary sensor (look for StateChangedSensor
)However, I do see an issue with these sensors, they are updated in HA on every message.
There are no issue here. The current optional variant was created specifically for the test. All this is not yet included in the version for users. For the release version, there will be a completely different algorithm. The release version requires encryption support.
The current version of 'TelinkMiFlasher.html' and options for new betas are already built in and working:
In the "counter" mode, the counter is transmitted only by "report interval" and when the count overflows 16 bits (device cannot store more than 16 counter bits across hard/soft resets. But the counter size is 32 bits.). In the "switch" mode - by change event, and confirmations go through the "report interval".
Great.
I'm a bit worried about the 4 GB per day, you showed. Not sure where this is coming from.
I have now added the possibility to specify a MAC in the payload. You can even use a different MAC, to make some "virtual" device.
One question for you. I have now specified a fixed "factor" per measurement type, to get a sufficient number of digits. Is this "workable" for you? Or should we add the possibility to overrule the factor?
I'm a bit worried about the 4 GB per day, you showed. Not sure where this is coming from.
This is normal "HA" behavior with default settings.
cat /sys/block/mmcblk0/stat | awk '{printf "Uptime read: %.3fMiB (%.1f%% I/Os merged) written: %.3f MiB (%.1f%% I/Os merged)\n", $3*512/1048576, $2/$1*100, $7*512/1048576, $6/$5*100}'
The actual size of the overwrite is larger, because disk devices rewrite not one sector of 512 bytes, but a block of 16 or more kilobytes. I use an SSD and at 4GB per day it will work for years...
One question for you. I have now specified a fixed "factor" per measurement type, to get a sufficient number of digits. Is this "workable" for you? Or should we add the possibility to overrule the factor?
Let's look at the requests as the format evolves.
The first release with "HA_BLE" in version 3.7 is ready.
Log (temperature and humidity = ADC1 and ADC2 for test):
04:21:24: 582D340CCBA4, RSSI: -62, Data: 181C:0200E823027B040303DB03020164 HA-BLE, PacketId: 232, temperature: 11.470°C, humidity: 9.870%, battery: 100%
04:21:26: 582D340CCBA4, RSSI: -62, Data: 181C:0200E823027B040303DB03020164 HA-BLE, PacketId: 232, temperature: 11.470°C, humidity: 9.870%, battery: 100%
04:21:29: 582D340CCBA4, RSSI: -70, Data: 181C:0200E9021001030CA70C HA-BLE, PacketId: 233, switch: 1, voltage: 3.239V
04:21:34: 582D340CCBA4, RSSI: -70, Data: 181C:0200EA2302D3040303F303020164 HA-BLE, PacketId: 234, temperature: 12.350°C, humidity: 10.110%, battery: 100%
04:21:36: 582D340CCBA4, RSSI: -70, Data: 181C:0200EA2302D3040303F303020164 HA-BLE, PacketId: 234, temperature: 12.350°C, humidity: 10.110%, battery: 100%
04:21:39: 582D340CCBA4, RSSI: -78, Data: 181C:0200EA2302D3040303F303020164 HA-BLE, PacketId: 234, temperature: 12.350°C, humidity: 10.110%, battery: 100%
04:21:41: 582D340CCBA4, RSSI: -70, Data: 181C:0200EB021001030C960C HA-BLE, PacketId: 235, switch: 1, voltage: 3.222V
04:21:44: 582D340CCBA4, RSSI: -70, Data: 181C:0200EC2302BC050303C404020164 HA-BLE, PacketId: 236, temperature: 14.680°C, humidity: 12.200%, battery: 100%
04:21:46: 582D340CCBA4, RSSI: -62, Data: 181C:0200EC2302BC050303C404020164 HA-BLE, PacketId: 236, temperature: 14.680°C, humidity: 12.200%, battery: 100%
04:21:49: 582D340CCBA4, RSSI: -62, Data: 181C:0200EC2302BC050303C404020164 HA-BLE, PacketId: 236, temperature: 14.680°C, humidity: 12.200%, battery: 100%
04:21:51: 582D340CCBA4, RSSI: -62, Data: 181C:0200ED021001030C980C HA-BLE, PacketId: 237, switch: 1, voltage: 3.224V
04:21:54: 582D340CCBA4, RSSI: -62, Data: 181C:0200EE2302550503038B04020164 HA-BLE, PacketId: 238, temperature: 13.650°C, humidity: 11.630%, battery: 100%
04:21:55: 582D340CCBA4, RSSI: -62, Data: 181C:0200EF021100050969000000 HA-BLE, PacketId: 239, opened: 0, count: 105
04:21:55: 582D340CCBA4, RSSI: -62, Data: 181C:0200EF021100050969000000 HA-BLE, PacketId: 239, opened: 0, count: 105
04:21:55: 582D340CCBA4, RSSI: -62, Data: 181C:0200EF021100050969000000 HA-BLE, PacketId: 239, opened: 0, count: 105
04:21:55: 582D340CCBA4, RSSI: -62, Data: 181C:0200EF021100050969000000 HA-BLE, PacketId: 239, opened: 0, count: 105
04:21:56: 582D340CCBA4, RSSI: -70, Data: 181C:0200F02302550503038B04020164 HA-BLE, PacketId: 240, temperature: 13.650°C, humidity: 11.630%, battery: 100%
04:21:57: 582D340CCBA4, RSSI: -62, Data: 181C:0200F1021101050969000000 HA-BLE, PacketId: 241, opened: 1, count: 105
04:21:57: 582D340CCBA4, RSSI: -62, Data: 181C:0200F1021101050969000000 HA-BLE, PacketId: 241, opened: 1, count: 105
04:21:57: 582D340CCBA4, RSSI: -62, Data: 181C:0200F1021101050969000000 HA-BLE, PacketId: 241, opened: 1, count: 105
04:21:57: 582D340CCBA4, RSSI: -62, Data: 181C:0200F22302550503038B04020164 HA-BLE, PacketId: 242, temperature: 13.650°C, humidity: 11.630%, battery: 100%
04:21:59: 582D340CCBA4, RSSI: -70, Data: 181C:0200F3021001030C960C HA-BLE, PacketId: 243, switch: 1, voltage: 3.222V
04:22:02: 582D340CCBA4, RSSI: -70, Data: 181C:0200F32302550503038B04020164 HA-BLE, PacketId: 243, temperature: 13.650°C, humidity: 11.630%, battery: 100%
Excellent. I have started working on the encryption now. Will use the same type as you have in the ATC firmware.
Encryption... The size of the counter/packet ID must be 32 bits. mi_encrypt_beacon uses 32 bits for the counter. With a smaller number of bits, the protection level drops (theoretically, without taking into account the transmitted temperature and humidity as additional randoms).
My suggestion:
In the encrypted data, the last 8 bytes of the message mean: 32 bits - the counter and 32 bits "Message integrity check (MIC)". "Bindkey" is used - by analogy with the mija format.
Package type: | Head UUID16 | crypted data | count | mic |
---|---|---|---|---|
4 bytes | xxxxxxx | 32 bits | 32 bits |
The ATC and PVVX encryption example uses an 8-bit counter to achieve maximum power savings (minimum data packet size) and low cryptographic strength. The use of AES:CCM was taken for the sake of compatibility.
PS: If you do not provide fake theoretical security that is not used in a real process, then there will be many people who want to declare their literacy after reading documentation on cryptography...
Ok, will use that. This encryption stuff is new for me.
@pvvx I start to understand it a bit (the encoding/decoding). I also managed to add the decryption (locally) for my own test, but I can't figure out where the counter
fits in the picture. I found a script on your GitHub, which I have slightly modified to get closer to HA BLE. But I don't see a counter in that script.
Should the 4 bytes counter be part of the nonce
?
import struct
import binascii
from Cryptodome.Cipher import AES
def parse_value(hexvalue):
vlength = len(hexvalue)
# print("vlength:", vlength, "hexvalue", hexvalue.hex(), "typecode", typecode)
if vlength >= 3:
temp = round(int.from_bytes(hexvalue[2:4], "little", signed=False) * 0.01, 2)
humi = round(int.from_bytes(hexvalue[6:8], "little", signed=False) * 0.01, 2)
print("Temperature:", temp, "Humidity:", humi)
return 1
print("MsgLength:", vlength, "HexValue:", hexvalue.hex())
return None
def decrypt_payload(payload, key, nonce):
mic = payload[-4:] # mic
cipherpayload = payload[:-4] # EncodeData
print("Nonce: %s" % nonce.hex())
print("CryptData: %s" % cipherpayload.hex(), "Mic: %s" % mic.hex())
cipher = AES.new(key, AES.MODE_CCM, nonce=nonce, mac_len=4)
cipher.update(b"\x11")
data = None
try:
data = cipher.decrypt_and_verify(cipherpayload, mic)
except ValueError as error:
print("Decryption failed: %s" % error)
return None
print("DecryptData:", data.hex())
print()
if parse_value(data) != None:
return 1
print('??')
return None
def decrypt_aes_ccm(key, mac, data):
print("MAC:", mac.hex(), "Binkey:", key.hex())
print()
adslength = len(data)
if adslength > 8 and data[0] <= adslength and data[0] > 7 and data[1] == 0x16 and data[2] == 0x1C and data[3] == 0x18:
pkt = data[:data[0]+1]
# nonce: mac[6] + head[4] + cnt[1]
nonce = b"".join([mac, pkt[:4]])
return decrypt_payload(pkt[4:], key, nonce)
else:
print("Error: format packet!")
return None
#=============================
# main()
#=============================
def main():
print()
print("====== Test encode -----------------------------------------")
temp = 25.06
humi = 50.55
print("Temperature:", temp, "Humidity:", humi)
print()
data = bytes(bytearray.fromhex('2302CA090303BF13'))
mac = binascii.unhexlify('5448E68F80A5') # MAC
binkey = binascii.unhexlify('231d39c1d7cc1ab1aee224cd096db932')
print("MAC:", mac.hex(), "Binkey:", binkey.hex())
adshead = struct.pack(">BBH", len(data) + 7, 0x16, 0x1C18) # ad struct head: len, id, uuid16
beacon_nonce = b"".join([mac, adshead])
cipher = AES.new(binkey, AES.MODE_CCM, nonce=beacon_nonce, mac_len=4)
cipher.update(b"\x11")
ciphertext, mic = cipher.encrypt_and_digest(data)
print("Data:", data.hex(), adshead.hex())
print("Nonce:", beacon_nonce.hex())
print("CryptData:", ciphertext.hex(), "Mic:", mic.hex())
adstruct = b"".join([adshead, ciphertext, mic])
print()
print("AdStruct:", adstruct.hex())
print()
print("====== Test decode -----------------------------------------")
decrypt_aes_ccm(binkey, mac, adstruct);
if __name__ == '__main__':
main()
At the entry we have:
Ad Structure: | Head_UUID16 | crypted_data | count_id | mic |
---|---|---|---|---|
4 bytes | N bytes | 4 bytes | 4 bytes |
Head_UUID16: | size | id | UUID16 |
---|---|---|---|
1 bytes | 1 bytes | 2 bytes |
nonce = b"".join([mac, uuid16, count_id]) # 6+2+4 = 12 bytes
cipher = AES.new(bindkey, AES.MODE_CCM, nonce=nonce, mac_len=4)
cipher.update(b"\x11")
decrypted_data = cipher.decrypt_and_verify(crypted_data, mic)
or
nonce = b"".join([mac, head_uuid16, count_id]) # 6+4+4 = 14 bytes
decrypted_data = HA-BLE Data Array
You can accept the condition that 'count_id' should only increase in the positive direction. If in 5 seconds there was a change in 'count_id' by more than 100 units - a warning "attack - manipulation of encrypted data" and/or message is not accepted.
If 'count_id' does not differ from the previous one for more than 16 data receptions or within an hour - a similar warning "Attack - manipulation of encrypted data" and/or message is not accepted. (If 'count_id' does not differ from the previous one, then in the current design such a message is not accepted, but is considered a duplication.)
If 'count_id' differs negatively (starts counting from a new value), then the message is rejected and only subsequent messages that meet the conditions described above are allowed. :)
Added encryption in the HA_BLE-new branch.
Below the script to encrypt the message (and decrypt). Will add that to the docs later, after some cleaning.
import struct
import binascii
from Cryptodome.Cipher import AES
def parse_value(hexvalue):
vlength = len(hexvalue)
# print("vlength:", vlength, "hexvalue", hexvalue.hex(), "typecode", typecode)
if vlength >= 3:
temp = round(int.from_bytes(hexvalue[2:4], "little", signed=False) * 0.01, 2)
humi = round(int.from_bytes(hexvalue[6:8], "little", signed=False) * 0.01, 2)
print("Temperature:", temp, "Humidity:", humi)
return 1
print("MsgLength:", vlength, "HexValue:", hexvalue.hex())
return None
def decrypt_payload(payload, mic, key, nonce):
print("Nonce: %s" % nonce.hex())
print("CryptData: %s" % payload.hex(), "Mic: %s" % mic.hex())
cipher = AES.new(key, AES.MODE_CCM, nonce=nonce, mac_len=4)
cipher.update(b"\x11")
try:
data = cipher.decrypt_and_verify(payload, mic)
except ValueError as error:
print("Decryption failed: %s" % error)
return None
print("DecryptData:", data.hex())
print()
if parse_value(data) != None:
return 1
print('??')
return None
def decrypt_aes_ccm(key, mac, data):
print("MAC:", mac.hex(), "Bindkey:", key.hex())
print()
adslength = len(data)
if adslength > 15 and data[0] == 0x1E and data[1] == 0x18:
pkt = data[:data[0] + 1]
uuid = pkt[0:2]
encrypted_data = pkt[2:-8]
count_id = pkt[-8:-4]
mic = pkt[-4:]
# nonce: mac [6], uuid16 [2], count_id [4] # 6+2+4 = 12 bytes
nonce = b"".join([mac, uuid, count_id])
return decrypt_payload(encrypted_data, mic, key, nonce)
else:
print("Error: format packet!")
return None
# =============================
# main()
# =============================
def main():
print()
print("====== Test encode -----------------------------------------")
temp = 25.06
humi = 50.55
print("Temperature:", temp, "Humidity:", humi)
print()
data = bytes(bytearray.fromhex('2302CA090303BF13')) # HA BLE data
count_id = bytes(bytearray.fromhex('00112233')) # count id
mac = binascii.unhexlify('5448E68F80A5') # MAC
uuid16 = b"\x1E\x18"
bindkey = binascii.unhexlify('231d39c1d7cc1ab1aee224cd096db932')
print("MAC:", mac.hex(), "Binkey:", bindkey.hex())
nonce = b"".join([mac, uuid16, count_id]) # 6+2+4 = 12 bytes
cipher = AES.new(bindkey, AES.MODE_CCM, nonce=nonce, mac_len=4)
cipher.update(b"\x11")
ciphertext, mic = cipher.encrypt_and_digest(data)
print("Data:", data.hex())
print("Nonce:", nonce.hex())
print("CryptData:", ciphertext.hex(), "Mic:", mic.hex())
adstruct = b"".join([uuid16, ciphertext, count_id, mic])
print()
print("AdStruct:", adstruct.hex())
print()
print("====== Test decode -----------------------------------------")
decrypt_aes_ccm(bindkey, mac, adstruct)
if __name__ == '__main__':
main()
Fixed the packet id. It is taking the packet id
counter
that is used for the encryption. Will be overruled when using the packet id
in the payloadpacket id
in the payload
Request by @scrambledleek to create support for home brew BLE sensors. BLE monitor should: