Open pfink opened 6 years ago
Hi @pfink, thanks for that.
I'm happy to help you, let's start with getting them into inbox first.
OMG... Before my first post, I troubleshooted two days to get the buttons discovered including that I tried to manually add them, everything unsuccessful. The solution was as dumb as easy: I did not press the button often enough :( As you suggested, I flooded openHAB now with very often button presses (> 10) and now they're discovered..... Thank you very much!
Via online channel and by putting online timeout to 1 sec, I'm now already able to detect whether a button was pressed at all. Next step would be to distinguish which button was pressed. My preferred solution would be to implement/offer a system.rawrocker
channel.
Of course, I'm also open to simpler/easier solutions for a start :)
Good stuff! Very interesting to see battery-less devices to work with the binding :) What channels have been discovered? Can you post a screenshot?
Sure! This is a complete list:
Device Name is just the MAC address.
how about "unknown" channels?
That's how you do stuff with BT binding. The binding should discover some channels (BT characteristics) automatically. So that you can use them. I assume there will be several channels per each button. If it is like that, you only need to extend the binding with some XML files to get the channels to be recognised automatically, I assume they will be radioboxes/switches in OH.
See here: https://github.com/sputnikdev/eclipse-smarthome-bluetooth-binding/blob/master/gatt-extensions.md
Once you've changed "GATT parsing strategy" you will need to wait until the binding reloads and then refresh PaperUI page with your things to see new channels. IMPORTANT: they will appear only after the binding receives data, e.g. you will need to switch on/off your buttons to discover all unknown channels.
This would be an example of "unknown" characteristics (MiFlora device):
000071fe-0000-1000-8000-00805f9b34fb
is an unknown characteristic.
I removed the Thing again, activated recognition of unknown channels as per your advice, and discovered + added it again. Where should I see these unknown channels? I don't find anything in PaperUI. I also checked /var/lib/openhab2/jsondb/org.eclipse.smarthome.core.thing.Thing.json
, but there seems to be nothing more than I already posted.
Try to switch on / off more, continuously switch it for 30-60 secs let's say :) . Wait some time, reload browser tab with PaperUI to see new channels.
How many buttons does your switch have?
Right, I've skimmed the doc you have attached.
The way how your switch transmit data is via so called "manufacturer data". The binding supports it, a new channel (single) should be created for it. That channel contains the following (as per the doc):
Sequence Counter (4 byte) The Sequence Counter is a continuously incrementing counter used for security processing. It is initialized to 0 at the time of production and incremented for each telegram (data telegram or commissioning telegram) sent. Switch Status (1 byte) The Switch Status field reports the button action. The encoding of this field is described in chapter 4.7. Optional Data (0 / 1 / 2 or 4 byte) PTM 215B provides the option to transmit additional user-defined data within each data telegram. This data can be used to identify user-specific properties. The length of the Optional Data field is defined in the Configuration register as described in chapter 6.7.7. Security Signature (4 byte) The Security Signature is used to authenticate PTM 215B radio telegrams as described in chapter 4.8
we are interested in Switch Status (1 byte)
.
I'll need to double check if the binding supports the "manufacturer data", should be.
I have rockers with 2 buttons and also those with 4 buttons. But I think there is the very same radio module behind them. Unfortunately, no additional channels get discovered. I guess this could be because the binding does not know the security key. I think the whole security and commissioning process described in the docs would make no sense if everyone can just publicly read the data ;-)
(see section 4.8 in the docs)
Right, I've checked it :) It is not implementing manufacturer data thingy, but it would be easy to add as this is very-very similar to an existing functionality (advertised service data).
That one implements "advertised service data" which is very similar to "manufacturer service data", in fact the later should be easier as there will be only 1 data point comparing to many for "service data".
Here is what needs to be implemented: https://github.com/sputnikdev/bluetooth-manager/blob/d239d59a5368d05d06f59dc62faeb3e355716f23/src/main/java/org/sputnikdev/bluetooth/manager/BluetoothSmartDeviceListener.java#L70
Let me know if it makes sense for you?
that argument manufacturerData
will contain exactly this:
Sequence Counter (4 byte) The Sequence Counter is a continuously incrementing counter used for security processing. It is initialized to 0 at the time of production and incremented for each telegram (data telegram or commissioning telegram) sent. Switch Status (1 byte) The Switch Status field reports the button action. The encoding of this field is described in chapter 4.7. Optional Data (0 / 1 / 2 or 4 byte) PTM 215B provides the option to transmit additional user-defined data within each data telegram. This data can be used to identify user-specific properties. The length of the Optional Data field is defined in the Configuration register as described in chapter 6.7.7. Security Signature (4 byte) The Security Signature is used to authenticate PTM 215B radio telegrams as described in chapter 4.8
Where key of that map would be 0x03DA
, and the value is the rest of it:
I'll do a detailed review within the next days, thank you so far! In case you have more hints, don't hesitate to append them here :) The longer I read the more I recognize that I probably misunderstood the docs. Now I think that I don't need the security key to read the data, I just need it to verify that it comes from my device and not from an attacker that imitates my device (but for my use cases, I think it's not necessary to protect against such attack scenarios).
On the weekend, I'll setup a proper development environment and try to get it running.
Yep, the doc does not say that it is absolutely required to implement security checks. For basic usage, it should be more than enough. The telegram message contains signature part, that one can be used to validate if the message is sent by the original device by using keys that was exchanged initially. Not sure if we need it though :), however it is nice to have.
The receiver performs the same signature calculation based on sequence counter, source address and the remaining telegram data of the received telegram using the security key it received from PTM 215B during commissioning. The receiver then compares the signature reported as part of the telegram with the signature it has calculated. If these two signatures match then the following statements are true: Sender (PTM 215B) and receiver use the same security key The message content (address, sequence counter, data) has not been modified
Effectively this ^^ is not required, the data should be unencrypted and available to read as far as I can see.
Yes, of course, I also see it definitely as a TODO for the binding. At the very latest when you use BLE devices to open your front door, you should have the posiibility to verify the origin of the telegrams ;-)
Have a look at the architecture design of the binding: https://github.com/sputnikdev/eclipse-smarthome-bluetooth-binding/blob/master/implementation-notes.md This should give you a bit of overview.
Hey @pfink, I've added a basic support for advertised manufacturer data into the binding, have a look here: https://github.com/sputnikdev/eclipse-smarthome-bluetooth-binding/blob/8bff492a68b54235aba284dce283f497df6058e5/src/main/java/org/sputnikdev/esh/binding/bluetooth/handler/BluetoothDeviceHandler.java#L175
This creates binary channels like so:
0x59 here is an id of a manufacturer that device reported:
89 0x0059 Nordic Semiconductor ASA
Could you please grub this snapshot and try your switch with it?
Obviously you will need to try to switch on/off multiple times like you did before in order to make your device to send some data so that the binding will pick it up.
Once you've discovered a new channel for your switch, we will need to think about how could we extend the binding so that it can parse binary telegram message into a set of switches.
Just so you know, at the moment the binding does not have ANY specific logic for any device, everything is made so that it is dynamically creates channels by using BT manager and GATT parser.
I'd like to continue this approach to make the binding as abstract as possible, however looking at the format of telegram message it seems to me it won't be possible as this format is not compatible with GATT specs due to that each field (switch state) depends also on the action type
field. Looks like we need to design a mechanism for the binding that allows us to extend it. First thing that comes to my mind is to use osqi DI to load some custom handlers for unknown/custom characteristics / service data / manufacturer data. For example, once BT device handler comes across an unknown entity, it will try to find a handler in OSGi registry by using a unique ID (in our case if would be UUID 00000059-...
) and load it if found.
What do you think?
I've added a basic support for advertised manufacturer data into the binding
Thank you very much!
Could you please grub this snapshot and try your switch with it?
Sure! In this version, one additional String channel named 000003da-0000-0000-0000-000000000000
is discovered. Anyhow, this channel does never send/contain any data, disregarding how often I click the buttons.
Looks like we need to design a mechanism for the binding that allows us to extend it.
Absolutely!
For example, once BT device handler comes across an unknown entity, it will try to find a handler in OSGi registry by using a unique ID (in our case if would be UUID 00000059-... ) and load it if found.
I think it would be better not to let the main binding decide upon which criteria things are distributed to extensions. Probably it makes more sense that all extensions are subscribed to everything and decide theirselves if something that the BLE binding discovers is relevant for them. This provides more flexibility implementing an extension.
Another question: It seems that there is an issue with reliability. With zwave and HomeMatic, I'm used to at least 98% reliability. For the Dolphin/EnOcean PTM 215B, it's 85-95% when the switch is 2.5 meters away from the adapter. On 8 meters distance, it already drops to 75-85% measured on the real operation as a wall switch. Of course, this is extremely bad for a wall switch.
Under lab/debugging conditions, the reliability is much higher, I'd say > 95%. That's because often (but not always) the first click after the device was not active for a longer time does not work, while nearly all subsequent clicks are recognized correctly. I observed a very similar behavior when I use Flic Buttons together with my Flic Button Binding, so I wouldn't consider this issue related to a specific device. Also, it does not seem to be related to Bluez as Bluez is not used for the Flic buttons. Additionally, it does not seem to depend on the BT adapter, at least I don't notice any changes when switching BT adapters. Do you have any idea about this or any recommendations to increase reliability? I already tried to:
ControllerMode = le
FastConnectable = true
Bluetooth devices update rate
to 1s.... but without any success / improvement in reliability, I believe.
Last but not least one more thing: With which bluez version would you recommend to setup my development environment? My Raspberry is working with 5.43. I'm tending to set it up based on Ubuntu Cosmic which provides the most recent version (5.50) which has significant changes compared to 5.43 (especially Mesh support).
Hi @pfink,
Good idea re giving the ability to choose if an extension can do something.
Re reliability, how do you calculate it if you don't even get the data right? :) That new channel, is it empty?
I think I know why you'd get a low reliability rate. Here is it why. I assume you are using a generic transport (not bluegiga transport)? Bluez framework is a terrific piece of junk, furthermore the generic transport talks to Bluez through TinyB library and DBus, this also adds a bit of complexity and less stability. Having said that, there is an issue with Bluez/Dbus/Tinyb (I have not figured out which component is contributing the most) when an object that is responsible for a device gets stale, it just does not report any failure just hangs like there was not any issue. So I have implemented a workaround for this case, it is very basic thing, the BM just checks how long ago was any communication from/to the object and if it exceeds a timeout, it initiates a reset for that object (releases it and gets a new "native" object). Can't remember the timeout, but it should be around 2 mins or so (maybe even less). That's why you probably loose the very first event from the device, because it tries to re-initialize a native object. This is to be confirmed though.
BTW, in general the blugiga transport is much more stable, however you will still get that "workaround" mechanism with it. I believe we need to come up with a setting that would control whether that workaround is on/off.
I'd recommend to switch to BLuegiga :) but I'd also recommend to stay with 4.43 as it proved to be the most stable.
Okay, ordered the BLED112 dongle now. It's always good for debugging to have the possibility to compare. Then I'll probably setup devenv based on 4.50 to have the newest and hottest shit which will probably not work, and after doing that I can use the BlueGiga for the serious stuff :D
reliability, how do you calculate it if you don't even get the data right? :)
As I said in the beginning, currently I'm using the online channel and an online timeout of 1s to just toggle the lights. With that setup, I have a "pilot" switch in my living room which I click from time to time. Then I capture if light got actually on or not.
That new channel, is it empty?
What do you mean with empty?
This one:
Sure! In this version, one additional String channel named 000003da-0000-0000-0000-000000000000 is discovered. Anyhow, this channel does never send/contain any data, disregarding how often I click the buttons.
Is it empty at all? No values? Any errors in logs?
Yes, no values. Regarding log, I just found these messages:
2018-12-02 20:35:29.562 [WARN ] [.core.thing.binding.BaseThingHandler] - Handler BluetoothDeviceHandler of thing bluetooth:ble:E215000053AF tried checking if channel online is linked although the handler was already disposed.
2018-12-02 20:35:29.567 [WARN ] [.core.thing.binding.BaseThingHandler] - Handler BluetoothDeviceHandler of thing bluetooth:ble:E215000053AF tried checking if channel last-updated is linked although the handler was already disposed.
2018-12-02 20:35:29.571 [WARN ] [.core.thing.binding.BaseThingHandler] - Handler BluetoothDeviceHandler of thing bluetooth:ble:E215000053AF tried checking if channel rssi is linked although the handler was already disposed.
2018-12-02 20:35:29.574 [WARN ] [.core.thing.binding.BaseThingHandler] - Handler BluetoothDeviceHandler of thing bluetooth:ble:E215000053AF tried checking if channel tx-power is linked although the handler was already disposed.
2018-12-02 20:35:29.578 [WARN ] [.core.thing.binding.BaseThingHandler] - Handler BluetoothDeviceHandler of thing bluetooth:ble:E215000053AF tried checking if channel estimated-distance is linked although the handler was already disposed.
2018-12-02 20:35:29.581 [WARN ] [.core.thing.binding.BaseThingHandler] - Handler BluetoothDeviceHandler of thing bluetooth:ble:E215000053AF tried checking if channel adapter is linked although the handler was already disposed.
2018-12-02 20:35:29.585 [WARN ] [.core.thing.binding.BaseThingHandler] - Handler BluetoothDeviceHandler of thing bluetooth:ble:E215000053AF tried checking if channel location is linked although the handler was already disposed.
2018-12-02 20:35:29.589 [WARN ] [.core.thing.binding.BaseThingHandler] - Handler BluetoothDeviceHandler of thing bluetooth:ble:E215000053AF tried checking if channel connected is linked although the handler was already disposed.
2018-12-02 20:35:29.592 [WARN ] [.core.thing.binding.BaseThingHandler] - Handler BluetoothDeviceHandler of thing bluetooth:ble:E215000053AF tried checking if channel connection-control is linked although the handler was already disposed.
2018-12-02 20:35:29.596 [WARN ] [.core.thing.binding.BaseThingHandler] - Handler BluetoothDeviceHandler of thing bluetooth:ble:E215000053AF tried checking if channel connected-adapter is linked although the handler was already disposed.
2018-12-02 20:35:29.599 [WARN ] [.core.thing.binding.BaseThingHandler] - Handler BluetoothDeviceHandler of thing bluetooth:ble:E215000053AF tried checking if channel authenticated is linked although the handler was already disposed.
They're thrown once when the device is added from the Inbox. Online channel still works fine as before.
Disregard those errors. It is a different thing that needs to be addressed after OH update.
You will need to enable debug levels for BM module and transport modules.
log:set DEBUG org.sputnikdev.bluetooth.manager.impl
log:set DEBUG org.sputnikdev.bluetooth.manager.transport.bluegiga
log:set DEBUG org.sputnikdev.bluetooth.manager.transport.tinyb
On each click, the following is thrown:
2018-12-02 20:54:34.380 [DEBUG] [ooth.manager.impl.DeviceGovernorImpl] - Manufacturer data changed (notification): /XX:XX:XX:XX:XX:XX/E2:15:00:00:53:xx : 1 : 1
2018-12-02 20:54:34.382 [DEBUG] [ager.impl.CombinedDeviceGovernorImpl] - Manufacturer data changed (listener): /XX:XX:XX:XX:XX:XX/E2:15:00:00:53:xx : 1 : true
2018-12-02 20:54:34.613 [DEBUG] [ooth.manager.impl.DeviceGovernorImpl] - Manufacturer data changed (notification): /XX:XX:XX:XX:XX:XX/E2:15:00:00:53:xx: 1 : 1
2018-12-02 20:54:34.615 [DEBUG] [ager.impl.CombinedDeviceGovernorImpl] - Manufacturer data changed (listener): /XX:XX:XX:XX:XX:XX/E2:15:00:00:53:xx: 1 : true
That's good. Bad thing it does not say what changed though :) Things to improve...
It is still unknown why you don't get anything in PaperUI...
Fortunately you can enable TRACE
level for TinyB, it will print you manufacturer data when it changes.
log:set TRACE org.sputnikdev.bluetooth.manager.transport.tinyb
Now the following is thrown per click:
2018-12-03 21:27:21.005 [TRACE] [.manager.transport.tinyb.TinyBDevice] - RSSI tinyb:/xxxxxxxx/E2:15:00:00:53:xx: -65
2018-12-03 21:27:21.024 [TRACE] [.manager.transport.tinyb.TinyBDevice] - Manufacturer data changed: tinyb:/xxxxxxxx/E2:15:00:00:53:xx: {986=[0f, 02, 00, 00, 05, 44, c7, 27, 7c]}
2018-12-03 21:27:21.156 [TRACE] [.manager.transport.tinyb.TinyBDevice] - RSSI tinyb:/xxxxxxxx/E2:15:00:00:53:xx: -65
2018-12-03 21:27:21.159 [TRACE] [.manager.transport.tinyb.TinyBDevice] - Manufacturer data changed: tinyb:/xxxxxxxx/E2:15:00:00:53:xx: {986=[10, 02, 00, 00, 04, 0e, f4, 01, 57]}
2018-12-03 21:27:21.195 [TRACE] [.manager.transport.tinyb.TinyBDevice] - RSSI tinyb:/xxxxxxxx/E2:15:00:00:53:xx: -65
I can also confirm that the data contains the information which button is clicked :)
Good, do you still get nothing in the PaperUI for that channel?
Now something changed...
good but it is different Manufacturer ID :)
0x3da vs 0x986
Right, it is decimal in logs: https://www.google.com/search?q=0x3da%20to%20decinal
Mhm, but it seems to be a big chance that the item is updated... I cannot reproduce any item updates anymore and on all the other switches it also did not work a single time, anyhow the log message posted above (with changing values, of course) is always thrown.
Are you saying that you get log messages, but it does not reflect to PaperUI?
Yes, exactly. I did not observe an update a single time, even though I had plenty of button clicks in the meantime. It still has exactly the same value than in the screenshot, even though I see completely different values in the logs.
right, strange, this might be a bug.
Btw, this debugging/tracing mode is extremely nice! Now I found out why the reliability is so bad: After 20 seconds, it seems like the device gets disposed. After disposal, a wake up needs MUCH more signal strength and a higher number of reapeated telegrams than in the "active" mode.
Log:
--- Initial wakeup: Needs much more signal strength / repeated telegrams than normal ---
2018-12-03 21:55:01.112 [DEBUG] [.manager.transport.tinyb.TinyBDevice] - Enable RSSI notifications: tinyb:/XX:XX:XX:XX:XX:XX/XX:XX:XX:XX:XX:XX
2018-12-03 21:55:01.117 [DEBUG] [.manager.transport.tinyb.TinyBDevice] - Enable connected notifications: tinyb:/XX:XX:XX:XX:XX:XX/XX:XX:XX:XX:XX:XX
2018-12-03 21:55:01.119 [DEBUG] [.manager.transport.tinyb.TinyBDevice] - Enable service resolved notifications: tinyb:/XX:XX:XX:XX:XX:XX/XX:XX:XX:XX:XX:XX
2018-12-03 21:55:01.123 [DEBUG] [.manager.transport.tinyb.TinyBDevice] - Enable blocked notifications: tinyb:/XX:XX:XX:XX:XX:XX/XX:XX:XX:XX:XX:XX
2018-12-03 21:55:01.126 [DEBUG] [.manager.transport.tinyb.TinyBDevice] - Enable manufacturer data notifications: tinyb:/XX:XX:XX:XX:XX:XX/XX:XX:XX:XX:XX:XX
2018-12-03 21:55:01.129 [DEBUG] [.manager.transport.tinyb.TinyBDevice] - Enable service data notifications: tinyb:/XX:XX:XX:XX:XX:XX/XX:XX:XX:XX:XX:XX
2018-12-03 21:55:01.132 [DEBUG] [.manager.transport.tinyb.TinyBDevice] - Checking if device connected: tinyb:/XX:XX:XX:XX:XX:XX/XX:XX:XX:XX:XX:XX
2018-12-03 21:55:01.136 [DEBUG] [.manager.transport.tinyb.TinyBDevice] - Getting blocked: tinyb:/XX:XX:XX:XX:XX:XX/XX:XX:XX:XX:XX:XX
2018-12-03 21:55:01.139 [DEBUG] [.manager.transport.tinyb.TinyBDevice] - Checking if device connected: tinyb:/XX:XX:XX:XX:XX:XX/XX:XX:XX:XX:XX:XX
2018-12-03 21:55:01.143 [DEBUG] [.manager.transport.tinyb.TinyBDevice] - Getting TxPower: tinyb:/XX:XX:XX:XX:XX:XX/XX:XX:XX:XX:XX:XX
2018-12-03 21:55:01.146 [DEBUG] [.manager.transport.tinyb.TinyBDevice] - Getting bluetooth class: tinyb:/XX:XX:XX:XX:XX:XX/XX:XX:XX:XX:XX:XX
2018-12-03 21:55:01.149 [DEBUG] [.manager.transport.tinyb.TinyBDevice] - Getting ble enabled: tinyb:/XX:XX:XX:XX:XX:XX/XX:XX:XX:XX:XX:XX
2018-12-03 21:55:01.151 [DEBUG] [.manager.transport.tinyb.TinyBDevice] - Getting bluetooth class: tinyb:/XX:XX:XX:XX:XX:XX/XX:XX:XX:XX:XX:XX
2018-12-03 21:55:01.153 [DEBUG] [.manager.transport.tinyb.TinyBDevice] - Getting name: tinyb:/XX:XX:XX:XX:XX:XX/XX:XX:XX:XX:XX:XX
2018-12-03 21:55:01.155 [DEBUG] [.manager.transport.tinyb.TinyBDevice] - Setting alias: tinyb:/XX:XX:XX:XX:XX:XX/XX:XX:XX:XX:XX:XX / E2-15-00-00-53-A2
2018-12-03 21:55:01.158 [DEBUG] [.manager.transport.tinyb.TinyBDevice] - Getting RSSI: tinyb:/XX:XX:XX:XX:XX:XX/XX:XX:XX:XX:XX:XX
2018-12-03 21:55:01.162 [DEBUG] [.manager.transport.tinyb.TinyBDevice] - Getting blocked: tinyb:/XX:XX:XX:XX:XX:XX/XX:XX:XX:XX:XX:XX
2018-12-03 21:55:01.165 [DEBUG] [.manager.transport.tinyb.TinyBDevice] - Checking if device connected: tinyb:/XX:XX:XX:XX:XX:XX/XX:XX:XX:XX:XX:XX
2018-12-03 21:55:01.168 [DEBUG] [.manager.transport.tinyb.TinyBDevice] - Is services resolved?: tinyb:/XX:XX:XX:XX:XX:XX/XX:XX:XX:XX:XX:XX
--- Repeatedly some Reachability Checks. Within this time, realiability is VERY good. ---
2018-12-03 21:55:21.146 [DEBUG] [.manager.transport.tinyb.TinyBDevice] - Getting blocked: tinyb:/XX:XX:XX:XX:XX:XX/XX:XX:XX:XX:XX:XX
2018-12-03 21:55:21.149 [DEBUG] [.manager.transport.tinyb.TinyBDevice] - Checking if device connected: tinyb:/XX:XX:XX:XX:XX:XX/XX:XX:XX:XX:XX:XX
--- After 20s, device gets disposed ---
2018-12-03 21:56:17.015 [DEBUG] [.manager.transport.tinyb.TinyBDevice] - Disposing device: XX:XX:XX:XX:XX:XX
Is there a way to deactivate this disposal for added BLE things?
this is what I was talking about, I believe it applies that workaround for stale devices, we will need to add a setting for "thing" to prevent this for this type of devices that goes offline very often but that does not mean that they are offline.
Do you see anything related to "stale" in the logs? There should be something like "checking if object is tale..." (not exactly that).
Ah, yes, I see that.
I have some battery-less wall switches based on Dolphin/EnOcean PTM 215B that I'd like to integrate (technical manual can be found here).
My setup:
Some information that I found out so far:
bluetoothctl
(also not when trying to do radio-based commissioning according the docs, chapter 5.3). Other BLE devices are found without any issues.@vkolotov: I'm a ESH/openHAB developer, but I'm quite new to the BLE protocol. If you could give me some hints and point me to the right direction, I may could come up with a PR.