search

Koenkk / zigbee-herdsman

A Node.js Zigbee library
MIT License
456 stars 277 forks source link

Differentiate PTM215Z and PTM215ZE ZGP switches #398

Closed fredericseiler closed 2 years ago

fredericseiler commented 2 years ago

Hi,

I noticed the EnOcean PTM215ZE module is handled as a PTM215Z (commonly known as "Friends Of Hue" switch), but there are some differences between the two: commandID is not the same for every button event.

I'm trying to make a custom converter and device profile for the PTM215ZE, but since it's tagged with the model ID "GreenPower_2" (because of the same Device Type: 0x02), I have no way to catch the device.

Here are the differences from the commissioning ZCL data (tested with a deconz adapter, 2 PTM215Z and 1 PTM215ZE):

The full ZCL data for one of the PTM215Z ```json { "frame": { "Header": { "frameControl": { "frameType": 1, "manufacturerSpecific": false, "direction": 0, "disableDefaultResponse": false, "reservedBits": 0 }, "transactionSequenceNumber": 205, "manufacturerCode": null, "commandIdentifier": 4 }, "Payload": { "options": 62149, "srcID": 24139860, "frameCounter": 717, "commandID": 224, "payloadSize": 46, "commandFrame": { "deviceID": 2, "options": 197, "extendedOptions": 242, "securityKey": { "type": "Buffer", "data": [ ... ] }, "keyMic": 2743600281, "outgoingCounter": 717 } }, "Command": { "ID": 4, "parameters": [ { "name": "options", "type": 33 }, { "name": "srcID", "type": 35 }, { "name": "frameCounter", "type": 35 }, { "name": "commandID", "type": 32 }, { "name": "payloadSize", "type": 32 }, { "name": "commandFrame", "type": 1009 } ], "name": "commisioningNotification" } }, "address": 24139860, "endpoint": 242, "linkquality": 127, "groupID": 2948, "wasBroadcast": false, "destinationEndpoint": 1 } ```
The full ZCL data for the PTM215ZE ```json { "frame": { "Header": { "frameControl": { "frameType": 1, "manufacturerSpecific": false, "direction": 0, "disableDefaultResponse": false, "reservedBits": 0 }, "transactionSequenceNumber": 144, "manufacturerCode": null, "commandIdentifier": 4 }, "Payload": { "options": 62081, "srcID": 22046452, "frameCounter": 842, "commandID": 224, "payloadSize": 27, "commandFrame": { "deviceID": 2, "options": 129, "extendedOptions": 242, "securityKey": { "type": "Buffer", "data": [ ... ] }, "keyMic": 1919727306, "outgoingCounter": 842 } }, "Command": { "ID": 4, "parameters": [ { "name": "options", "type": 33 }, { "name": "srcID", "type": 35 }, { "name": "frameCounter", "type": 35 }, { "name": "commandID", "type": 32 }, { "name": "payloadSize", "type": 32 }, { "name": "commandFrame", "type": 1009 } ], "name": "commisioningNotification" } }, "address": 22046452, "endpoint": 242, "linkquality": 127, "groupID": 2948, "wasBroadcast": false, "destinationEndpoint": 1 } ```

I think there is something to be done before the onDeviceJoinedGreenPower method of controller.ts create the model ID (GreenPower_${payload.deviceID}), but I don't know where to start (maybe add some infos to GreenPowerDeviceJoinedPayload in greenPower.ts, then the controller can check those infos and build the model ID accordingly?).

fredericseiler commented 2 years ago

In addition, I wonder what's the best naming strategy to differentiate those (and probably others soon): GreenPower_* ?

Koenkk commented 2 years ago

Green power devices have no modelID, they only expose a device type which is shared among a lot of green power devices.

What exactly do you want to change regarding the exposed action.

fredericseiler commented 2 years ago

Sorry, maybe I wasn't clear enough.

The main issue here is, like you said, that controller.ts gives a model ID to each ZGP device based on its device ID, something like "GreenPower_2" for a device ID of 0x02.

Then, the device is interviewed against known profiles and converters with this model ID.

Unfortunately, many ZGP switches shares the same device ID, but not the same button events. The actual converter for PTM215Z (FoH) module, doesn't fit at all PTM215ZE or Legrand ZLGP17 / ZLGP18, which all have the same device ID (0x02) and are wrongly recognized as PTM215Z (because of the "GreenPower_2" model ID).

So, is there a simple way to differentiate those energy harvesting modules, to build a more accurate model ID (like "GreenPower_PTM215Z" or something like that) based on values found in the ZCL payload when commissioning ?

What I have in mind is to add indications on the eventData generated by greenPower.ts, and parse those on controller.ts against a matching map:

{
    "ptm215z": {
        "frame.Payload.options": 62149,
        "frame.Payload.payloadSize": 46,
        "frame.Payload.commandFrame.options": 197
    },
    "ptm215ze": {
        "frame.Payload.options": 62081,
        "frame.Payload.payloadSize": 46,
        "frame.Payload.commandFrame.options": 129
    },
    "zlgp1x": {
        "frame.Payload.options": 62149,
        "frame.Payload.payloadSize": 31,
        "frame.Payload.commandFrame.options": 197
    }
}

But, then you have to deal with specific devices outside of zigbee-herdsman-converters. Another option is to transport those infos or the whole ZCL frame to the converters, and then we can make accurate fingerprints.

Koenkk commented 2 years ago

The lookup of https://github.com/Koenkk/zigbee-herdsman-converters/blob/6c4a12f28a22c0d9ce7d626f93994aaeee2fa913/converters/fromZigbee.js#L3677 is based on the Zigbee spec, so it's not specific for the PTM215Z.

are wrongly recognized as PTM215Z (because of the "GreenPower_2" model ID).

It is recognised as GreenPower_2 right? (not as PTM215Z AFAIK)

It only makes sense to separately identify those devices in case they do not follow the Zigbee spec (since then the actions would mismatch). So my question is, what exactly do you want to change in the published actions?

fredericseiler commented 2 years ago

Ok, I found the specs you're talking about (starting page 170, if anyone wants to take a look).

And you're right, the other modules are recognized as GreenPower_2, because the Device Type of those devices is also 0x02.

Only the PTM215Z follows the specs (correct me if I'm wrong), the GPD Command IDs are slightly different for the other modules, and I suspect that more ZGP switches are coming with their own command IDs.

That's why I think it's important to accurately identify those devices.

Take for example the Legrand ZLGP17 (Device Type 0x02), it's a 1 gang switch, with the same events if you press up or down:

The same goes for the 2 gang Legrand ZLGP18 switch (Device Type 0x02), each gang has to be commissioned separately, treated as a ZLGP17.

For the Legrand ZLGP15, the Device Type is 0xFE (254), so no problem there, you just have to make a custom device profile (only single presses are supported, I can make a PR if you want):

Legrand ZLGP15 custom profile ```js const e = exposes.presets; const { hasAlreadyProcessedMessage } = require('zigbee-herdsman-converters/lib/utils'); const fzLocal = { legrand_zlgp15: { cluster: 'greenPower', type: ['commandNotification', 'commandCommisioningNotification'], convert: (model, msg, publish, options, meta) => { const commandID = msg.data.commandID; if (hasAlreadyProcessedMessage(msg, msg.data.frameCounter, `${msg.device.ieeeAddr}_${commandID}`)) return; if (commandID === 224) return; const lookup = { 0x14: 'button_1', 0x15: 'button_2', 0x16: 'button_3', 0x17: 'button_4', }; return {action: lookup[commandID] || commandID.toString()}; } } } const definition = { zigbeeModel: ['GreenPower_254'], model: 'ZLGP15', vendor: 'Legrand', description: 'Legrand ZLGP15', fromZigbee: [fzLocal.legrand_zlgp15], toZigbee: [], exposes: [e.action(['button_1', 'button_2', 'button_3', 'button_4'])], }; module.exports = definition; ```

And now for the PTM215ZE (Device Type 0x02):

Some are not the same button events as the PTM215Z.

All the possible combinations can be found in the user manual (page 15):

2021-08-01-23-49-09 Google Chrome PTM_215ZE_User_Manual (4) pdf
Koenkk commented 2 years ago

Only the PTM215Z follows the specs (correct me if I'm wrong)

I expect all green power devices that are compatible with Hue to follow the spec.

I doubt if any identification can be done based on the eventData you mentioned in https://github.com/Koenkk/zigbee-herdsman/issues/398#issuecomment-890548203 . This looks something that is very fragile and could easily break other green power devices (e.g. if the Hue tap would use the same payload size). As can be seen there some values already overlap.

An alternative would be to provide the raw value. e.g. publish action: "22" (but I'm not a fan of that either).

Another alternative would be to let the user choose what device it has and based on that use a custom definition (but this will require user intervention).

fredericseiler commented 2 years ago

I expect all green power devices that are compatible with Hue to follow the spec.

The PTM215ZE and Legrand ZLGP1* are not "Friends of Hue" labelled, only ZGP. Other modules from different brands are on the way, with a ZGP certification.

While I agree that playing with the eventData may not be the more robust solution, it still seems more precise than just use the Device Type.

if the Hue tap would use the same payload size

The Hue Tap has an EnOcean PTM215Z inside, so it shares the same payload fingerprint and button events as other ZGP FoH switches based on the PTM215Z (NodOn, Niko, Vimar...).

An alternative would be to provide the raw value. e.g. publish action: "22" (but I'm not a fan of that either).

If the device is already recognized as a GreenPower_On_Off_Switch because of the Device Type, there's nothing we can do afaik.

Another alternative would be to let the user choose what device it has and based on that use a custom definition (but this will require user intervention).

Similarly to the way Signify is doing it in the Hue app. I thought about it, and like you also said, the user intervention stuff is far from ideal: you'll have to select the device before starting the commissioning procedure (for every product you want to add), there are so many things that can go wrong in between. Signify solves this with a whitelist.

The only solution I found so far (which works with all the devices I have been able to check against), is the fingerprint detection. I didn't find any device that shares all the attributes mentioned earlier in the ZCL data, for now. But then again, what if it happens some day?

Koenkk commented 2 years ago

Lets give the event data way a try, I would suggest to:

fredericseiler commented 2 years ago

Great, I'll come back to you with a PR.

fredericseiler commented 2 years ago

So, I just had a look, and have something working.

2021-08-06-19-52-11 Google Chrome Zigbee2MQTT

Before I submit the PRs, what are your preferences for the greenPowerKey structure ?

For now, I've simply done it this way (first time with TypeScript, don't judge me 😅):

const greenPowerKey = `options:${dataPayload.frame.Payload.options},`
    + `payloadSize:${dataPayload.frame.Payload.payloadSize},`
    + `commandFrame.options:${dataPayload.frame.Payload.commandFrame.options},`
    + `commandFrame.extendedOptions:${dataPayload.frame.Payload.commandFrame.extendedOptions}`;

A custom converter using the fingerprint:

const definition = {
    fingerprint: [{
        modelID: 'GreenPower_2',
        greenPowerKey: 'options:62149,payloadSize:31,commandFrame.options:197,commandFrame.extendedOptions:242'
    }],
    model: 'ZLGP17',
    vendor: 'Legrand',
    description: 'Legrand ZLGP17',
    fromZigbee: [fzLocal.legrand_zlgp17],
    toZigbee: [],
    exposes: [e.action(['single_press', 'double_press'])],
};

I can also try something less verbose like 62149.31.197.242 but maybe too cryptic?

Koenkk commented 2 years ago

@fredericseiler looks promising! I actually prefer the short greenPowerKey 62149.31.197.242 or 62149_31_197_242

fredericseiler commented 2 years ago

@Koenkk, after @danieledwardgeorgehitchcock's suggestion, I asked EnOcean's opinion about this... and just got their answer:

Indeed you cannot easily recognize the exact device for the case of Zigbee Green Power.

To a certain extend this is intentional from specification perspective to ensure that devices from different manufacturers behave the same.

That being said, the fact that ZGP source addresses are allocated on per-manufacturer base and change over time gives you a helping hand here.

So if you take the combination of SourceId and DeviceId then you can figure out the various EnOcean derivates as follows:

PTM 215Z (Philips Hue Tap – the one that looks like a hockey puck) Device ID 0x02 Source Address starts with 0x004 Commissioning telegram does not include command list

PTM 215Z FOH (Philips Friends of Hue) Device ID 0x02 Source Address starts with 0x017 Commissioning telegram includes command list

PTM 215ZE (Broad market, this is indeed different from FOH since you can configure it onto 16 channels) Device ID 0x02 Source Address starts with 0x015 Commissioning telegram does not include command list

PTM 216Z (Broad market, this uses Generic Switch) Device ID 0x07 Source Address starts with 0x015 Commissioning telegram includes switch info

I am not in a position to comment on Legrand, but why not check the Source Address there as well?

Many thanks for the contact at EnOcean! (I'm not sure I can disclose his name here)

I'll temporarily reopen the issue, tell me if it's worth a shot to update the greenPowerKey with the beginning of the source address (also remove payload size/options/extended options?), and separate again the Hue Tap from the PTM 215ZE.

Koenkk commented 2 years ago

@fredericseiler that would be great and seems like a better way to identify. Note that I have to revert the current green power changes first since the next release is on 1 september and if we still want to update we shouldn't include this.

Koenkk commented 2 years ago

I'm wondering if the green power key is needed at all now, don't we already have all the required information without the green power key? If yes, that would be great as we can identify the devices without requiring users to repair them. Note that the fingerprint currently doesn't allow to check ieeeAddr (which is the sourceID of the green power device), but we could easily add that with e.g. a regex.

fredericseiler commented 2 years ago

Note that I have to revert the current green power changes first

@Koenkk I totally agree, no need to rush if we can make this better.

I'm wondering if the green power key is needed at all now

Also agree.

I'm looking for a public registry of address space allocation (something like IANA does for IPv4/v6), but nothing on the CSA website. Bummer, that would have been very useful...

fredericseiler commented 2 years ago

@Koenkk What if there are some manufacturer devices that shares the same address space and also the same Device Type? I don't have a practical example right now, but in theory? Isn't the greenPowerKey still useful to have in the toolbelt (even if not used for most cases)?

Koenkk commented 2 years ago

@fredericseiler in that case we can easily add it back, I think it does not make sense to have the geenPowerKey in while not being used.

So the Legrand green power devices also have a unique address allocation + deviceID combo?

fredericseiler commented 2 years ago

@Koenkk for the Legrand devices I own (ZLGP15, ZLGP17/18): source address starts with 0x005, and the Device ID is not the same, so it's OK for me.

One thing I hope is that other Legrand ZGP devices (ZLGP14, ZLGP16...) also have different Device IDs.

fredericseiler commented 2 years ago

@Koenkk do you want fresh PRs for the Source Address + Device Type method (including profiles, docs and images, separating PTM 215ZE from the Hue Tap)?

In the meantime, I got confirmation that there is a registry for address space allocation, but it's on a members-only website, so I (gently) asked the CSA for an access, or a PDF copy. It doesn't hurt to ask, wait and see...

Koenkk commented 2 years ago

Great, sounds like a much better way to identify those devices. I currently reverted all the green power changes since we don't want to include them in the 1 September release.

Would be great if you could make the PRs again! (for zigbee-herdsman-converters and zigbee2mqtt.io)

fredericseiler commented 2 years ago

@Koenkk I also added the missing actions for PTM 215Z and PTM 215ZE (press_energy_bar, etc.)

fredericseiler commented 2 years ago

Many thanks for your hard work @Koenkk (and community)!