TS011F_plug_1 onWithTimedOff behaves as a delayed toggle

schauveau commented 5 months ago

What happened?

I recently bought two plugs _TZ3000_cehuw1lw and _TZ3000_w0qqde0g from two different vendors on AliExpress. They are recognized as TS011F_plug_1 and both have appVersion:192 after an OTA update.

I tested the On with timed off currently described as follow in the documentation

When setting the state to ON, it might be possible to specify an automatic shutoff after a certain 
amount of time. To do this add an additional property on_time to the payload which is the time in 
seconds the state should remain on. Additionnaly an off_wait_time property can be added to the 
payload to specify the cooldown time in seconds when the switch will not answer to other on with 
timed off commands. Support depend on the switch firmware. Some devices might require both 
on_time and off_wait_time to work 
Examples : 
  {"state" : "ON", "on_time": 300}, 
  {"state" : "ON", "on_time": 300, "off_wait_time": 120}.

At first that did not seem to work but I noticed that the plug was turning on and off unexpectedly.

I eventually figured out that the behavior is a lot simpler than the one described in the documentation.

Doing /set { "state": "on" , "on_time": 15} toggles the state after 150 seconds so 10 times the specified amount. This is a real toggle so both ON --> OFF and OFF --> ON are possible. Also, a pending toggle can be cancelled by setting the state.

This implementation of the 'onWithTimedOff` tuya request expects a delay in seconds. The scaling by 10 is caused by the current implementation in https://github.com/Koenkk/zigbee-herdsman-converters/blob/7727e110580a38c913cbd30f8c00ada394b0ad54/src/converters/toZigbee.ts#L45

As a proof of concept, I created a new command toggle_wait_cmd in an external converter derived from the current TS011F_plug_1. For example, /set { "toggle_wait_cmd" : 34 } will toggle the state after 34 second (and so is equivalent to /set { "state" : "ON" , "on_time": 3.4 } )

I believe that this is related to the OTA update (appVersion 192 ?) but I have no other plug to compare.

const fz = require('zigbee-herdsman-converters/converters/fromZigbee');
const tz = require('zigbee-herdsman-converters/converters/toZigbee');
const exposes = require('zigbee-herdsman-converters/lib/exposes');
const tuya = require('zigbee-herdsman-converters/lib/tuya');
const utils = require('zigbee-herdsman-converters/lib/utils');
const e = exposes.presets;
const ea = exposes.access;
const ota = require('zigbee-herdsman-converters/lib/ota');
const globalStore = require('zigbee-herdsman-converters/lib/store');

// fzLocal.TS011F_electrical_measurement is simply a copy from tuya.js
const fzLocal = {
    TS011F_electrical_measurement: {
        ...fz.electrical_measurement,
        convert: async (model, msg, publish, options, meta) => {
            const result = await fz.electrical_measurement.convert(model, msg, publish, options, meta) ?? {};
            const lookup = {power: 'activePower', current: 'rmsCurrent', voltage: 'rmsVoltage'};

            // Wait 5 seconds before reporting a 0 value as this could be an invalid measurement.
            // https://github.com/Koenkk/zigbee2mqtt/issues/16709#issuecomment-1509599046
            if (result) {
                for (const key of ['power', 'current', 'voltage']) {
                    if (key in result) {
                        const value = result[key];
                        clearTimeout(globalStore.getValue(msg.endpoint, key));
                        if (value === 0) {
                            const configuredReporting = msg.endpoint.configuredReportings.find((c) =>
                                c.cluster.name === 'haElectricalMeasurement' && c.attribute.name === lookup[key]);
                            const time = ((configuredReporting ? configuredReporting.minimumReportInterval : 5) * 2) + 1;
                            globalStore.putValue(msg.endpoint, key, setTimeout(() => {
                                const payload = {[key]: value};
                                // Device takes a lot of time to report power 0 in some cases. When current == 0 we can assume power == 0
                                // https://github.com/Koenkk/zigbee2mqtt/discussions/19680#discussioncomment-7868445
                                if (key === 'current') {
                                    payload.power = 0;
                                }
                                publish(payload);
                            }, time * 1000));
                            delete result[key];
                        }
                    }
                }
            }

            // Device takes a lot of time to report power 0 in some cases. When the state is OFF we can assume power == 0
            // https://github.com/Koenkk/zigbee2mqtt/discussions/19680#discussioncomment-7868445
            if (meta.state.state === 'OFF') {
                result.power = 0;
            }

            return result;
        },
    }
};

//
//  /set { 'toggle_wait_cmd' : delay }
//
//    Toggle the state after 'delay' seconds (from 1 to 65534).
//
//    The pending toggle delay can be cancelled by setting the state
//    or by using toggle_wait_cmd again.
//
const toggle_wait_cmd = {
    key: ['toggle_wait_cmd'],
    convertSet: async (entity, key, value, meta) => {
        utils.assertNumber(value);
        const errmsg = 'The toggle delay must be a number of seconds between 1 and 65534' ;
        if (typeof value !== 'number' ) {
            throw Error(errmsg);
        } 

        const delay = Math.round(value)
        if ( delay <= 0 || delay >= 0xFFFE ) {
            throw Error(errmsg);
        }

        const payload = {ctrlbits: 0, ontime: delay, offwaittime: 100};
        await entity.command('genOnOff', 'onWithTimedOff', payload, utils.getOptions(meta.mapped, entity));
        return {}
    }
};

const definition =
    {
        fingerprint: [ {modelID: 'TS011F', manufacturerName: '_TZ3000_cehuw1lw'}, // appVersion:192
                       {modelID: 'TS011F', manufacturerName: '_TZ3000_w0qqde0g'}, // appVersion:192
                     ], 
        model: 'TS011F_plug_1_toggle_wait',
        description: 'Smart plug (with power monitoring) with toggle_wait_cmd',
        vendor: 'TuYa',

        ota: ota.zigbeeOTA,
        toZigbee: [ toggle_wait_cmd  ], 
        extend: [tuya.modernExtend.tuyaOnOff({
            electricalMeasurements: true, electricalMeasurementsFzConverter: fzLocal.TS011F_electrical_measurement,
            powerOutageMemory: true, indicatorMode: true, childLock: true})],
        configure: async (device, coordinatorEndpoint) => {
            await tuya.configureMagicPacket(device, coordinatorEndpoint);
            const endpoint = device.getEndpoint(1);
            await reporting.bind(endpoint, coordinatorEndpoint, ['genOnOff', 'haElectricalMeasurement', 'seMetering']);
            await reporting.rmsVoltage(endpoint, {change: 5});
            await reporting.rmsCurrent(endpoint, {change: 50});

            if (!['_TZ3000_0zfrhq4i', '_TZ3000_okaz9tjs', '_TZ3000_typdpbpg'].includes(device.manufacturerName)) {
                // Gives INVALID_DATA_TYPE error for _TZ3000_0zfrhq4i (as well as a few others in issue 20028)
                // https://github.com/Koenkk/zigbee2mqtt/discussions/19680#discussioncomment-7667035
                await reporting.activePower(endpoint, {change: 10});
            }
            await reporting.currentSummDelivered(endpoint);
            const acCurrentDivisor = device.manufacturerName === '_TZ3000_typdpbpg' ? 2000 : 1000;
            endpoint.saveClusterAttributeKeyValue('haElectricalMeasurement', {acCurrentDivisor, acCurrentMultiplier: 1});
            endpoint.saveClusterAttributeKeyValue('seMetering', {divisor: 100, multiplier: 1});
            utils.attachOutputCluster(device, 'genOta');
            device.save();
        },
    };

module.exports = definition;

What did you expect to happen?

No response

How to reproduce it (minimal and precise)

No response

Zigbee2MQTT version

1.36.1 commit: ffc2ff1d

Adapter firmware version

-

Adapter

-

Setup

-

Debug log

No response

schauveau commented 5 months ago

I just tried on a wall switch https://www.zigbee2mqtt.io/devices/WHD02.html and it behaves the same : state: ON triggers a toggle after 10 times the specified on_time value.

{
  "modelId": "TS0001",
  "manufName": "_TZ3000_kycczpw8",
  "appVersion": 70,
  "stackVersion": 0,
  "hwVersion": 1
}

schauveau commented 5 months ago

I think that I now understand what is going on.

A lot of manufacturers appear to be using the OnWithTimedOff command as it is currently implemented in Z2M so a ON state that turns to OFF after a delay. That seems to be mostly a Lighting feature to turn off the lights after a delay. This is also used in Matter or z-wave. The arguments are indeed specified as 1/10 seconds.

However, Tuya is (ab)using the same command as a timer/countdown. This is documented for all Tuya standards that support the On/Off cluster in https://developer.tuya.com/en/docs/connect-subdevices-to-gateways/Zigbee_2?id=Kcww7qppbe87m

The links in the page body are all in Chinese but can be changed to English using the 'World' icon at the top of the page (or replace /cn/ by /en/ in the URL).

For example, the Standard for Smart Metering Socket documents the Timer feature as follow:

================================

Direction	Cluster ID	Command/Attribute ID
Client to server	0x0006: On/Off	0x42: On With Timed Off	Payload
Server to client	0x0006: On/Off	0x4001: OnTime 0x4002: OffWaitTime	0 to 43200

Format of payload: On/off Control: 0x00. On time and Off Wait Time must be set to the same value. Example: To send a command of a local countdown for 60s, the payload contains the following settings:

On/off Control: 0x00
On Time: 0x003c
Off Wait Time: 0x003c

================================

A few remarks:

0x003c = 60 so the time is expressed in seconds and not in 1/10s
OffWaitTime should be equal to OnTime. I did not follow that rule in my external converter but it seems to work fine at least for now.
The server is supposed to send OnTime and OffWaitTime back to the client probably to inform it how much time is remaining. Z2M should probably publish those attributes if they are effectively available.
why 0 to 43200 ? because 43200s = 12 * 3600s = 12 hours. The full 16bits range 0-0xFFFF may not be allowed. Some tests are needed.
It is never explicitly stated that the countdown action is a 'toggle' but that makes sense as long as we ignore the names OnTime and OffWaitTime that were obviously borrowed from the non-tuya standard behavior.

The Standard for Smart Switch is slightly different but the idea remains the same:

Countdown
    - The countdown function is implemented with the 0x42 command of 
    the Zigbee on/off cluster.

    - After the countdown command is received, the device starts to count down. 
    The count value is decreased by 1 every second. The countdown data is 
    written to the OnTime and OffWaitTime attributes.

    - After the count value becomes 0, the bits are reversed and the switch status
    and values of OnTime and Off Wait Time are reported to the gateway.

schauveau commented 5 months ago

I started a discussion https://github.com/Koenkk/zigbee2mqtt/discussions/22191 about how to improve support for the Tuya on-off cluster.

hutchx86 commented 4 months ago

Hi @schauveau how were you able to update these? I've got a bunch of the _TZ3000_cehuw1lw plugs and I'm stuck on appversion 112 and z2m just won't see an update for them. They report amperage incorrectly, and won't report kWh at all.

Thanks in advance!

schauveau commented 4 months ago

I only have one TZ3000_cehuw1lw that I bought about one month ago (on Amazon, not AliExpress) with another tuya plug and and energy meter. Those were my first zigbee devices and if I remember well, I was able to apply an OTA update to two of them (probably the two plugs since the energy meter does not support OTA).

I do not know the versions numbers before the OTA update but what I have now is

            "modelId": "TS011F",
            "manufacturerName": "_TZ3000_cehuw1lw",
            "zclVersion": 3,
            "appVersion": 192,
            "stackVersion": 0,
            "hwVersion": 1,

Did you try to reset the device? Also, the hw version may be different which could explain why you do not have any update.

hutchx86 commented 4 months ago

I think you're right, it does indeed seem to be a different hardware revision!

"modelId":"TS011F",
"manufacturerName":"_TZ3000_cehuw1lw",
"zclVersion":3,
"appVersion":112,
"stackVersion":2,
"hwVersion":0

Was the update available in Z2M for you or did you use a tuya hub? Ordered a Tuya hub just in case, and currently reading into supporting devices to see if I can fix it myself. Although I don't think that would really be possible considering it's the same model number as working devices, but a different firmware version.

Tried resetting the plugs, didn't seem to have an effect unfortunately.

schauveau commented 4 months ago

The update was in Z2M. I do not really know how OTA updates work for Tuya. Nothing is listed in https://github.com/Koenkk/zigbee-herdsman-converters/blob/master/src/lib/ota/OTA_URLs.md so I assume that this is using a Tuya web api.

I assume that you already tried to tune the reporting setting. For the kWh of my TZ3000_cehuw1lw, the relevant reporting is currently seMettering / currentSumDelivered / 5 / 60 / empty (so probably 0) which is probably the default.

Anyways, having both bad values for current (and probably power I suppose) and no kWh reporting probably means that the device is broken. I just have to hope than mine with a newer hwVersion will be more resilient. Humm... cheap Chinese devices ... When did you buy it?

schauveau commented 4 months ago

As a side note, I have two new similar plugs _TZ3000_fukaa7nc that are currently at appVersion 78. I pressed the 'check for new update" button yesterday and now they show a possible OTA update. One has hwVersion 1 while the other does not show any hwVersion (?!!!). So the hwVersion my not be required after all. I am currently starting the update.

If it goes to appVersion 192 then that could mean that all TS011F plugs share the same firmware.

If you are not afraid to brick your device then you could try to change its identifier in database.db to another value such as _TZ3000_fukaa7nc and see if it works.

schauveau commented 4 months ago

After a bit of digging, I found the source for the firmware : https://github.com/Koenkk/zigbee-OTA/blob/2fddaba1bedbe007e777bd775451cc4343690e7e/index.json#L1636

As you can see, there is a single version 192 for all TS011F. The important values are "manufacturerCode": 4417, "imageType": 54179, "modelId": "TS011F"

They have to match the values expected by the device:

- Set the log level to debug
- in the OTA web interface, press the button check for new updates
- filter or grep the log for zhc:ota
- Do not forget to restore the log level

Here is what I get for my device:

[2024-04-29 08:43:57] debug:    zhc:ota:common: Checking if an update is available for '0xa4c138137f39a0ed' (TS011F)
[2024-04-29 08:43:57] debug:    zhc:ota:common: Using endpoint '1'
[2024-04-29 08:43:57] debug:    zhc:ota:common: Got request '{"fieldControl":0,"manufacturerCode":4417,"imageType":54179,"fileVersion":192}'
[2024-04-29 08:43:57] debug:    zhc:ota:common: Is new image available for '0xa4c138137f39a0ed' (TS011F), current '{"fieldControl":0,"manufacturerCode":4417,"imageType":54179,"fileVersion":192}'
[2024-04-29 08:43:57] debug:    zhc:ota: Getting image metadata for 'TS011F'
[2024-04-29 08:43:58] debug:    zhc:ota: Downloaded main index
[2024-04-29 08:43:58] debug:    zhc:ota:common: Is new image available for '0xa4c138137f39a0ed' (TS011F), latest meta '{"fileVersion":192,"fileSize":307682,"url":"https://images.tuyaeu.com/smart/firmware/upgrade/20220907/1662545193-oem_zg_tl8258_plug_OTA_3.0.0.bin","sha512":"01939ca4fc790432d2c233e19b2440c1e0248d2ce85c9299e0b88928cb2341de675350ac7b78187a25f06a2768f93db0a17c4ba950b60c82c072e0c0833cfcfb"}'
[2024-04-29 08:43:58] debug:    zhc:ota:common: Update available for '0xa4c138137f39a0ed' (TS011F): NO

The 1st line tells me that the modelId is TS011F. The 3rd line gives and manufacturerCode and imageType required by the device and fileVersion is the current firmware.

hutchx86 commented 4 months ago

Thank you very much for your replies! I'll give all of your suggestions a go as soon as I've got some time after assignments.

hutchx86 commented 4 months ago

After a bit of digging, I found the source for the firmware : https://github.com/Koenkk/zigbee-OTA/blob/2fddaba1bedbe007e777bd775451cc4343690e7e/index.json#L1636

As you can see, there is a single version 192 for all TS011F. The important values are "manufacturerCode": 4417, "imageType": 54179, "modelId": "TS011F"

They have to match the values expected by the device:

Set the log level to debug

in the OTA web interface, press the button check for new updates

filter or grep the log for zhc:ota

Do not forget to restore the log level

Here is what I get for my device:
[2024-04-29 08:43:57] debug:  zhc:ota:common: Checking if an update is available for '0xa4c138137f39a0ed' (TS011F)
[2024-04-29 08:43:57] debug:  zhc:ota:common: Using endpoint '1'
[2024-04-29 08:43:57] debug:  zhc:ota:common: Got request '{"fieldControl":0,"manufacturerCode":4417,"imageType":54179,"fileVersion":192}'
[2024-04-29 08:43:57] debug:  zhc:ota:common: Is new image available for '0xa4c138137f39a0ed' (TS011F), current '{"fieldControl":0,"manufacturerCode":4417,"imageType":54179,"fileVersion":192}'
[2024-04-29 08:43:57] debug:  zhc:ota: Getting image metadata for 'TS011F'
[2024-04-29 08:43:58] debug:  zhc:ota: Downloaded main index
[2024-04-29 08:43:58] debug:  zhc:ota:common: Is new image available for '0xa4c138137f39a0ed' (TS011F), latest meta '{"fileVersion":192,"fileSize":307682,"url":"https://images.tuyaeu.com/smart/firmware/upgrade/20220907/1662545193-oem_zg_tl8258_plug_OTA_3.0.0.bin","sha512":"01939ca4fc790432d2c233e19b2440c1e0248d2ce85c9299e0b88928cb2341de675350ac7b78187a25f06a2768f93db0a17c4ba950b60c82c072e0c0833cfcfb"}'
[2024-04-29 08:43:58] debug:  zhc:ota:common: Update available for '0xa4c138137f39a0ed' (TS011F): NO
The 1st line tells me that the modelId is TS011F. The 3rd line gives and manufacturerCode and imageType required by the device and fileVersion is the current firmware.

This is what I get for an update check:

Info 2024-05-01 22:17:32Checking if update available for '0xa4c13806a55864d9'
Warning 2024-05-01 22:17:32Images currently unavailable for device 'TS011F', hardwareVersion '0', manufacturerName _TZ3000_cehuw1lw, {"fieldControl":0,"manufacturerCode":4107,"imageType":517,"fileVersion":268514823}'
Info 2024-05-01 22:17:32No update available for '0xa4c13806a55864d9'

I bought the plugs about 2 weeks ago. I've also tried changing the identifier in the database.db but it keeps changing back on every boot if z2m and the update checks for the old identifier. The confusing part is that they do seem to report kWh, but reported as a list with "0,measurement" where measurement is the amount (we presume). But it doesn't show up in the UI, only in the logs. UI just keeps saying NULL or N/A. I've tried the seMettering setting but it doesn't seem to ever populate

hutchx86 commented 4 months ago

Managed to use the custom ota index option to force it to see the update, but i unfortunately get "update failed, manufacturer code mismatch", guess that would have been too easy :D

schauveau commented 4 months ago

So your device needs a firmware with imageType 517 (0x0205) instead of 54179 (0xD3A3).

The imageType is also stored at offset 12 in the firmware file in little-endian format (a3 d3) .

# hexdump -C 1662545193-oem_zg_tl8258_plug_OTA_3.0.0.bin.1 | head  -n 2
00000000  1e f1 ee 0b 00 01 38 00  00 00 41 11 a3 d3 c0 00  |......8...A.....|
00000010  00 00 02 00 54 65 6c 69  6e 6b 20 4f 54 41 20 53  |....Telink OTA S|

You could try to change the a3 d3 to 05 02 but this is probably a bad idea. That could brick the device.

The fact that the kwh is reported as "0,measurement" is probably good news. That means that writing an external converter should not be that difficult.

Also the TS011F plug already has 3 converters TS011F_plug_1, TS011F_plug_2 and TS011F_plug_3 so it could well be that one of them works for your device.

schauveau commented 4 months ago

TS011F_plug_3 is the most likely candidate since TS011F_plug_2 is without power monitoring.

Try adding 112 to the list of known applicationVersion https://github.com/Koenkk/zigbee-herdsman-converters/blob/37bac5b3536c127866c9544d0f77423f9129b066/src/devices/tuya.ts#L4270

You may also want to add it here if the kwh are not properly reported: https://github.com/Koenkk/zigbee-herdsman-converters/blob/37bac5b3536c127866c9544d0f77423f9129b066/src/devices/tuya.ts#L4294

PS: For testing, the file you need to edit is .../zigbee2mqtt/node_modules/zigbee-herdsman-converters/devices/tuya.js

hutchx86 commented 4 months ago

Just tried my luck by changing both the image type and manufacturer ID in the firmware.bin, but I get "device did not request any image blocks" so I presume either OTA isn't properly implemented on the version I have or it's doing a firmware check. I'll try my luck with a custom converter! I've been playing around with them but I haven't really figured it out yet

hutchx86 commented 4 months ago

TS011F_plug_3 is the most likely candidate since TS011F_plug_2 is without power monitoring.

Try adding 112 to the list of known applicationVersion https://github.com/Koenkk/zigbee-herdsman-converters/blob/37bac5b3536c127866c9544d0f77423f9129b066/src/devices/tuya.ts#L4270

You may also want to add it here if the kwh are not properly reported: https://github.com/Koenkk/zigbee-herdsman-converters/blob/37bac5b3536c127866c9544d0f77423f9129b066/src/devices/tuya.ts#L4294

PS: For testing, the file you need to edit is .../zigbee2mqtt/node_modules/zigbee-herdsman-converters/devices/tuya.js

This worked! It is now reporting amperage properly, and I'm currently running a load on it to see if it is reporting total kWh as well (via polling).

Seriously, thank you SO much for your help! I will see if I can merge this in a PR for this specific model (if it's appropriate)

Edit: IT WORKS! It now correctly reports kWh usage! I'll read through the docs a bit more after assignments and then I'll see if I can make/adapt the converter, whatever needs to be done! Thank you so much, this was an extreme motivation for me to start contributing! :D

schauveau commented 4 months ago

Make sure that your PR does NOT add _TZ3000_cehuw1lw in the condition for TS011F_plug_3 since that would break my device.

hutchx86 commented 4 months ago

Yep, I was thinking of adding only the firmware version, basically exactly what you've told me to do. These are the UK plugs at my partners house, my EU plugs in my place are the same firmware version, but a slightly different model, but those report fine. Don't want to break mine either :D

hutchx86 commented 4 months ago

I have finally found a difference between my UK and EU plugs. They are both on the same appversion, but have different swbuildIDs. The UK ones are

"swBuildId":"1.0.5\u0000"

and the EU plugs are on:

"swBuildId":"0122052017"

Given that, I think it would be more prudent to determine the fingerprint for the converter by swBuildID instead of appversion, given that they have the same app version, but different Builds, and behave differently

schauveau commented 4 months ago

I do not know enough about fingerprints to help you. This is probably something you should discuss in your PR.

Koenkk / zigbee2mqtt