search

zigpy / zha-device-handlers

ZHA device handlers bridge the functionality gap created when manufacturers deviate from the ZCL specification, handling deviations and exceptions by parsing custom messages to and from Zigbee devices.
Apache License 2.0
720 stars 669 forks source link

[Device Support Request] LIDL Plug USB HG06338-FR _TZ3000_vzopcetz/TS011F #1428

Closed mdeweerd closed 1 year ago

mdeweerd commented 2 years ago

This plug already has a quirk but version 70 of this plug shows only one endpoint when it's new and the "Tuya Magic Spell" sequence has to be called.

I did not see any other issue for this and I do not have such a plug - I have the previous version that works out of the box but I was trying to help out on a forum.

I am not sure that this works, the "magic spell" would need to be sent as soon as possible and I did not find any other quirk handling something like that.

I created PR #1427 so that anyone wanting to fix that has something to start from (or just ignore it).

MattWestb commented 2 years ago

I also only have (2) "old" power strips so cant testing the updated quirk only that is not making bad things with the old one If you is having one user with one new its great for testing also if its changing the signature after getting the spell casted on it and then need to have 2 device classes in the quirk for working ok if its rejoining and ZHA is reading its endpoints and saving it in the database.

It shall being one closed issue with this but i must finding it and trying getting some user testing the code how its working.

Im not one fan of having different quirks for tuya devices and i think its better having all on the tuya and not in MOES and CO that is making its not easy maintaining in the log run but its somthing for the maintainers but i like moving all LIDL to tuya folder.

PS: I was looking inside the power strip and i cant getting the debug connection on the Zigbee module without braking it (its very tight to one relay on the EU version but free on the UK one) so i dont like upgrading my with tuya ZBGW until knowing i can getting them working OK then i using both in production with automatons and like have the possibility reading and writing the firmware if not working OK that is not possible.

MattWestb commented 2 years ago

The issue was in HA and its closed and locked https://github.com/home-assistant/core/issues/59894 :-(

Hope some of the user is reading this issue.

mdeweerd commented 2 years ago

Sure, common code is better, but it should also be easily identifiable - so Parent/Child classes is also a good method that can be combined.

The magic spell could be in the "tuya" library.

For now, my goal was to share this work, allow testing and when it works, things can be "refactored".

MattWestb commented 2 years ago

I agree with you but LIDL TRV is being implanted in tuya TRV and its using the DP / MCU code and making 2 implanting of the larges quirk and maintaining it is one bad scenario, Enough of "political" things aand back to the hard software things !

I was putting you code in My TS004F dimmer switch and restarting HA after deleting it and removing the battery (so its loosing the 3 "extra" endpoints) and rejoining it and its was doing it nice and after switching the working mode its using all 4 endpoints !!!!!!!!!!!!!!!!!!!

So I its working on all routing device every time ZHA is loading the quirk = OK.

The problem is coming with sleeping devices how we like doing it then trying sending to the device then HA is stating is only bad and not needed then the device is already "fixed".

In the Z2M issue one deCONZ user was sniffing one LIDL power stripe and the tuya ZBGW doing the magic. For remotes tuya looks not doing it (have looking on 2) and instead instructing the user pressing one button and sending one leave with rejoin and then the device is rejoined they is sending the spell and the device is changing its hardware setting until the battery is out.

Also is the problem if not sending it then the device is joined its wot working without using the leave trick that working but its little more work and logic getting it working.

Is it possible getting the quirk looking if the device is initiated and if yes skipping the code block and of no running it ?

I need doing more testing how the system and the device is working but for 110% you implementation is working !!

GREAT THANKS !!!

mdeweerd commented 2 years ago

😃 I didn't think it was so close.

The way I figured this would work is that initialy this device matches the quirk with a single endpoint, and on a subsequent launch it matches the quirk with three endpoints. So the magic spell is done only in the single endpoint quirk. No need for a specific test.

For sleepy devices, the magic spell would just have to repeat itself enough times. The current read_attributes call does not have tries yet (should be fixed in next release), so it has to be extended with retryable (as done in zha-toolkit/utils/cluster_read_attributes for instance).

For other devices, I suppose that there is some difference between the initial device signature and the altered one. so as long as the signatures are different, there is no need for extra testing to do this initialisation only once.

Further, as you indicate that the device can be "reset" by removing the battery, it should be done each time the device is reset. So that is a good match for the proposed method.

mdeweerd commented 2 years ago

But I remember now why I concluded this might not work: ZHA/zigpy does not rediscover the device, so the database still has the original configuration. So for that maybe trigger the rejoin or a zigpy rediscovery.

I think it would be nice that zigpy would have a force parameter to discover the device as if it were new.

MattWestb commented 2 years ago

In sniffs for the LIDL power strips its start sending and reviving commands on the 3 new endpoints after tuya ZBGW have casting its magic and also the TS004F. I was thinking it shall have 4 endpoints after that but ZHA is not rereading the endpoints after first INIT so i dont knowing if its true. Can tookit reading active endpoints ? Then its possible having 2 device classes one with 1 EP (pus GPP) and one with 4 EP (plus GPP) and the one EP is casting the spell and reading the active endpoints and the ZHA is finding the "new" device with 4 endpoints and using the new device class.

But i think its not working so well the the TS004F is only having one EP after deleted and retested then its joining as on new device and is having being casted and the battery is not being out :-((

I dont knowing only brainstorming what i think can working.

dmulcahey commented 2 years ago

But I remember now why I concluded this might not work: ZHA/zigpy does not rediscover the device, so the database still has the original configuration. So for that maybe trigger the rejoin or a zigpy rediscovery.

I think it would be nice that zigpy would have a force parameter to discover the device as if it were new.

This will get built into the websocket server most likely

MattWestb commented 2 years ago

I think we need help of Puddly and Alexei and perhaps David for getting it working OK in ZHA but also in other system that is using the Zigpy libs and not braking our France coleuses systems that is using the same code in there system.

mdeweerd commented 2 years ago

I've tried to do some stuff in misc_reinitialize but it doesn't work out that well - it has to be updated.

I read the code that was related to setting up the quirks in zigpy the other other day.

I think that device.initialize should have a "force" parameter so that it skips the checks that indicate if it's already configured for this and that. So then, after the magic spell, the initialize(force=True) could be called.

Another way would be to write a "quirk manufacturer attribute" that would be kept in the database and that could be read back using cache. That could also work to trigger the magick spell. Yet another (complementary) method is to write a location to the device (if available) or to add a special scene/group that would not be used otherwise. I suggest "complementary" because initially the virtual attribute could indicate that the device is already configured, but due to battery removal it might have lost it. So then the absence of a group or scene could indicate that it isn't. Of course the group should not be the same for controls and for receivers. Lamps could be 0xFFF0 and controllers (remotes, switches) could be 0xFFF1 (example).

MattWestb commented 2 years ago

The group shall being very resister in the device but the problem if removing the battery of one working TS004F that is fully configuration its having all information (network, binding and groups ETC) but is stop using the 3 "new" endpoints and must being kicked and casted then rejoining for start using them agen.

One 2 way approach: only cating on new joining devices and implanting one service that is kicking and casting on rejoin or then re config the device as David have saying.

PS: not all devices need being in INIT mode = kicked and casted but i can asking witch its needing it later then we have getting it working.

MattWestb commented 2 years ago

Comment to the last comment in the PR:

It was the leave with rejoining issue our puddly was writing but i was not getting reading and writing attribute working so i have not testing it if its working OK. https://github.com/zigpy/zigpy/issues/831#issuecomment-1022768410

    async def _leave_and_rejoin(self):
        # Tell the device to leave and re-join
        await self.zdo.Mgmt_Leave_req(self.ieee, 1)

        # Delete the device from the database
        self.application.listener_event("device_removed", self)

        # Delete the device from zigpy
        self.application.devices.pop(self.ieee, None)

The LIDL power stripe is more then likely changing endpoints configuration and also the TS004F and i shall testing it tomorrow.

mdeweerd commented 2 years ago

@MattWestb I updated the code but did not and added the removal from the db and from zigpy.

MattWestb commented 2 years ago

I trying it little later i have just rigging the soldering equipment for "converting" 2 new IKEA light with MG21 modules to "Markus EUSP / NCP" = de-soldering, think cables and dumping firmware and then working on working EZSP and later one NCP for it so i can getting some more open thread routers in my test OTBR setup in HA.

I reporting back then have testing it !!

mdeweerd commented 2 years ago

FYI: a user on french forum hacs confirmed that the quirk (first version I suppose) worked for the LIDL plug.

MattWestb commented 2 years ago

Thanks for info !! So the 3rd to do = moving one of the "old" power strips to LIDL ZBGW and trying updating it and back to ZHA for testing if its working OK.

MattWestb commented 2 years ago

I have testing the updated code and having problem getting it working with TS004F DS. If taking out battery its not getting the 3 new EPs and its having large problem being joined and gritting timeout reading node description. I have testing around 10 times then getting it pared (around 30 resetting the device for stating joining) and not have getting getting new EPs. With original its looks 100% getting new endpoints then paring new after taking the battery out. For being sure i deleting the device changing quirk to one without magic and taking battery out and restarting HA and then paring it for verifying no extra EP, and redoing the same with magic quirk and its away working first try.

I have moving more device to my test system and running full debug so its not so easy see what is working OK or not then the paring taking very long time and spamming the log.

My feeling is that the LIDL power strip is not needing more then one "simple spell casting" and its looks that the TS004F is working very well with it also and it shall being the most tricky device getting working with timing and its not getting any more endpoints that ZHA have found = no positive effects only bad things.

For my is the feeling that the leave with rejoin is not needed for TS004F but need testing more if its working good enough for user in production system (that is 90% working then doing new joining the device) and also is the system is getting problem then restarting HA and the quirk is loaded and trying casting the spell so it not locking the system with retry for minutes.

@mdeweerd have your user seen any problem then restarting the system then using the first version of the spell or is it working OK ? If working OK with simple cast i saying go for it if not getting problem after restarting HA and also then the device is coming back OK after power was removed.

The problem i can see with the TS004F is if the battery is going out and the user is putting in one new battery and need getting the spell being casted without deleting the device and loosing automatons and other linked functions.

One way is making one cluster command / attribute that the quirk is catching then the user is "sending" it and running one code with leave with rejoin and casting the spell or only the spell then the device is getting one new battery. Is it possible making in the quirk ?

More thinking ?

mdeweerd commented 2 years ago

I do not understand what you want to convey exactly for every part of your message.

In summary what I understand is:

On my end, two users now reported that the quirk worked and I am supposing that they used the "first version". As far as I read between the lines, one of them even restarted and did not report issues.

My thoughts:

Next steps:

MattWestb commented 2 years ago

Sorry for making on mess of most things !!

I was doing wrong then copy your "advanced" version and it was not working but i have getting the implement working OK = like the first one.

In summary what I understand is:

My thoughts:

Next steps:

My thinking:

I still think normally we dont need casting the spell but can being needed in some cases we is knowing (battery out + resettle sleepers or re power the stripes). Perhaps listening on device accouterments (device is back after re power / firmware reboot or have loosing the network and coming back) and if its only soft reboot it no danger casting the spell and if its was one hardware (battery was out and perhaps lost the network) is the device in INIT mode and can getting the spell fixing the EPs (the device is in Zigbee.db as insinuated).

I shall making more test for understanding then the remote is loosing the "new" endpoints so not new NCP and RCP firmware for my "IKEA Markus modules".

I dont have time updating one of my 2 Power stripes and its only making one mess is i trying doing more thing at the same time and testing the re power it.

mdeweerd commented 2 years ago

Ok. After reading all that, I can think of these questions that seem important:

If recasting the spell is not harmfull, then we can leave it more or less like this - if not rediscovering the device is more important.

As the function "outside" the class is working, it can also be called from a TuyaBasicCluster and we can try to call as @dmulcahey suggested (in such a way that the spell is also called when reconfiguring from the UI) - that avoids that users have to setup zha-toolkit.

I propose to continue testing in a monolithic file and move the spell to tuya's init later when prepping for release. (That makes it easier for users to test - they can just drop the quirk in their own custom directory).

MattWestb commented 2 years ago

110% agree with all !!

I need doing little more testing but can you giving one code part for doing one log record then the spell is being casted ?

My log is very spammed of bellows debug output then have the joining window open and nearly impossible see then the code is running.

I think (but need little more testing) the spell is OK as it is and can later being moved to tuya INIT and perhaps being used in ZHA reconfigure or if needed one version with leave with rejoin and one casting of the spell but its coming later.

MattWestb commented 2 years ago

I have not seen the casting the restart of HA and not then rejoining but im not 110% sure.

Some debug logging is helpful for see then is being casted and if its doing bad things (locking loading ZHA or making rejoining not working).

mdeweerd commented 2 years ago

OK, I'll add a log message.

I just had another idea, but it requires a change in zigpy. The magic spell could be done in zigpy when requesting the model/identifier. They are part of the request. If the device needs the magic spell, then the endpoints should be available right after that.

That would impact all devices of course, but in principle it's harmless.

MattWestb commented 2 years ago

Logging is GREAT !! so pleas adding it

If can catching the first red model/identifier we can casting very early = good so the device is not exiting INIT mode (I think its doing that in some 10th seconds or then start config / bindings of the device).

Sniffs have showing tuya ZBGW doing casting after network key is being handled, reading model/identifier and before any more "normal" config and then the device is start using the "new" EPs (i think it was before updating the TC Link Key but im not sure).

mdeweerd commented 2 years ago

I added logging, use attribute names instead of numbers, removed the original code, prepared a TuyaBasicCluster which can be tested after uncommenting a line and renaming the init method so that it's not used.

Instructions are in the code.

I did not execute this code, linting is ok.

https://github.com/zigpy/zha-device-handlers/blob/659ae7f1c809a3b6907cf782d4b35e90aae3b086/zhaquirks/lidl/ts011f_plug.py

MattWestb commented 2 years ago

Was adding all changes to my test quirk and fixing so indents problem that i was making and the quirk is loading OK. But i cant see any longing only one interesting thing after the quirk is loaded: [zigpy.device] [0x2358] Extending timeout for 0x95 request. But i cant see what was the request. The log file from start joining till its in and have changing mode and sending from new EPs: TS004fXX.txt

After restarting HA i is getting this then the quirk is loading:

2022-03-13 16:22:20 DEBUG (MainThread) [zigpy.quirks.registry] Considering <class 'ts004f5B.TuyaSmartRemote004FDMS'>
2022-03-13 16:22:20 DEBUG (MainThread) [zigpy.quirks.registry] Found custom device replacement for 60:a4:23:ff:fe:fd:dc:14: <class 'ts004f5B.TuyaSmartRemote004FDMS'>
2022-03-13 16:22:20 DEBUG (MainThread) [zigpy.device] [0x2358] Extending timeout for 0x01 request

If the quirk is trying casting the spell its OK its timing out then we dont need it then the device is sleeping and its not in INIT mode. The WIP quirk i using: ts004f5B.py.txt

If you can finding somthing wrong with the logging pleas say how i can fixing it.

My log setting for quirks looks like this:

    zhaquirks: debug
    custom_zha_quirks: debug
    custom_components.zha_toolkit: debug

But from how the device is reacting and the system its looks very OK then the spell is working then the device is retested and is doing on new joining and dont looks having problem then bring rejoined or HA is restarting.

I have getting the little more experience then the device is loosing its casted magic. Resetting and (re)joining = OK Battery out 20 min then being in the network = OK Resetting the device + taking out the battery 10 seconds + resetting for (re)joining = losing the magic.

So if the user is adding the device first time its getting the magic and its OK. If battery is running out and its still in the network its OK then new battery is inserted (if tuya have fixing some bugs in the Silabs SDK that is deleting the NVM like IKEA was having problem with). If the user is deleting and and joining the device it shall being OK as long the battery is OK also if only rejoining the device then its being lost in the network.

The only way one normal user can loosing the magic is if the battery is nearly empty and resetting the device and the battery is out before its being joined in one network. I think getting one look on the code tuya have doing for getting this working must being interesting then they is using 3 different components (normally only NVM is being used that is deleted then resetting the device).

With this knowledge i feel OK using the spell as it is for tuya sleeping devices that we is knowing for the moment. Its no need adding it to re-configuring device in ZHA for the TS004F but can being good for other devices (like the Power stripes) in the future and only reading some attributes is not making any harm to the device and the system.

If you can getting the logging working OK i can looking more if the system is trying casting the spell on HA restart and rejoining the device but its not very importing (but interesting).

Great work done !!

mdeweerd commented 2 years ago

I have the following in my logger setup for another quirk that I placed in my custom quirks:

  rgbcct: debug

So it matches the name of the file.

You can change '.debug' to '.error' so that you get the messages. They're not errors of course. You can then also learn about the logging name space.

It is logical that a device reset ("hw reset") looses the cast and good that it does not in the other cases.

For your remote, I'ld set 'tries' to 100 - you then have ample time to wake up the remote multiple times using button presses.

MattWestb commented 2 years ago

So the debug/log "name" shall being : ts004f5B for my quirk. I testing it !!

I having retry = 3 Then pairing its not needed as long the coordinator is sending commands to it and then its finished its still pulling some more seconds and the magic is being sent very early so it shall being OK if the joining is being OK. The debug logging was not working so i dont know if it was trying 3 times but im looking for it.

MattWestb commented 2 years ago

Then joining one resettted and battery removed and i is getting this after the quirk is loaded:

2022-03-13 19:28:57 DEBUG (MainThread) [zigpy.quirks.registry] Checking quirks for _TZ3000_xabckq1v TS004F (60:a4:23:ff:fe:fd:dc:14)
2022-03-13 19:28:57 DEBUG (MainThread) [zigpy.quirks.registry] Considering <class 'ts004f5B.TuyaSmartRemote004FDMS'>
2022-03-13 19:28:57 DEBUG (MainThread) [zigpy.quirks.registry] Found custom device replacement for 60:a4:23:ff:fe:fd:dc:14: <class 'ts004f5B.TuyaSmartRemote004FDMS'>
2022-03-13 19:28:57 DEBUG (MainThread) [ts004f5B] Cast Tuya Magic Spell on 60:a4:23:ff:fe:fd:dc:14
2022-03-13 19:28:57 DEBUG (MainThread) [zigpy.device] [0x0662] Extending timeout for 0x57 request
2022-03-13 19:28:57 DEBUG (MainThread) [homeassistant.components.zha.core.gateway] device - 0x0662:60:a4:23:ff:fe:fd:dc:14 entering async_device_initialized - is_new_join: False
2022-03-13 19:28:57 DEBUG (MainThread) [homeassistant.components.zha.core.gateway] device - 0x0662:60:a4:23:ff:fe:fd:dc:14 has been reset and re-added or its nwk address changed
2022-03-13 19:28:57 DEBUG (MainThread) [homeassistant.components.zha.core.gateway] skipping discovery for previously discovered device - 0x0662:60:a4:23:ff:fe:fd:dc:14
2022-03-13 19:28:57 DEBUG (MainThread) [homeassistant.components.zha.core.device] [0x0662](TS004F): started configuration
2022-03-13 19:28:57 DEBUG (MainThread) [homeassistant.components.zha.core.channels.base] [0x0662:ZDO](TS004F): 'async_configure' stage succeeded
2022-03-13 19:28:57 DEBUG (MainThread) [homeassistant.components.zha.core.device] [0x0662](TS004F): completed configuration
2022-03-13 19:28:57 DEBUG (MainThread) [homeassistant.components.zha.core.device] [0x0662](TS004F): stored in registry: ZhaDeviceEntry(name='unk_manufacturer unk_model', ieee='60:a4:23:ff:fe:fd:dc:14', last_seen=1647196137.3147435)
2022-03-13 19:28:57 DEBUG (MainThread) [homeassistant.components.zha.core.device] [0x0662](TS004F): started initialization
2022-03-13 19:28:57 DEBUG (MainThread) [homeassistant.components.zha.core.channels.base] [0x0662:ZDO](TS004F): 'async_initialize' stage succeeded
2022-03-13 19:28:57 DEBUG (MainThread) [homeassistant.components.zha.core.device] [0x0662](TS004F): power source: Battery or Unknown
2022-03-13 19:28:57 DEBUG (MainThread) [homeassistant.components.zha.core.device] [0x0662](TS004F): completed initialization
2022-03-13 19:28:57 DEBUG (MainThread) [zigpy.zdo] [0x0662:zdo] ZDO request ZDOCmd.Node_Desc_req: [0x0000]
2022-03-13 19:28:57 DEBUG (MainThread) [zigpy.zdo] [0x0662:zdo] No handler for ZDO request:ZDOCmd.Node_Desc_req([0x0000])
2022-03-13 19:28:57 DEBUG (MainThread) [zigpy.zcl] [0x0662:1:0x0000] ZCL deserialize: <ZCLHeader frame_control=<FrameControl frame_type=GLOBAL_COMMAND manufacturer_specific=False is_reply=True disable_default_response=True> manufacturer=None tsn=87 command_id=Command.Read_Attributes_rsp>
2022-03-13 19:28:57 DEBUG (MainThread) [ts004f5B] Tuya Magic Spell result ({'manufacturer': '_TZ3000_xabckq1v', 'zcl_version': 3, 'app_version': 65, 'model': 'TS004F', 'power_source': <PowerSource.Battery: 3>, 65534: <enum8.undefined_0x00: 0>}, {}) for 60:a4:23:ff:fe:fd:dc:14
2022-03-13 19:28:58 DEBUG (MainThread) [zigpy.zcl] [0xff9c:1:0x0006] ZCL deserialize: <ZCLHeader frame_control=<FrameCon

so the casting is working very well and its fast and looks working every time i have testing. Then restarting HA i only getting:

2022-03-13 19:20:10 DEBUG (MainThread) [zigpy.quirks.registry] Checking quirks for _TZ3000_xabckq1v TS004F (60:a4:23:ff:fe:fd:dc:14)
2022-03-13 19:20:10 DEBUG (MainThread) [zigpy.quirks.registry] Considering <class 'ts004f5B.TuyaSmartRemote004FDMS'>
2022-03-13 19:20:10 DEBUG (MainThread) [zigpy.quirks.registry] Found custom device replacement for 60:a4:23:ff:fe:fd:dc:14: <class 'ts004f5B.TuyaSmartRemote004FDMS'>
2022-03-13 19:20:10 DEBUG (MainThread) [ts004f5B] Cast Tuya Magic Spell on 60:a4:23:ff:fe:fd:dc:14
2022-03-13 19:20:10 DEBUG (MainThread) [zigpy.device] [0x2cc8] Extending timeout for 0x01 request

And no error then the spell dont have implanting it but its OK. Was trying with tries=10 and its sitl working OK its timing out before the system have loading the coordinator (bellows) so it cant making any bad with the device also if it not sleeping for the moment. I dont knowing if its working OK with the power stripe if like getting it re casted.

I feel very comfortable with the spell casting and can doing more test with my device that needing it and making one PR with the casting or if you have getting it in INIT i implanted it to my quirk. I only need little help doing the fixes for the import and making the right code changes if i cant copying it from your finished quirk.

I must looking little more in my quirk on my implementation then i have braking default response sending to the no ZCL commands but the quirk is saying its sending it.

Thanks for help !!

PS: I shall also trying upgrading me Power stripes so can testing it, but i think its not being this week then the must do list is being little long.

mdeweerd commented 2 years ago

Very good news for this.

MattWestb commented 2 years ago

If having this on the device class:

    def __init__(self, *args, **kwargs):
        """Initialize with task."""
        super().__init__(*args, **kwargs)

        # Use internal version, "validated", commented to try next method
        # self._init_plug_task = asyncio.create_task(self.spell())

        # Use 'external' version that could be called from cluster
        # customiation
        tuya_magic_spell(self, tries=3)

    async def spell(self) -> None:
        """Initialize device so that all endpoints become available."""

        basic_cluster = self.endpoints[1].in_clusters[0]

        # The magic spell is needed only once.
        # TODO: Improve by doing this only once (successfully).

        # Magic spell - part 1
        attr_to_read = [4, 0, 1, 5, 7, 0xFFFE]
        await basic_cluster.read_attributes(attr_to_read)

        # Magic spell - part 2 (skipped - does not seem to be needed)
        # attr_to_write={0xffde:13}
        # basic_cluster.write_attributes(attr_to_write)

Its working OK every time i joining the device new and its have being "hardware resented" = reseed and battery out so have losing the extra EPs (verified by loading one quirk with the the magic for being sure) and its not making any problems with restart or rejoining the device only its not working if the user is resetting the device and taking the battery out then its need deleting the device and doing one new joining.

I have trying the basic cluster implementation and have seen casting of the spell but then only using the TuyaBasicCluster its not being casted also only that the quirk is loaded in the log and no [ts004f5B] Cast Tuya Magic Spell on 60:a4:23:ff:fe:fd:dc:14 that i getting then using the device class version.

I think i is doing somthing wrong with the basic cluster and how its looking in my Quirk:

sync def tuya_magic_spell_wand(dev, tries=3, rejoin=False) -> None:
#async def cast_tuya_magic_spell_task(
#    dev: zigpy.device.Device, tries: int = 3, rejoin: bool = False
#) -> None:
    """Initialize device so that all endpoints become available."""
    import inspect

    basic_cluster = dev.endpoints[1].in_clusters[0]
    # The magic spell is needed only once.
    # TODO: Improve by doing this only once (successfully).

    # Magic spell - part 1
    # attr_to_read = [4, 0, 1, 5, 7, 0xFFFE]
    # Note: attribute order is important
    attr_to_read = [
        "manufacturer",
        "zcl_version",
        "app_version",
        "model",
        "power_source",
        0xFFFE,
    ]
    if "tries" in inspect.getfullargspec(basic_cluster.read_attributes)[0]:
        await basic_cluster.read_attributes(attr_to_read, tries=tries)
        _LOGGER.debug(f"Cast Tuya Magic Spell on {dev.ieee!r} with {tries} tries")
        res = await basic_cluster.read_attributes(attr_to_read, tries=tries)
    else:
        await basic_cluster.read_attributes(attr_to_read)
        _LOGGER.debug(f"Cast Tuya Magic Spell on {dev.ieee!r}")
        res = await basic_cluster.read_attributes(attr_to_read)

    _LOGGER.debug(f"Tuya Magic Spell result {res!r} for {dev.ieee!r}")

    # Magic spell - part 2 (skipped - does not seem to be needed)
    # attr_to_write={0xffde:13}
    # basic_cluster.write_attributes(attr_to_write)

#    if rejoin:
        # Leave with rejoin - may need to be adjuste to work everywhere
        # or require a minimum zigpy version
        # This should have the device leave and rejoin immediately triggering
        # the discovery of the endpoints that appear after the magic trick

        # Note: this is not validated yet and disabled by default
##        await dev.zdo.request(0x0034, dev.ieee, 0x01, tries)
#        _LOGGER.debug(f"Send leave with rejoin request to {dev.ieee!r}")
#        res = await dev.zdo.request(0x0034, dev.ieee, 0x01, tries)
#        _LOGGER.debug(f"Leave with rejoin result {res!r} for {dev.ieee!r}")
#        _LOGGER.debug(f"Send leave with rejoin request to {dev.ieee!r}")
#        res = await dev.zdo.request(0x0034, dev.ieee, 0x01, tries)
#        _LOGGER.debug(f"Leave with rejoin result {res!r} for {dev.ieee!r}")
#
#        app = dev.application
        # Delete the device from the database
#        app.listener_event("device_removed", dev)
#
        # Delete the device from zigpy
#        app.devices.pop(dev.ieee, None)

#def tuya_magic_spell(dev, tries=3) -> None:
def tuya_magic_spell(dev: zigpy.device.Device, tries: int = 3) -> None:
    """Set up the magic spell asynchronously."""

    dev._magic_spell_task = asyncio.create_task(tuya_magic_spell_wand(dev, tries=tries))
    # Note for sleepy devices the number of tries may need to be increased to 100.
#
#    dev._magic_spell_task = asyncio.create_task(
#        cast_tuya_magic_spell_task(dev, tries=tries)
#    )

I have comment out the leave and rejoin then its not needed for this device and very likely no other then its working great without on TS004F dimer switches and also the bind is not needed and making paring failing with time out of the binding.

I have one user testing the device class and hi was getting the device working OK and is using device automatons so its looks working very good.

I think getting it working then the quirk is being added on the basic cluster is one good thing then we can getting it working OK. Is it also possible implanting it as one cluster commands so user can triggering it from the device card and manage cluster ?

Can you hinting my what i is doing wrong with the basic cluster version ??

mdeweerd commented 2 years ago

The code you share as the "Basic Cluster" code is not the basic cluster at all - it's the functions that the Basic Cluster can call.

The TuyaBasicCluster I proposed is:

class TuyaBasicCluster(CustomCluster, Basic):
    """Provide Tuya Basic Cluster with magic spell."""

    manufacturer_attributes = {
        0xFFDE: ("tuya_FFDE", t.uint8_t),
        # 0xffe0: ("tuya_FFE0", TODO.Array),
        # 0xffe1: ("tuya_FFE1", TODO.Array),
        0xFFE2: ("tuya_FFE2", t.uint8_t),
        # 0xffe3: ("tuya_FFE3", TODO.Array),
    }

    async def bind(self):
        """Bind cluster."""

        _LOGGER.debug(
            f"Requesting Tuya Magic Spell for {self.ieee!r} in basic bind method"
        )
        tries = 3
        await asyncio.create_task(cast_tuya_magic_spell_task(self, tries=tries))

        return await super().bind()

You can change LOGGER.debug to LOGGER.error to make sure the message appears in your log during debug (without setting the debug level for the quirk).

So the first question is:

How did you add the TuyaBasicCluster ? I proposed:

    replacement[1][INPUT_CLUSTERS][0] = TuyaBasicCluster

in the device class. You could add a LOGGER.error("replacement {replacement!r}") to ensure that the content of the replacement is complete and that it is set when zigpy loads the available quirks. If the replacement is incorrect and still shows the original basic cluster (id), you can adjust the assignment. If the remplacement is incomplete, the just modifying the parent's value is not working and the entire replacement would have to be specified.

P.S:

MattWestb commented 2 years ago

Then restart i is getting this in the log:

2022-03-19 14:41:14 DEBUG (MainThread) [zigpy.quirks.registry] Checking quirks for _TZ3000_xabckq1v TS004F (60:a4:23:ff:fe:fd:dc:14)
2022-03-19 14:41:14 DEBUG (MainThread) [zigpy.quirks.registry] Considering <class 'ts004f5B.TuyaSmartRemote004FDMS'>
2022-03-19 14:41:14 DEBUG (MainThread) [zigpy.quirks.registry] Found custom device replacement for 60:a4:23:ff:fe:fd:dc:14: <class 'ts004f5B.TuyaSmartRemote004FDMS'>
2022-03-19 14:41:14 DEBUG (MainThread) [ts004f5B] Device clss is calling Tuya Magic Spell from devce class

Its only one log print from the device class but no from the tuya basic cluster then its not initiated at the moment = OK for my.

Reconfigure is not logging any magic at all.

I was changing my device class to this:

    def __init__(self, *args, **kwargs):
        """Initialize with task."""
        super().__init__(*args, **kwargs)

        # Use internal version, "validated", commented to try next method
        # self._init_plug_task = asyncio.create_task(self.spell())

        # Use 'external' version that could be called from cluster
        # customiation
        tuya_magic_spell(self, tries=3)

        _LOGGER.debug(f"Device clss is calling Tuya Magic Spell from devce class")

And i is getting this then adding the device then its hardware resented:

2022-03-19 14:14:30 DEBUG (MainThread) [zigpy.quirks.registry] Checking quirks for _TZ3000_xabckq1v TS004F (60:a4:23:ff:fe:fd:dc:14)
2022-03-19 14:14:30 DEBUG (MainThread) [zigpy.quirks.registry] Considering <class 'ts004f5B.TuyaSmartRemote004FDMS'>
2022-03-19 14:14:30 DEBUG (MainThread) [zigpy.quirks.registry] Found custom device replacement for 60:a4:23:ff:fe:fd:dc:14: <class 'ts004f5B.TuyaSmartRemote004FDMS'>
2022-03-19 14:14:30 DEBUG (MainThread) [ts004f5B] Device class is calling Tuya Magic Spell from device class
2022-03-19 14:14:30 DEBUG (MainThread) [zigpy.device] [0x09b1] Extending timeout for 0x5c request
2022-03-19 14:14:30 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'button' component -> 'ZHAIdentifyButton' using ['identify']
2022-03-19 14:14:30 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'sensor' component -> 'Battery' using ['power']
2022-03-19 14:14:30 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'sensor' component -> 'RSSISensor' using ['basic']
2022-03-19 14:14:30 DEBUG (MainThread) [homeassistant.components.zha.core.discovery] 'sensor' component -> 'LQISensor' using ['basic']
2022-03-19 14:14:30 DEBUG (MainThread) [homeassistant.core] Bus:Handling <Event device_registry_updated[L]: action=create, device_id=f0aca6ce4c51450cf16577d5d2d4e4b4>
2022-03-19 14:14:30 DEBUG (MainThread) [homeassistant.components.zha.core.gateway] device - 0x09B1:60:a4:23:ff:fe:fd:dc:14 entering async_device_initialized - is_new_join: True
2022-03-19 14:14:30 DEBUG (MainThread) [homeassistant.components.zha.core.gateway] device - 0x09B1:60:a4:23:ff:fe:fd:dc:14 has joined the ZHA zigbee network
2022-03-19 14:14:30 DEBUG (MainThread) [homeassistant.components.zha.core.device] [0x09B1](TS004F): started configuration
2022-03-19 14:14:30 DEBUG (MainThread) [homeassistant.components.zha.core.channels.base] [0x09B1:ZDO](TS004F): 'async_configure' stage succeeded
2022-03-19 14:14:30 DEBUG (MainThread) [zigpy.appdb] Error handling '_save_attribute' event with (60:a4:23:ff:fe:fd:dc:14, 1, 0, 4, '_TZ3000_xabckq1v') params: FOREIGN KEY constraint failed
2022-03-19 14:14:30 DEBUG (MainThread) [homeassistant.components.zha.core.channels.base] [0x09B1:1:0x0003]: finished channel configuration
2022-03-19 14:14:30 DEBUG (MainThread) [zigpy.device] [0x09b1] Extending timeout for 0x5e request
2022-03-19 14:14:30 DEBUG (MainThread) [homeassistant.components.zha.core.channels.base] [0x09B1:1:0x0000]: finished channel configuration
2022-03-19 14:14:30 DEBUG (MainThread) [zigpy.device] [0x09b1] Extending timeout for 0x60 request
2022-03-19 14:14:30 DEBUG (MainThread) [zigpy.device] [0x09b1] Extending timeout for 0x62 request
2022-03-19 14:14:30 DEBUG (MainThread) [zigpy.appdb] Error handling '_save_attribute' event with (60:a4:23:ff:fe:fd:dc:14, 1, 0, 5, 'TS004F') params: FOREIGN KEY constraint failed
2022-03-19 14:14:31 DEBUG (MainThread) [zigpy.zcl] [0x09b1:1:0x0000] ZCL deserialize: <ZCLHeader frame_control=<FrameControl frame_type=GLOBAL_COMMAND manufacturer_specific=False is_reply=True disable_default_response=True> manufacturer=None tsn=92 command_id=Command.Read_Attributes_rsp>
2022-03-19 14:14:31 DEBUG (MainThread) [ts004f5B] Cast Tuya Magic Spell on 60:a4:23:ff:fe:fd:dc:14
2022-03-19 14:14:31 DEBUG (MainThread) [zigpy.device] [0x09b1] Extending timeout for 0x64 request
2022-03-19 14:14:31 DEBUG (MainThread) [homeassistant.components.zha.core.channels.base] [0x09B1:1:0x0001]: bound 'power' cluster: Status.SUCCESS
2022-03-19 14:14:31 DEBUG (MainThread) [zigpy.device] [0x09b1] Extending timeout for 0x66 request
2022-03-19 14:14:31 DEBUG (MainThread) [zigpy.zcl] [0x09b1:1:0x1000] ZCL deserialize: <ZCLHeader frame_control=<FrameControl frame_type=GLOBAL_COMMAND manufacturer_specific=False is_reply=True disable_default_response=False> manufacturer=None tsn=96 command_id=Command.Default_Response>
2022-03-19 14:14:31 DEBUG (MainThread) [zigpy.device] [0x09b1] Extending timeout for 0x68 request
2022-03-19 14:14:31 DEBUG (MainThread) [zigpy.zcl] [0x09b1:1:0x0006] ZCL deserialize: <ZCLHeader frame_control=<FrameControl frame_type=GLOBAL_COMMAND manufacturer_specific=False is_reply=True disable_default_response=False> manufacturer=None tsn=0 command_id=Command.Report_Attributes>
2022-03-19 14:14:31 DEBUG (MainThread) [zigpy.zcl] [0x09b1:1:0x0006] ZCL request 0x000a: [[Attribute(attrid=32772, value=<TypeValue type=enum8, value=enum8.undefined_0x00>)]]
2022-03-19 14:14:31 DEBUG (MainThread) [zigpy.zcl] [0x09b1:1:0x0006] Attribute report received: switch_mode=0
2022-03-19 14:14:31 DEBUG (MainThread) [homeassistant.core] Bus:Handling <Event zha_event[L]: device_ieee=60:a4:23:ff:fe:fd:dc:14, unique_id=60:a4:23:ff:fe:fd:dc:14:1:0x0006, device_id=f0aca6ce4c51450cf16577d5d2d4e4b4, endpoint_id=1, cluster_id=6, command=attribute_updated, args=attribute_id=32772, attribute_name=switch_mode, value=SwitchMode.Command>
2022-03-19 14:14:31 DEBUG (MainThread) [homeassistant.core] Bus:Handling <Event zha_event[L]: device_ieee=60:a4:23:ff:fe:fd:dc:14, unique_id=60:a4:23:ff:fe:fd:dc:14:1:0x0006, device_id=f0aca6ce4c51450cf16577d5d2d4e4b4, endpoint_id=1, cluster_id=6, command=attribute_updated, args=attribute_id=32772, attribute_name=switch_mode, value=SwitchMode.Command>
2022-03-19 14:14:31 DEBUG (MainThread) [zigpy.zcl] [0x09b1:1:0x0000] ZCL deserialize: <ZCLHeader frame_control=<FrameControl frame_type=GLOBAL_COMMAND manufacturer_specific=False is_reply=True disable_default_response=True> manufacturer=None tsn=100 command_id=Command.Read_Attributes_rsp>
2022-03-19 14:14:31 DEBUG (MainThread) [homeassistant.components.zha.core.channels.base] [0x09B1:1:0x0008]: bound 'level' cluster: Status.SUCCESS
2022-03-19 14:14:31 DEBUG (MainThread) [homeassistant.components.zha.core.channels.base] [0x09B1:1:0x0008]: finished channel configuration
2022-03-19 14:14:31 DEBUG (MainThread) [ts004f5B] Tuya Magic Spell result ({'manufacturer': '_TZ3000_xabckq1v', 'zcl_version': 3, 'app_version': 65, 'model': 'TS004F', 'power_source': <PowerSource.Battery: 3>, 65534: <enum8.undefined_0x00: 0>}, {}) for 60:a4:23:ff:fe:fd:dc:14
2022-03-19 14:14:31 DEBUG (MainThread) [homeassistant.components.zha.core.channels.base] [0x09B1:1:0x0019]: finished channel configuration
2022-03-19 14:14:31 DEBUG (MainThread) [zigpy.device] [0x09b1] Extending timeout for 0x6b request
2022-03-19 14:14:31 DEBUG (MainThread) [zigpy.zcl] [0x09b1:1:0x0001] ZCL deserialize: <ZCLHeader frame_control=<FrameControl frame_type=GLOBAL_COMMAND manufacturer_specific=False is_reply=True disable_default_response=True> manufacturer=None tsn=102 command_id=Command.Configure_Reporting_rsp>
2022-03-19 14:14:31 DEBUG (MainThread) [homeassistant.components.zha.core.channels.base] [0x09B1:1:0x0001]: Successfully configured reporting for '{'battery_voltage': (3600, 10800, 1), 'battery_percentage_remaining': (3600, 10800, 1)}' on 'power' cluster: [ConfigureReportingResponseRecord(status=0)]
2022-03-19 14:14:31 DEBUG (MainThread) [homeassistant.components.zha.core.channels.base] [0x09B1:1:0x0001]: finished channel configuration

That showing that my log print "Device class is calling Tuya Magic Spell from device class" is comming vers fast after the quirks is being loaded but the sending is being done little before the binding of power cluster.

That is making my thinking then the basic cluster is self sending the magic its to late for the TS004F dimer switches then have starting configuring the cluster the device is going out from INIT mode and we is not getting the "new" endpoints.

For my its OK calling the tuya_magic_spell/tuya_magic_spell_wand as long its not doing other strange things that is making it being delayed and is not being casted very early to the device then its joining the network.

I also thinking for more normal tuya device is OK using the basic cluster for casting the spell (then the casting from basic cluster is working) and its easier making it more general adding it to all device quirk that can needing it in the future.

I think i doing one PR for adding the new TS004F knob and the scene mode automatons and leaving this out for the moment so it can stabilizing and user can using local quirk for adding the magic on there device and then using the standard quirk in ZHA.

mdeweerd commented 2 years ago
MattWestb commented 2 years ago

OK im over and out .

I have making one PR for adding it for our TS004F devices and is using the device class method that working 100% for the TS004F Dimmer Switch.

If some one like implanting it in one other way and have the knowledge, the devices and understanding how the TS004F is working in real Hi is very welcome doing one PR then we have users that have demanding this functions for 10 mouths and its needed.

Thanks for helping !!

Then im have time i shall testing upgrading my LIDL power strips and looking if its working OK with the magic spell but i have no time for that for the moment (still 10 mount work with users in the TS004F case).

MattWestb commented 2 years ago

One more tuya device that was needing the spell being casted but not the joining but later and its one DP / MCU device. https://github.com/Koenkk/zigbee2mqtt/issues/7695#issuecomment-1079660118 So i think the basic cluster is being good putting in all tuya devices quirks also DP/MCU device after testing its not braking the function.

MattWestb commented 2 years ago

I was moving one of my (2) LIDL power strips from my production ZHA to my LIDL ZBGW and its reporting firmware for the zigbee module 1.0.5 that is the original Zigbee certified version and and its have no update to the new problematic version :-(((

I letting it being one the tuya / LIDL ZBGW over the weekend and see if LIDL is changing its mind :-))

By the way power on state and LED functions is not prescient in the LIDL app but in ZHA !!

github-actions[bot] commented 1 year ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.