Koenkk / zigbee2mqtt

Zigbee 🐝 to MQTT bridge 🌉, get rid of your proprietary Zigbee bridges 🔨
https://www.zigbee2mqtt.io
GNU General Public License v3.0
12.24k stars 1.69k forks source link

ZY-M100-S_1 sensors report every second #19045

Open seblu opened 1 year ago

seblu commented 1 year ago

What happened?

The Tuya ZY-M100-S_1 (_TZE204_sxm7l9xa) is reporting its values every second. This values can or can not changes.

Here is an example with values not changing: Screenshot_20230921_231622

What did you expect to happen?

A delay or minimal value change that can be configured in the Reporting tab.

How to reproduce it (minimal and precise)

Pair the sensor and it starts sending data every second or 2.

Zigbee2MQTT version

1.33.0

Adapter firmware version

20221226

Adapter

Silicon_Labs_Sonoff_Zigbee_3.0_USB_Dongle_Plus_0001

Debug log

debug 2023-09-21 23:20:33: Received Zigbee message from 'Toilet Presence', type 'commandDataReport', cluster 'manuSpecificTuya', data '{"dpValues":[{"data":{"data":[0,0,0,0],"type":"Buffer"},"datatype":2,"dp":109}],"seq":6912}' from endpoint 1 with groupID 0
info  2023-09-21 23:20:33: MQTT publish: topic 'zigbee2mqtt/Toilet Presence', payload '{"detection_delay":0,"fading_time":5,"illuminance_lux":3,"last_seen":"2023-09-21T21:20:33.910Z","linkquality":105,"maximum_range":1.95,"minimum_range":0.75,"presence":false,"radar_sensitivity":0,"target_distance":0}'
debug 2023-09-21 23:20:35: Received Zigbee message from 'Toilet Presence', type 'commandDataReport', cluster 'manuSpecificTuya', data '{"dpValues":[{"data":{"data":[0,0,0,0],"type":"Buffer"},"datatype":2,"dp":109}],"seq":7168}' from endpoint 1 with groupID 0
info  2023-09-21 23:20:35: MQTT publish: topic 'zigbee2mqtt/Toilet Presence', payload '{"detection_delay":0,"fading_time":5,"illuminance_lux":3,"last_seen":"2023-09-21T21:20:35.506Z","linkquality":105,"maximum_range":1.95,"minimum_range":0.75,"presence":false,"radar_sensitivity":0,"target_distance":0}'
debug 2023-09-21 23:20:37: Received Zigbee message from 'Toilet Presence', type 'commandDataReport', cluster 'manuSpecificTuya', data '{"dpValues":[{"data":{"data":[0,0,0,0],"type":"Buffer"},"datatype":2,"dp":109}],"seq":7424}' from endpoint 1 with groupID 0
info  2023-09-21 23:20:37: MQTT publish: topic 'zigbee2mqtt/Toilet Presence', payload '{"detection_delay":0,"fading_time":5,"illuminance_lux":3,"last_seen":"2023-09-21T21:20:37.101Z","linkquality":105,"maximum_range":1.95,"minimum_range":0.75,"presence":false,"radar_sensitivity":0,"target_distance":0}'
debug 2023-09-21 23:20:38: Received Zigbee message from 'Toilet Presence', type 'commandDataReport', cluster 'manuSpecificTuya', data '{"dpValues":[{"data":{"data":[0,0,0,0],"type":"Buffer"},"datatype":2,"dp":109}],"seq":7680}' from endpoint 1 with groupID 0
info  2023-09-21 23:20:38: MQTT publish: topic 'zigbee2mqtt/Toilet Presence', payload '{"detection_delay":0,"fading_time":5,"illuminance_lux":3,"last_seen":"2023-09-21T21:20:38.697Z","linkquality":102,"maximum_range":1.95,"minimum_range":0.75,"presence":false,"radar_sensitivity":0,"target_distance":0}'
debug 2023-09-21 23:20:40: Received Zigbee message from 'Toilet Presence', type 'commandDataReport', cluster 'manuSpecificTuya', data '{"dpValues":[{"data":{"data":[0,0,0,0],"type":"Buffer"},"datatype":2,"dp":109}],"seq":7936}' from endpoint 1 with groupID 0
info  2023-09-21 23:20:40: MQTT publish: topic 'zigbee2mqtt/Toilet Presence', payload '{"detection_delay":0,"fading_time":5,"illuminance_lux":3,"last_seen":"2023-09-21T21:20:40.293Z","linkquality":105,"maximum_range":1.95,"minimum_range":0.75,"presence":false,"radar_sensitivity":0,"target_distance":0}'
debug 2023-09-21 23:20:41: Received Zigbee message from 'Toilet Presence', type 'commandDataReport', cluster 'manuSpecificTuya', data '{"dpValues":[{"data":{"data":[0,0,0,0],"type":"Buffer"},"datatype":2,"dp":109}],"seq":8192}' from endpoint 1 with groupID 0
info  2023-09-21 23:20:41: MQTT publish: topic 'zigbee2mqtt/Toilet Presence', payload '{"detection_delay":0,"fading_time":5,"illuminance_lux":3,"last_seen":"2023-09-21T21:20:41.889Z","linkquality":109,"maximum_range":1.95,"minimum_range":0.75,"presence":false,"radar_sensitivity":0,"target_distance":0}'
debug 2023-09-21 23:20:43: Received Zigbee message from 'Toilet Presence', type 'commandDataReport', cluster 'manuSpecificTuya', data '{"dpValues":[{"data":{"data":[0,0,0,0],"type":"Buffer"},"datatype":2,"dp":109}],"seq":8448}' from endpoint 1 with groupID 0
info  2023-09-21 23:20:43: MQTT publish: topic 'zigbee2mqtt/Toilet Presence', payload '{"detection_delay":0,"fading_time":5,"illuminance_lux":3,"last_seen":"2023-09-21T21:20:43.684Z","linkquality":105,"maximum_range":1.95,"minimum_range":0.75,"presence":false,"radar_sensitivity":0,"target_distance":0}'
debug 2023-09-21 23:20:45: Received Zigbee message from 'Toilet Presence', type 'commandDataReport', cluster 'manuSpecificTuya', data '{"dpValues":[{"data":{"data":[0,0,0,0],"type":"Buffer"},"datatype":2,"dp":109}],"seq":8704}' from endpoint 1 with groupID 0
info  2023-09-21 23:20:45: MQTT publish: topic 'zigbee2mqtt/Toilet Presence', payload '{"detection_delay":0,"fading_time":5,"illuminance_lux":3,"last_seen":"2023-09-21T21:20:45.081Z","linkquality":102,"maximum_range":1.95,"minimum_range":0.75,"presence":false,"radar_sensitivity":0,"target_distance":0}'
debug 2023-09-21 23:20:46: Received Zigbee message from 'Toilet Presence', type 'commandDataReport', cluster 'manuSpecificTuya', data '{"dpValues":[{"data":{"data":[0,0,0,0],"type":"Buffer"},"datatype":2,"dp":109}],"seq":8960}' from endpoint 1 with groupID 0
info  2023-09-21 23:20:46: MQTT publish: topic 'zigbee2mqtt/Toilet Presence', payload '{"detection_delay":0,"fading_time":5,"illuminance_lux":3,"last_seen":"2023-09-21T21:20:46.674Z","linkquality":105,"maximum_range":1.95,"minimum_range":0.75,"presence":false,"radar_sensitivity":0,"target_distance":0}'
debug 2023-09-21 23:20:48: Received Zigbee message from 'Toilet Presence', type 'commandDataReport', cluster 'manuSpecificTuya', data '{"dpValues":[{"data":{"data":[0,0,0,0],"type":"Buffer"},"datatype":2,"dp":109}],"seq":9216}' from endpoint 1 with groupID 0
info  2023-09-21 23:20:48: MQTT publish: topic 'zigbee2mqtt/Toilet Presence', payload '{"detection_delay":0,"fading_time":5,"illuminance_lux":3,"last_seen":"2023-09-21T21:20:48.272Z","linkquality":105,"maximum_range":1.95,"minimum_range":0.75,"presence":false,"radar_sensitivity":0,"target_distance":0}'
debug 2023-09-21 23:20:49: Received Zigbee message from 'Toilet Presence', type 'commandDataReport', cluster 'manuSpecificTuya', data '{"dpValues":[{"data":{"data":[0,0,0,0],"type":"Buffer"},"datatype":2,"dp":109}],"seq":9472}' from endpoint 1 with groupID 0
info  2023-09-21 23:20:49: MQTT publish: topic 'zigbee2mqtt/Toilet Presence', payload '{"detection_delay":0,"fading_time":5,"illuminance_lux":3,"last_seen":"2023-09-21T21:20:49.866Z","linkquality":109,"maximum_range":1.95,"minimum_range":0.75,"presence":false,"radar_sensitivity":0,"target_distance":0}'
debug 2023-09-21 23:20:51: Received Zigbee message from 'Toilet Presence', type 'commandDataReport', cluster 'manuSpecificTuya', data '{"dpValues":[{"data":{"data":[0,0,0,0],"type":"Buffer"},"datatype":2,"dp":109}],"seq":9728}' from endpoint 1 with groupID 0
info  2023-09-21 23:20:51: MQTT publish: topic 'zigbee2mqtt/Toilet Presence', payload '{"detection_delay":0,"fading_time":5,"illuminance_lux":3,"last_seen":"2023-09-21T21:20:51.462Z","linkquality":105,"maximum_range":1.95,"minimum_range":0.75,"presence":false,"radar_sensitivity":0,"target_distance":0}'
debug 2023-09-21 23:20:53: Received Zigbee message from 'Toilet Presence', type 'commandDataReport', cluster 'manuSpecificTuya', data '{"dpValues":[{"data":{"data":[0,0,0,0],"type":"Buffer"},"datatype":2,"dp":109}],"seq":9984}' from endpoint 1 with groupID 0
info  2023-09-21 23:20:53: MQTT publish: topic 'zigbee2mqtt/Toilet Presence', payload '{"detection_delay":0,"fading_time":5,"illuminance_lux":3,"last_seen":"2023-09-21T21:20:53.058Z","linkquality":105,"maximum_range":1.95,"minimum_range":0.75,"presence":false,"radar_sensitivity":0,"target_distance":0}'
debug 2023-09-21 23:20:54: Received Zigbee message from 'Toilet Presence', type 'commandDataReport', cluster 'manuSpecificTuya', data '{"dpValues":[{"data":{"data":[0,0,0,0],"type":"Buffer"},"datatype":2,"dp":109}],"seq":10240}' from endpoint 1 with groupID 0
info  2023-09-21 23:20:54: MQTT publish: topic 'zigbee2mqtt/Toilet Presence', payload '{"detection_delay":0,"fading_time":5,"illuminance_lux":3,"last_seen":"2023-09-21T21:20:54.653Z","linkquality":105,"maximum_range":1.95,"minimum_range":0.75,"presence":false,"radar_sensitivity":0,"target_distance":0}'
debug 2023-09-21 23:20:56: Received Zigbee message from 'Toilet Presence', type 'commandDataReport', cluster 'manuSpecificTuya', data '{"dpValues":[{"data":{"data":[0,0,0,0],"type":"Buffer"},"datatype":2,"dp":109}],"seq":10496}' from endpoint 1 with groupID 0
info  2023-09-21 23:20:56: MQTT publish: topic 'zigbee2mqtt/Toilet Presence', payload '{"detection_delay":0,"fading_time":5,"illuminance_lux":3,"last_seen":"2023-09-21T21:20:56.250Z","linkquality":109,"maximum_range":1.95,"minimum_range":0.75,"presence":false,"radar_sensitivity":0,"target_distance":0}'
jvdburgt commented 3 months ago

Can any of you with the hub awaiting an OTA make sure to grab the FW file and share it in zigbee-OTA repo? https://www.zigbee2mqtt.io/advanced/more/tuya_xiaomi_ota_url.html

I'm very happy to do this, but I'm a bit confused as to what I should do now. I'm waiting for the FW update to be pushed by the seller, right? And this update will either be installed automatically, or I'll do it via the Smartlife app when it is available. But then what? How does the Zigbee ota-repo fit in to all that? What should I do (and when) step-by-step?

Great. Have a look at the link I posted above, it'll give you a step by step of what you need to do. Basically you need an emulator with the correct/linked APK installed where you check for updates (through the emulator, not your phone) and then capture the traffic via the mitmproxy.

Ah, wait, I got it (I think): this is not part of my device update process, but rather to share the OTA with the community, so they can also update their devices, right?

Andrik45719 commented 3 months ago

ZY-M100-L (_TZE204_ztc6ggyl) solving network spamming problem by disabling Target Distance reporting

itwtds commented 3 months ago

ZY-M100-L (_TZE204_ztc6ggyl) solving network spamming problem by disabling Target Distance reporting

Great work! Let's keep 'em coming for other revisions as well.

vdhmax commented 3 months ago

I bumped into this thread looking for a more shallow ceiling sensor. Are there any sellers that are, or planning to, sell these sensor with updated 1.0.5 firmware pre-installed? I don't have or plan to buy a Tuya gateway and am at the start of my journey with these sensors.

LeeWiu commented 3 months ago

I recommend Wenzhi Factory Store https://a.aliexpress.com/_EyVo3Zf Please contact them if they can send devices with updated firmware.

vdhmax commented 3 months ago

FYI for everyone else: they will happily send new version of the 5.8g one if you ask specifically when you order (as a comment) starting tomorrow.

bsaurusrex commented 3 months ago

Does 5.8g interfere with wifi?

On Tue, Aug 13, 2024 at 6:33 PM vdhmax @.***> wrote:

FYI for everyone else: they will happily send new version of the 5.8g one if you ask specifically when you order (as a comment) starting tomorrow.

— Reply to this email directly, view it on GitHub https://github.com/Koenkk/zigbee2mqtt/issues/19045#issuecomment-2285920643, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATUKSJYOQ7V27NR4NE2S6M3ZRHOGBAVCNFSM6AAAAAA5CFPYD2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBVHEZDANRUGM . You are receiving this because you commented.Message ID: @.***>

bsaurusrex commented 3 months ago

I have requested both Wenzhi and Loginovo Aliexpress stores to push the updated firmware.

Wenzhi: No response or read receipt of daily reminders for over 7 days. Loginovo: Responding but keeps telling me "the engineer is checking it" for over 7 days.

tandarra commented 3 months ago

Have had a reply from Loginovo, however i'm away for work for another 2 weeks and unable to test.

If someone else is able to test that would be appreciated

https://drive.google.com/file/d/1EbwO5oBQPw5lyIdVV39ZZh7O50PMvfPM/view?spm=a1z65.imconversation.0.0&statusBarHeight=107&_dognoseId=WllrczgyaVNuallEQU56Z0FnQ25NRFABkW8aQsVhbGlleHByZQAAARqA&_currency=NZD&_lang=en_NZ&fromApp=true -1089637104885814915 (1) Screenshot_2024-08-20-18-16-44-42_57e717c094f371a1dada6567a1123b99

JackTalisker commented 3 months ago

I have the _TZE204_ijxvkhd0 from Haozee vendor (https://www.aliexpress.com/item/1005004724689720.html) 24G wall mount. Zigbee fw is 1.0.10 and MCU fw is 1.1.8

It spams the log starting once every two minutes then it stabilizes spamming every second with a dozen of lines (every single second!)

Info 2024-08-20 12:54:19z2m:mqtt: MQTT publish: topic 'zigbee2mqtt/0xa4c138fbxxxxxx', payload '{"distance":0,"illuminance_lux":65,"last_seen":"2024-08-20T10:54:19.146Z","linkquality":200,"move_sensitivity":6,"presence":false,"presence_sensitivity":6,"presence_timeout":3,"radar_range":1.5,"state":"none"}' Info 2024-08-20 12:54:19z2m:mqtt: MQTT publish: topic 'zigbee2mqtt/0xa4c138fbxxxxxx', payload '{"distance":0,"illuminance_lux":65,"last_seen":"2024-08-20T10:54:19.236Z","linkquality":200,"move_sensitivity":6,"presence":false,"presence_sensitivity":6,"presence_timeout":3,"radar_range":1.5,"state":"none"}' Info 2024-08-20 12:54:19z2m:mqtt: MQTT publish: topic 'zigbee2mqtt/0xa4c138fbxxxxxx', payload '{"distance":0,"illuminance_lux":65,"last_seen":"2024-08-20T10:54:19.326Z","linkquality":200,"move_sensitivity":6,"presence":false,"presence_sensitivity":6,"presence_timeout":3,"radar_range":1.5,"state":"none"}' Info 2024-08-20 12:54:19z2m:mqtt: MQTT publish: topic 'zigbee2mqtt/0xa4c138fbxxxxxx', payload '{"distance":0,"illuminance_lux":65,"last_seen":"2024-08-20T10:54:19.420Z","linkquality":200,"move_sensitivity":6,"presence":false,"presence_sensitivity":6,"presence_timeout":3,"radar_range":1.5,"state":"none"}' Info 2024-08-20 12:54:19z2m:mqtt: MQTT publish: topic 'zigbee2mqtt/0xa4c138fbxxxxxx', payload '{"distance":0,"illuminance_lux":65,"last_seen":"2024-08-20T10:54:19.512Z","linkquality":200,"move_sensitivity":6,"presence":false,"presence_sensitivity":6,"presence_timeout":3,"radar_range":1.5,"state":"none"}' Info 2024-08-20 12:54:19z2m:mqtt: MQTT publish: topic 'zigbee2mqtt/0xa4c138fbxxxxxx', payload '{"distance":0,"illuminance_lux":65,"last_seen":"2024-08-20T10:54:19.604Z","linkquality":196,"move_sensitivity":6,"presence":false,"presence_sensitivity":6,"presence_timeout":3,"radar_range":1.5,"state":"none"}' Info 2024-08-20 12:54:19z2m:mqtt: MQTT publish: topic 'zigbee2mqtt/0xa4c138fbxxxxxx', payload '{"distance":0,"illuminance_lux":65,"last_seen":"2024-08-20T10:54:19.694Z","linkquality":200,"move_sensitivity":6,"presence":false,"presence_sensitivity":6,"presence_timeout":3,"radar_range":1.5,"state":"none"}' Info 2024-08-20 12:54:19z2m:mqtt: MQTT publish: topic 'zigbee2mqtt/0xa4c138fbxxxxxx', payload '{"distance":0,"illuminance_lux":65,"last_seen":"2024-08-20T10:54:19.790Z","linkquality":200,"move_sensitivity":6,"presence":false,"presence_sensitivity":6,"presence_timeout":3,"radar_range":1.5,"state":"none"}' Info 2024-08-20 12:54:20z2m:mqtt: MQTT publish: topic 'zigbee2mqtt/0xa4c138fbxxxxxx', payload '{"distance":0,"illuminance_lux":65,"last_seen":"2024-08-20T10:54:19.883Z","linkquality":200,"move_sensitivity":6,"presence":false,"presence_sensitivity":6,"presence_timeout":3,"radar_range":1.5,"state":"none"}'

Any help for ijxvkhd0 is appreciated...

I just wrote a message to the customer care of Haozee vendor. I will report any progress here.

itwtds commented 3 months ago

Have had a reply from Loginovo, however i'm away for work for another 2 weeks and unable to test.

If someone else is able to test that would be appreciated

https://drive.google.com/file/d/1EbwO5oBQPw5lyIdVV39ZZh7O50PMvfPM/view?spm=a1z65.imconversation.0.0&statusBarHeight=107&_dognoseId=WllrczgyaVNuallEQU56Z0FnQ25NRFABkW8aQsVhbGlleHByZQAAARqA&_currency=NZD&_lang=en_NZ&fromApp=true -1089637104885814915 (1) Screenshot_2024-08-20-18-16-44-42_57e717c094f371a1dada6567a1123b99

I'm failing to see how this can be solved on the server side, as it would still congest the zigbee network with these reports. There is no support for turning this off for the earlier FW's.

tandarra commented 3 months ago

I agree it probably just removes the reporting from the logs in zigbee2mqtt. But at this stage what is there to lose except try what the vendor is suggesting and see exactly what impact it has on the system.

As mentioned, I would have tried it, but I work offshore for a month at a time so can't test anything just yet

JackTalisker commented 3 months ago

The solution from loginovo does not work. I Tried it (with my DPs) but masks only the distance value while the device floods the network anyway. I 'm talking with Haozee vendor which today proposed me the same useless workaround.

buhito81 commented 3 months ago

Just ordered a MTG075-ZB-RL of WzWenzhi on AliExpress. I'll let you know when it arrives, and will see how this goes with the firmware. Have (quite) some MTG075-ZB-RL's laying around with this issue, as some MTG275-ZB-RL's and some ZY-M100-S_1 devices. Setting up mitmproxy/mitmproxy in Proxmox on Alpine (for the firmware) was rather easy.

I'll try to grab the firmware, but AAPKS appears to be down that held the Tuya App version that was needed. Can someone point me in the right direction?

The order will be on the way for the next two weeks, after that I'll try grabbing the firmware.

image

pniewiadowski commented 3 months ago

@buhito81 I would say use this one instead: https://www.apkmirror.com/apk/volcano-technology-limited/smart-life-smart-living/smart-life-smart-living-3-12-6-release/

DeX2K commented 2 months ago

For info, wzWenzhi rereleased a firmware for WZ75 series:ZigBee 5.8G circular(MTG075-ZB-RL) , ZigBee 24G circular(MTG275-ZB-RL) , ZigBee 5.8G ceiling(MTG035-ZB-RL) , ZigBee 24G ceiling(MTG235-ZB-RL)

https://github.com/wzwenzhi/Wenzhi-ZigBee2mqtt/issues/11

They provide the bin file firmware. However i have not clue how to convert this to ota for Z2M

jvdburgt commented 2 months ago

Any chance that can be made to work for the ZY-M100 S1 and S2 as well?

Andrik45719 commented 2 months ago

https://github.com/Andrik45719/ZY-M100-L

Added solution for ZY-M100-S_2 _TZE204_qasjif9e

tandarra commented 2 months ago

@Andrik45719 you mention 5.8g but anyway this can be modified/applied to work with ZY-M100 24g?

buhito81 commented 2 months ago

https://github.com/Andrik45719/ZY-M100-L

Added solution for ZY-M100-S_2 _TZE204_qasjif9e

Judging from your solution it appears as if I need additional devices for this (JLINK Adapter)?

evil-life commented 2 months ago

https://github.com/Andrik45719/ZY-M100-L

Added solution for ZY-M100-S_2 _TZE204_qasjif9e

thank you for your work! I need to know if this is working also for 24g version and if it's possible to configure an OTA update for these devices via Zigbee2Mqtt as I'm not very confident in flashing them, also I don't have the hardware... Thank you again!

walberjunior commented 2 months ago

https://github.com/Andrik45719/ZY-M100-L Added solution for ZY-M100-S_2 _TZE204_qasjif9e

Judging from your solution it appears as if I need additional devices for this (JLINK Adapter)?

Yes. I bought the model from the link below and received it today. I flashed one of the sensors. I'll compare the difference tomorrow.

https://pt.aliexpress.com/item/1005006998022759.html?spm=a2g0o.order_list.order_list_main.11.4834caa4iSpNOy&gatewayAdapt=glo2bra

walberjunior commented 2 months ago

Thanks @Andrik45719

I'll try to do some more tests, but comparing 2 sensors in two rooms with no movement, there was a huge reduction in the amount of status reported.

image

image

jphooiveld commented 2 months ago

For info, wzWenzhi rereleased a firmware for WZ75 series:ZigBee 5.8G circular(MTG075-ZB-RL) , ZigBee 24G circular(MTG275-ZB-RL) , ZigBee 5.8G ceiling(MTG035-ZB-RL) , ZigBee 24G ceiling(MTG235-ZB-RL)

wzwenzhi/Wenzhi-ZigBee2mqtt#11

They provide the bin file firmware. However i have not clue who to convert this to ota for Z2M

I tried to add the firmware files manually to zigbee2mqtt but I think zigbee2mqtt just cant seem to match it to any of the MTG275-ZB-RL devices I have. I don´t see any new updates appearing under the OTA tab in zigbee2mqtt. Adding the referenced converters js files doesn´t seem to do anything either.

java-devil commented 2 months ago

@Koenkk would you be so kind as to put us out of our misery and confirm or deny if we are somehow able to release the firmware OTA in Z2M if we have the binaries? And if so - could you point us in the direction of how we could go around doing this?

I'm personally willing to go to considerable lengths here since... I have 8 of these devices embedded in my ceiling. Therefore I'm physically unable of flashing them or even re-paring. They are devastating to the connectivity in my network. And I don't currently have the money, nor the WAF to rip them out, flash them, and repair the ceiling.

It's a shame especially since otherwise... they are really awesome presence sensors from my experience.

Koenkk commented 2 months ago

@java-devil if we have the update files yes, but the ones from https://github.com/wzwenzhi/Wenzhi-ZigBee2mqtt/issues/11 are not Zigbee OTA files.

buhito81 commented 2 months ago

https://github.com/Koenkk/zigbee2mqtt/issues/19045#issuecomment-2318150370

Apparently that version of SmartLife doesn't support this type of Zigbee device:

image

Any alternatives for a obtaining the OTA url?

pniewiadowski commented 2 months ago

@buhito81 that's might be pity as the manual says newer version wouldn't work, but anyway I would try my luck with a newer version

tandarra commented 2 months ago

Thanks @Andrik45719 for your work, much appreciated! Managed to flash all my 24G (ZY-M100-24GV2 (_TZE204_7gclukjs)) and 5.8 (ZY-M100-S_2 (_TZE204_qasjif9e) ceiling presence sensors using a ST-LINK V2 Clone and OpenOCD

I did come across some slight variation in pcb across my devices (Loginovo Purchased) The 5.8 was actually using the same backbone pcb as the 24G - ZY-V230605-24G-1.3, just different sensor and I assume firmware. The 24G also differed between V1.2 and 1.3. (see pictures) The zigbee module was the ZT3L - https://developer.tuya.com/en/docs/iot/zt3l-module-datasheet?id=Ka438n1j8nuvu ZT3L has slower clock rate 48 MHz vs 80 MHz, but more flash memory 1024 KB vs 768 KB

The V1.3 has the SWCLK and SWDIO pads flipped compared to V1.2 IMG20240920004104 IMG20240920001617

jvdburgt commented 2 months ago

@tandarra :

Just curious, but I assume it was successfull? Could you disable distance reporting afterwards and did it result in eleiminating the spammy behavior?

Also: since an OTA update doesn;t seem to be fortcoming: how easy/difficult/risky is flashing the sensors this way for a noob?

Edit: and I see a lot of different options for J-Link (some >400 USD) as well as for the ST-Link v2. Can you link to the products you ordered please?

tandarra commented 2 months ago

Yes it was successful. This new firmware from @Andrik45719 has the distance reporting tuya data point disabled. So it no longer works from what I can see.

I do have a background as an industrial electrician/ instrumentation and control. So I do understand alittle and might not be considered a noob.

However the finer electronics / programming side is abit above me, but the GitHub has all the scripts and information needed.

The hardest part for me was getting OpenOCD installed, I had to manually add some directory folders for Msys2 as it wouldn't get past 50%

Took less than 5 minutes each and that's including soldering the jumper pins to flash it.

If you're stuck on something in particular I'll be happy to attempt to help where I can.

I just brought a ST-Link V2 USB clone from a local store for $8 NZD they go for about $3 on AliExpress https://a.aliexpress.com/_mMHciAs

java-devil commented 2 months ago

Eh. So this is how my ceiling sensors look like: image image

And this is how the non-ceiling ones look like: image

I'm no electrician and have no experience with micro-controllers. I am however, a software engineer by day.

TBH having looked at the linked repo it is completely not obvious to me how to parametrize the flashing commands to my specific hardware. Without your post @tandarra I wouldn't be even aware that I probably should. Not to mention that I am a noob of such epic proportions that I wasn't even ware of the need to solder the pins.

Am I doomed to forever suffer a saturated ZigBee network?

java-devil commented 2 months ago

And speaking of The repo provided by @Andrik45719. Where do the binaries come from anyway? I assume they were somehow intercepted from the OTA pushed directly onto his device by Wzwenzhi?

If so are we able to somehow reuse these in the Z2M OTA process @Koenkk?

Koenkk commented 2 months ago

@java-devil as you can see here, these files need to be flashed via JTAG, so they are not Zigbee OTA files. To get the OTA url see this guide

java-devil commented 2 months ago

@Koenkk this seems... pretty easily doable on paper. All I need to do is buy the Tuya bridge. Which begs the question Guys - what am I not seeing? Surely somebody would have done this by now if it was this simple? Is the requirement that Wzwenzhi somehow actively "push" the update onto your device what is screwing us over here?

kenni commented 2 months ago

@java-devil I would assume that we are talking about two different approaches here.

The Github repository uses the terminology "modified firmware", leading me to suspect that the original firmware was pulled from a device and then the ~3 specific values were modified through reverse engineering. Flashing the modified firmware back to the device changes the ~3 specific values.

I would not assume that this has anything to do with a semi-official firmware pushed from a reseller, which you can intercept using a proxy and download. But @Andrik45719 would be the right person to answer this :)

If I'm right in my assumption, the real question is if it is possible to convert a raw/modified firmware dump into an OTA update or if these has to be signed, etc. - I currently have no knowledge of this, but it could seem like image.py from zigpy (https://github.com/zigpy/zigpy/blob/dev/zigpy/ota/image.py) might be able to convert a raw image into a valid OTA image if you can get the various arguments right.

Some pointers: https://github.com/espressif/esp-zigbee-sdk/issues/146

buhito81 commented 2 months ago

@Koenkk this seems... pretty easily doable on paper. All I need to do is buy the Tuya bridge. Which begs the question Guys - what am I not seeing? Surely somebody would have done this by now if it was this simple? Is the requirement that Wzwenzhi somehow actively "push" the update onto your device what is screwing us over here?

I tried that, today, after ordering one. See above: https://github.com/Koenkk/zigbee2mqtt/issues/19045#issuecomment-2360820055

It doesn't work with these type of device on the latest tested version of the apk, as referenced.

jgarridoalcazar commented 2 months ago

Beyond if a OTA is doable or not, flashing is really simple. You don't really need to solder the pins. If you use a male pinheader and press it lightly with your hand while it flashes. It only takes few seconds to flash.

I have already flashed tens of these devices with no issues.

El jue, 19 sept 2024, 20:02, java-devil @.***> escribió:

Eh. So this is how my ceiling sensors look like: image.png (view on web) https://github.com/user-attachments/assets/1f4fd787-3188-41d3-bd59-7b778658de04 image.png (view on web) https://github.com/user-attachments/assets/3cbedcf2-6cb0-4d7c-b27b-a587e3128261

And this is how the non-ceiling ones look like: image.png (view on web) https://github.com/user-attachments/assets/d25c3c8e-53f1-4480-baff-177b0efd1cf8

I'm no electrician and have no experience with micro-controllers. I am however, a software engineer by day.

TBH having looked at the linked repo it is completely not obvious to me how to parametrize the flashing commands to my specific hardware. Without your post I wouldn't be even aware that I probably should. Not to mention that I am a noob of such epic proportions that I wasn't even ware of the need to solder the pins.

Am I doomed to forever suffer a saturated ZigBee network?

— Reply to this email directly, view it on GitHub https://github.com/Koenkk/zigbee2mqtt/issues/19045#issuecomment-2361845490, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABBRDARNKN3JDNDBSAIZ6FLZXMGSNAVCNFSM6AAAAAA5CFPYD2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRRHA2DKNBZGA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

kenni commented 2 months ago

Beyond if a OTA is doable or not, flashing is really simple. You don't really need to solder the pins. If you use a male pinheader and ...

Sure, but as the device currently can bring your entire Z2M Zigbee network to its knees, an OTA in the Z2M repository would be very much preferred over each individual Z2M user needing to take their devices apart and connect wires to manually flash a new firmware.

java-devil commented 2 months ago

@buhito81 The mirror you are using seems to be massively out of date. The current version is 5.17.1 and it is supposingly readily available on other mirrors. I don't know which of these can be trusted in the Android world (I can ask around in work tomorrow if you wish), but then again - you will be running them in the walled environment of an emulator - so it's probably worth a shot.

@jgarridoalcazar I don't have physical access to the backplate of 5 out of my 8 sensors. And besides - the real win for the entire community is the OTA.

buhito81 commented 2 months ago

@buhito81 The mirror you are using seems to be massively out of date. The current version is 5.17.1 and it is supposingly readily available on other mirrors. I don't know which of these can be trusted in the Android world (I can ask around in work tomorrow if you wish), but then again - you will be running them in the walled environment of an emulator - so it's probably worth a shot.

I know! But it's the massively out of date version that is the latest version that the Zigbee2mqtt page references as working, rather specifically!

kenni commented 2 months ago

I know! But it's the massively out of date version that is the latest version that the Zigbee2mqtt page references as working, rather specifically!

It would be interesting to hear what doesn't work, if you try the latest available version of the app. It could be that they have implemented a variant of SSL pinning, causing your interception of traffic to fail. If that is the case, it should be doable to circumvent the SSL pinning.

java-devil commented 2 months ago

Unfortunately git balme doesn't return any useful context for that version 3.12.6 disclaimer (you wouldn't happen to remember, would you @Koenkk? please find it in your heart to forgive me for tagging you so extensively in this thread).

So I guess... I would simply try with the newest version @buhito81 and come back to us with observations on how specifically it breaks. Odds are we might be able to help. And perhaps update the doc while we are at it.

Andrik45719 commented 2 months ago

Tuya

@java-devil as you can see here, these files need to be flashed via JTAG, so they are not Zigbee OTA files. To get the OTA url see this guide

This method works only on Zigbee module update.

To update Tuya MCU we have to implement this algorithm https://developer.tuya.com/en/docs/iot/tuya-zigbee-universal-docking-access-standard?id=K9ik6zvofpzql Cluster 0xef00 Commands: TUYA_MCU_VERSION_REQ TUYA_MCU_OTA_NOTIFY TUYA_OTA_BLOCK_DATA_RSP

As I see dumped firmware support Tuya MCU update protocol.

Andrik45719 commented 2 months ago

The Github repository uses the terminology "modified firmware", leading me to suspect that the original firmware was pulled from a device and then the ~3 specific values were modified through reverse engineering. Flashing the modified firmware back to the device changes the ~3 specific values.

You right, firmware dumped, reversed and patched to prevent spamming TargetDistance DP

Andrik45719 commented 2 months ago

ztc6ggyl 7A 74 63 36 67 67 79 6C 122 116 99 54 103 103 121 108

0x3663747A 912487546 0x6C796767 1819895655

mcu_ota_notify_event

[2024-09-08 14:39:03] debug: zh:controller:endpoint: ZCL command 0xdc8e95fffe4dc368/1 manuSpecificTuya.mcuOtaNotify({"crc":0,"imageSize":200,"key_hi":912487546,"key_lo":1819895655,"seq":1,"version":67}, {"timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"reservedBits":0,"writeUndiv":false}) [2024-09-08 14:39:03] debug: zh:zstack: sendZclFrameToEndpointInternal 0xdc8e95fffe4dc368:12450/1 (0,0,1) [2024-09-08 14:39:03] debug: zh:zstack:znp: SREQ: --> AF - dataRequest - {"dstaddr":12450,"destendpoint":1,"srcendpoint":1,"clusterid":61184,"transid":199,"options":0,"radius":30,"len":22,"data":{"type":"Buffer","data":[1,101,18,1,0,122,116,99,54,103,103,121,108,67,200,0,0,0,0,0,0,0]}} [2024-09-08 14:39:03] debug: zh:zstack:unpi:writer: --> frame [254,32,36,1,162,48,1,1,0,239,199,0,30,22,1,101,18,1,0,122,116,99,54,103,103,121,108,67,200,0,0,0,0,0,0,0,5]

mcu_ota_result_report(0) [2024-09-08 14:39:03] debug: zh:zstack:unpi:parser: <-- [254,25,68,129,0,0,0,239,162,48,1,1,0,153,0,121,245,154,0,0,5,8,101,11,18,0,162,48,29,208] [2024-09-08 14:39:03] debug: zh:zstack:unpi:parser: --- parseNext [254,25,68,129,0,0,0,239,162,48,1,1,0,153,0,121,245,154,0,0,5,8,101,11,18,0,162,48,29,208] [2024-09-08 14:39:03] debug: zh:zstack:unpi:parser: --> parsed 25 - 2 - 4 - 129 - [0,0,0,239,162,48,1,1,0,153,0,121,245,154,0,0,5,8,101,11,18,0,162,48,29] - 208 [2024-09-08 14:39:03] debug: zh:zstack:znp: AREQ: <-- AF - incomingMsg - {"groupid":0,"clusterid":61184,"srcaddr":12450,"srcendpoint":1,"dstendpoint":1,"wasbroadcast":0,"linkquality":153,"securityuse":0,"timestamp":10155385,"transseqnumber":0,"len":5,"data":{"type":"Buffer","data":[8,101,11,18,0]}} [2024-09-08 14:39:03] debug: zh:controller: Received payload: clusterID=61184, address=12450, groupID=0, endpoint=1, destinationEndpoint=1, wasBroadcast=false, linkQuality=153, frame={"header":{"frameControl":{"frameType":0,"manufacturerSpecific":false,"direction":1,"disableDefaultResponse":false,"reservedBits":0},"transactionSequenceNumber":101,"commandIdentifier":11},"payload":{"cmdId":18,"statusCode":0},"command":{"ID":11,"name":"defaultRsp","parameters":[{"name":"cmdId","type":32},{"name":"statusCode","type":32}]}} [2024-09-08 14:39:03] info: z2m:mqtt: MQTT publish: topic 'zigbee2mqtt_floor1/coatroom_mw', payload '{"detection_delay":0,"fading_time":0.5,"illuminance_lux":79,"last_seen":"2024-09-08T14:39:03+03:00","linkquality":153,"maximum_range":3,"minimum_range":0,"presence":false,"radar_sensitivity":5,"self_test":null,"target_distance":null}' [2024-09-08 14:39:03] debug: zh:zstack:unpi:parser: --- parseNext [] [2024-09-08 14:39:03] info: zhc:tz: Invoked '61184.18' with payload '{"crc":0,"imageSize":200,"key_hi":912487546,"key_lo":1819895655,"seq":1,"version":67}'

mcu_ota_fw_request() [2024-09-08 14:39:05] debug: zh:zstack:unpi:parser: <-- [254,39,68,129,0,0,0,239,162,48,1,1,0,156,0,155,213,156,0,0,19,9,75,19,0,84,122,116,99,54,103,103,121,108,67,0,0,0,0,48,162,48,29,117] [2024-09-08 14:39:05] debug: zh:zstack:unpi:parser: --- parseNext [254,39,68,129,0,0,0,239,162,48,1,1,0,156,0,155,213,156,0,0,19,9,75,19,0,84,122,116,99,54,103,103,121,108,67,0,0,0,0,48,162,48,29,117] [2024-09-08 14:39:05] debug: zh:zstack:unpi:parser: --> parsed 39 - 2 - 4 - 129 - [0,0,0,239,162,48,1,1,0,156,0,155,213,156,0,0,19,9,75,19,0,84,122,116,99,54,103,103,121,108,67,0,0,0,0,48,162,48,29] - 117 [2024-09-08 14:39:05] debug: zh:zstack:znp: AREQ: <-- AF - incomingMsg - {"groupid":0,"clusterid":61184,"srcaddr":12450,"srcendpoint":1,"dstendpoint":1,"wasbroadcast":0,"linkquality":156,"securityuse":0,"timestamp":10278299,"transseqnumber":0,"len":19,"data":{"type":"Buffer","data":[9,75,19,0,84,122,116,99,54,103,103,121,108,67,0,0,0,0,48]}} [2024-09-08 14:39:05] debug: zh:controller: Failed to parse frame: RangeError [ERR_OUT_OF_RANGE]: The value of "offset" is out of range. It must be >= 0 and <= 15. Received 18 [2024-09-08 14:39:05] debug: zh:controller: Received payload: clusterID=61184, address=12450, groupID=0, endpoint=1, destinationEndpoint=1, wasBroadcast=false, linkQuality=156, frame=undefined [2024-09-08 14:39:05] debug: zh:zstack:unpi:parser: --- parseNext [] [2024-09-08 14:39:05] debug: z2m: Received Zigbee message from 'coatroom_mw', type 'raw', cluster 'manuSpecificTuya', data '{"data":[9,75,19,0,84, 122,116,99,54,103,103,121,108, 67,0,0,0,0,48],"type":"Buffer"}' from endpoint 1 with groupID 0 [2024-09-08 14:39:05] debug: z2m: No converter available for 'ZY-M100-L' with cluster 'manuSpecificTuya' and type 'raw' and data '{"data":[9,75,19,0,84,122,116,99,54,103,103,121,108,67,0,0,0,0,48],"type":"Buffer"}'

mcu_ota_result_report(1) [2024-09-08 14:39:15] debug: zh:zstack:unpi:parser: <-- [254,35,68,129,0,0,0,239,162,48,1,1,0,153,0,182,94,166,0,0,15,9,80,21,0,89,1,122,116,99,54,103,103,121,108,67,162,48,29,213] [2024-09-08 14:39:15] debug: zh:zstack:unpi:parser: --- parseNext [254,35,68,129,0,0,0,239,162,48,1,1,0,153,0,182,94,166,0,0,15,9,80,21,0,89,1,122,116,99,54,103,103,121,108,67,162,48,29,213] [2024-09-08 14:39:15] debug: zh:zstack:unpi:parser: --> parsed 35 - 2 - 4 - 129 - [0,0,0,239,162,48,1,1,0,153,0,182,94,166,0,0,15,9,80,21,0,89,1,122,116,99,54,103,103,121,108,67,162,48,29] - 213 [2024-09-08 14:39:15] debug: zh:zstack:znp: AREQ: <-- AF - incomingMsg - {"groupid":0,"clusterid":61184,"srcaddr":12450,"srcendpoint":1,"dstendpoint":1,"wasbroadcast":0,"linkquality":153,"securityuse":0,"timestamp":10903222,"transseqnumber":0,"len":15,"data":{"type":"Buffer","data":[9,80,21,0,89,1,122,116,99,54,103,103,121,108,67]}} [2024-09-08 14:39:15] debug: zh:controller: Received payload: clusterID=61184, address=12450, groupID=0, endpoint=1, destinationEndpoint=1, wasBroadcast=false, linkQuality=153, frame={"header":{"frameControl":{"frameType":1,"manufacturerSpecific":false,"direction":1,"disableDefaultResponse":false,"reservedBits":0},"transactionSequenceNumber":80,"commandIdentifier":21},"payload":{"seq":22784,"status":1,"key_hi":912487546,"key_lo":1819895655,"version":67},"command":{"ID":21,"parameters":[{"name":"seq","type":33},{"name":"status","type":32},{"name":"key_hi","type":35},{"name":"key_lo","type":35},{"name":"version","type":32}],"name":"mcuOtaResult"}} [2024-09-08 14:39:15] debug: zh:controller:endpoint: ZCL command 0xdc8e95fffe4dc368/1 manuSpecificTuya.defaultRsp({"cmdId":21,"statusCode":0}, {"timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":true,"direction":0,"reservedBits":0,"transactionSequenceNumber":80,"writeUndiv":false}) [2024-09-08 14:39:15] debug: zh:zstack: sendZclFrameToEndpointInternal 0xdc8e95fffe4dc368:12450/1 (0,0,1) [2024-09-08 14:39:15] debug: zh:zstack:znp: SREQ: --> AF - dataRequest - {"dstaddr":12450,"destendpoint":1,"srcendpoint":1,"clusterid":61184,"transid":202,"options":0,"radius":30,"len":5,"data":{"type":"Buffer","data":[16,80,11,21,0]}} [2024-09-08 14:39:15] debug: zh:zstack:unpi:writer: --> frame [254,15,36,1,162,48,1,1,0,239,202,0,30,5,16,80,11,21,0,216] [2024-09-08 14:39:15] debug: zh:zstack:unpi:parser: --- parseNext [] [2024-09-08 14:39:15] debug: z2m: Received Zigbee message from 'coatroom_mw', type 'commandMcuOtaResult', cluster 'manuSpecificTuya', data '{"key_hi":912487546,"key_lo":1819895655,"seq":22784,"status":1,"version":67}' from endpoint 1 with groupID 0 [2024-09-08 14:39:15] debug: z2m: No converter available for 'ZY-M100-L' with cluster 'manuSpecificTuya' and type 'commandMcuOtaResult' and data '{"key_hi":912487546,"key_lo":1819895655,"seq":22784,"status":1,"version":67}'

tandarra commented 2 months ago

@java-devil I'll try my best to run through the steps I used. Unsure if everything was needed but this is how I got my one to work.

First connect them to zigbee2mqtt to see what model you have to confirm which firmware you need

zigb2mqtt

Follow @Andrik45719 github guide - https://github.com/Andrik45719/ZY-M100

Mods:

Suppoted models: as of 20/09/24

I assume the binaries first came from Andrik backing them up, reading them and modifying them to suit, other models possibly have been assisted from other members such as @jgarridoalcazar submitting them also, I don't want to assume what is going on, but I certainly appreciate everyone's input to get us here so far. https://github.com/Andrik45719/ZY-M100/issues/4

four folders

msys2-openocd

zadig

libusb

IMG202409200016491

yhiub1z1

7stymrc5

backup

backup1

userbin

flashfi

As @jgarridoalcazar mentioned, I just did a spare device whilst writing this out and didn't solder the pin to the pads just had the jumper pins sitting there under its own tension and worked, super easy.

You mean you don't have physical access to the devices as they are still in the ceiling? Just turn the breaker off to the power and remove them, honestly procrastinating and waiting for an OTA fix will take longer than doing. It seems daunting but should be fairly straightforward now you can follow along.

Mine were installed in the ceiling also so I can understand the frustration and you're right an OTA update would be amazing; I just didn't have the patience to wait while having so many issues on my zigbee network

I encourage everyone to critic my steps, more than happy to learn a better way of doing something and update the post to make it easier for the next person.

Andrik45719 commented 2 months ago

Tuya

@java-devil as you can see here, these files need to be flashed via JTAG, so they are not Zigbee OTA files. To get the OTA url see this guide

This method works only on Zigbee module update.

To update Tuya MCU we have to implement this algorithm https://developer.tuya.com/en/docs/iot/tuya-zigbee-universal-docking-access-standard?id=K9ik6zvofpzql Cluster 0xef00 Commands: TUYA_MCU_VERSION_REQ TUYA_MCU_OTA_NOTIFY TUYA_OTA_BLOCK_DATA_RSP

As I see dumped firmware support Tuya MCU update protocol.

https://github.com/Andrik45719/ZY-M100/blob/main/TuyaMCU_OTA.md

java-devil commented 2 months ago

@tandarra amzaing work. I'm certain that you personally saved the ZigBee networks of multiple people.

As for myself... Unfortunately my presence sensors are painted over with the same coat of paint as the rest of my ceiling (as I didn't much care for the light sensors). So now I would need to pry them with a knife. At best chipping away the paint, but also risking damaging the ceiling itself.

I have amazing Wife Approval Factor. She is generally very supportive of my home automation work. But for this she would murder me in my sleep ;)

Still. I can flash 3 out of 8 thanks to your excellent summary. That's not nothing ;)

@Andrik45719 I'm embarrassed by how little I'm able to discern from the linked markdown and the debug logs. Do I understand correctly that your recent work boils down to proving that it is possible to bump the firmware OTA (via the MCU protocol)? Is there a practical use of this information from where we are now thanks to your work (if @tandarra's is amazing than I do not have any words for yours) on the reverse-engineered firmware?

mremedi2023 commented 2 months ago

Hi, same problem here, these sensors flood the data buffer causing me to have to restart my coordinator every 2 days, I'm about to get rid of them. Sensor: Tuya MTG075-ZB-RL