search

ct-Open-Source / tuya-convert

A collection of scripts to flash Tuya IoT devices to alternative firmwares
MIT License
4.6k stars 496 forks source link

Support for the new sdk #273

Closed M4dmartig4n closed 5 years ago

M4dmartig4n commented 5 years ago

Hi all,

I did succeed in flashing a device with the new tuya sdk with tuya-convert. Changes needed are in my own fork, for reference

https://github.com/M4dmartig4n/tuya-convert

It is not ready for a pull-request, since it drops support for older sdk atm.

Enjoy !

EverythingSmartHome commented 5 years ago

Hey, are you saying this works for new firmware that was previously not compatible?

M4dmartig4n commented 5 years ago

Yes. Obviously, I haven't tested it against all new firmware, so your mileage may vary.

cr0ky commented 5 years ago

I hope it will work for the latest tuya devices, bought some door sensors and stuff that would be handy to flash like this.

naxxxe commented 5 years ago

I just tried this on a Nedis WIFIP310FWT that couldn't be flashed using the old method. I can confirm that this worked very well

EverythingSmartHome commented 5 years ago

Amazing job well done! Care to give a brief overview of how you figured it out? Just to satisfy my curiosity, I'm really interested! Thanks!

Chaotic commented 5 years ago

I'll be checking this out tonight got 4 gosunds with the new firmware on them I'll report back if they work.

cr0ky commented 5 years ago

So just use the build from CT? And import the recent files from you?

kueblc commented 5 years ago

@M4dmartig4n Go ahead and open a PR and we can work together to get this merged. Will require a few small tweaks to be compatible with #226. You can reach me directly at my gmail of the same name.

M4dmartig4n commented 5 years ago

@kueblc I will be afk for a moment, but will do that.

Apart from my coding style ;) , we need to figure a way to identify devices with previous/new sdk in order to provide them the right answers (for example, mqtt messages, and upgrade answers are quite different).

Maybe there is a parameter in the request, I dunno.

A "hack" would be to check the source ip in fake_registration server, since new sdk requests are proxyfied through psk-frontend and thus come from 127.0.0.1.

M4dmartig4n commented 5 years ago

@EverythingSmartHome I will probably write a small blog post about it in the near future

kueblc commented 5 years ago

@kueblc I will be afk for a moment, but will do that.

Do you have Telegram or Signal?

Apart from my coding style ;) , we need to figure a way to identify devices with previous/new sdk in order to provide them the right answers (for example, mqtt messages, and upgrade answers are quite different).

Maybe there is a parameter in the request, I dunno.

Sure, check for et=1

A "hack" would be to check the source ip in fake_registration server, since new sdk requests are proxyfied through psk-frontend and thus come from 127.0.0.1.

This would work fine too, this entire project is a "hack" :stuck_out_tongue:

Tollbringer commented 5 years ago

Thanks for the efforts. I was unable to install as the pre-req libssl-dev seemed to be missing/absent. I tried a Kali-linux environment, along with my own Proxmox/Debian LXC container script previously working well for Tuya 1.0.

I fixed the error, and was able to proceed.

I attempted conversion with Teckin Sp20 outlet with no success. I can see it finds the device and talks to it, and I would say it engages stage one - but never actually backs up or proceeds. It does the leave the device crippled/bricked and unable to proceed. (that is ok it is a test unit and has serial connections ready to re-flash back to stock). I am able to re-flash back to stock firmware which DOES work, and I am able to add it to the APP and it detects v1.0.3 and updates it, blocking Tuya Convert v1 -- as expected.

I have another AliExpress plug that doesn't have a firmware update for it, and therefor I can't test with.

[-=SrZ=-]

kueblc commented 5 years ago

@SirRedZ can you elaborate on what you mean by "bricked"? You can't put it back into EZ config mode after attempting the process?

Janghou commented 5 years ago

Another missing prerequisite is git ;)

Tollbringer commented 5 years ago

@SirRedZ can you elaborate on what you mean by "bricked"? You can't put it back into EZ config mode after attempting the process?

Correct. The module's behavior is modified, and button access is unresponsive. Unplugging and plugging in flashes the light once, but that is it. I recognize this behavior after the bootloader has been applied??? mid-state? Only serial flashing allows me to recover.

kueblc commented 5 years ago

Since you already have it connected to serial, it would be great to get a backup of the firmware in this state. Did you back it up before reflashing?

ThaStealth commented 5 years ago

Along with the missing libssl-dev pre-req which was already mentioned I noticed another issue with my Smart Connect bulbs.

After connecting it is posting to the following URL a couple of times (and failing):

POST /d.json?a=tuya.device.config.get&et=1&other={"token":"00000000","region":"US"}&t=14&uuid=some UUID&v=4.1&sign=some hex string

It seems to be asking for some sort of configuration, but I'm not sure which

haubke commented 5 years ago

i'm trying for an hour. i got the smart bulb in config mode. tuya convert is searching. but no luck. do i need to start pairing this device thrue my phone?if yes wich app do you use?

ThaStealth commented 5 years ago

@haubke,

You only need to connect to the WiFi with your phone, no apps or anything else

If it doesn’t switch to continuous and keeps on flashing, repeat the process. It took me 10 times for it to finally to connect

haubke commented 5 years ago

yes flashing 2 times a sec. no it wont stop flashing. i will check logs when i tried a few times. i will post logs in a minute. I checked the logs, nothing usefull there. Everything seems ok. i will dig deeper.

i guess i found a problem. log PSK Traceback (most recent call last): File "./psk-frontend.py", line 8, in from Crypto.Cipher import AES ImportError: No module named 'Crypto'

pjv commented 5 years ago

sudo apt-get install libssl-dev

...and then boom, it worked. Saved me from having to try to solder pins that are about 1/8 away from a tall plastic switch soldered to the pcb.

THANK YOU!!!

haubke commented 5 years ago

still no luck, thanks anyhow. i will keep trying and post my findings.

smadhukarHacs commented 5 years ago

Hi All, seeing this in the smarthack-web.log with the new sdk setup

URI:/gw.json?a=tuya.device.upgrade.silent.get&gwId=63120466c44f33863260&t=26&v=4.1&sign=31d615cf31fbc6a63e0031d0685a06c7 Answer s.gw.upgrade.get b'{"result":{"auto":3,"size":"506587","type":9,"pskUrl":"https://10.42.42.1/files/upgrade.bin","hmac":"7BA1C83AD95ED48CBAB9A3BA731B79DE1CFC61DEDC12CE84BBD480F49833F5F9","version":"9.0.0"},"t":10,"e":false,"success":true}' [I 190919 17:29:55 web:2246] 200 POST /gw.json?a=tuya.device.upgrade.silent.get&gwId=63120466c44f33863260&t=26&v=4.1&sign=31d615cf31fbc6a63e0031d0685a06c7 (10.42.42.29) 24.65ms

URI:/gw.json?a=s.gw.dev.timer.count&gwId=63120466c44f33863260&t=167&sign=28d676260773c01b4dbfba9d8a94d556 WARN: unknown request: s.gw.dev.timer.count (/gw.json?a=s.gw.dev.timer.count&gwId=63120466c44f33863260&t=167&sign=28d676260773c01b4dbfba9d8a94d556) [I 190919 17:32:32 web:2246] 200 POST /gw.json?a=s.gw.dev.timer.count&gwId=63120466c44f33863260&t=167&sign=28d676260773c01b4dbfba9d8a94d556 (10.42.42.29) 5.93ms

The last line keeps repeating

kueblc commented 5 years ago

This should be resolved when we merge this with #226

Tollbringer commented 5 years ago

Colin, do you still want a copy of firmware in the 'frozen state' ?

Tollbringer commented 5 years ago

I thought after reading the comments above I should check my logs.... Maybe it was working and crashed for me?>

URI:/gw.json?a=tuya.device.upgrade.silent.get&gwId=88020554cc50e3cf1954&t=7&v=4.1&sign=fab950f88c43806fa17e9bcca40bdf34 Answer s.gw.upgrade.get b'{"result":{"auto":3,"size":"506587","type":9,"pskUrl":"https://10.42.42.1/files/upgrade.bin","hmac":"7BA1C83AD95ED48CBAB9A3BA731B79DE1CFC61DEDC12CE84BBD480F$ ^[[32m[I 190919 07:53:07 web:2246]^[[m^O 200 POST /gw.json?a=tuya.device.upgrade.silent.get&gwId=88020554cc50e3cf1954&t=7&v=4.1&sign=fab950f88c43806fa17e9bcca$ ^CTraceback (most recent call last): File "./fake-registration-server.py", line 160, in main() File "./fake-registration-server.py", line 156, in main tornado.ioloop.IOLoop.current().start() File "/usr/local/lib/python3.5/dist-packages/tornado/platform/asyncio.py", line 148, in start self.asyncio_loop.run_forever() File "/usr/lib/python3.5/asyncio/base_events.py", line 421, in run_forever self._run_once() File "/usr/lib/python3.5/asyncio/base_events.py", line 1388, in _run_once event_list = self._selector.select(timeout) File "/usr/lib/python3.5/selectors.py", line 445, in select fd_event_list = self._epoll.poll(timeout, max_ev) KeyboardInterrupt

kueblc commented 5 years ago

@SirRedZ Yes, I can use the backup to hopefully debug what went wrong. Doesn't look like the server crashed, it's a KeyboardInterrupt exception, which is what Ctrl-C throws (the ^C at the beginning of the error message).

smadhukarHacs commented 5 years ago

This should be resolved when we merge this with #226

Worked with the changes. Thanks guys for all the great work.

Tollbringer commented 5 years ago

teckin_sp20_afterfw_convertattempt.zip

Here is the FW in "bricked" state. Here are the exact steps.

  1. Module defaulted to original FW (rev =unknown - via serial backup made before touching)

  2. Module tested for pairing mode/operation via manual button.

  3. Module put in to pairing mode, 6 seconds on power button.

  4. Device added to Tuya app per normal use.

  5. New FW update v1.0.3 identified, accepted and module upgraded.

  6. Confirm normal operation of device.

  7. Remove device from APP

  8. Attempt Tuya convert 2.0 process.

  9. Connect mobile device to vtrust-flash

  10. Device in pairing mode as a result of APP removal - GO ..... endless dots, connection made to module lights stop flashing, never flash again. process aborted, - UNPLUG module

  11. Restart Tuya-Convert process, reconnect to Vtrust-flash, start process and then plugin module F/W backup actually starts - it tries, no data received. Enter a few times, multiple lines showing no data and speed of transfer declining.

  12. Restart process, Vtrust-recovery visible, module reports flashed with userspace1 please flash with userspace 2 (after an attempt to flash 3rd party firmware)

  13. Flashed Userspace 2- rebooted.

  14. Restarted flashing process, unplugged module and plugged back in. Mopdule reports online. Firmware backup fails again (expected - file exists?)

  15. JUMP - UP - AND - DOWN - SUCCESS !!!!!

Sonoff-xxx access point visible. IT WORKS. I had to jump through hoops - it works.

This was the device I originally used to report the failure- block also!

GREAT WORK!.

[-=SrZ=-]

Tollbringer commented 5 years ago

My current conclusion is brute force (unplug/plugin/restart tuya convert process, etc...) gets it to go eventually.

Lightbulbs are a No-go after trying 4-5 different bulbs Zero response. I did my test module 4 times andDigiBlurDIY did 2 or 3 different devices and all eventually converted. There were all previously confirmed BLOCKED in the past. All were added to the app to make sure they had latest firmware, and then removed from the app to begin the conversion.

Thanks everyone- I hope this helps!

[-=SrZ=-]

EverythingSmartHome commented 5 years ago

Unfortunately this didn't work for my light bulbs either, great we are moving forward with this though!

Janghou commented 5 years ago

Happy to report: I does work for my Action LSC Smart bulb.

====================================================== Getting Info from IoT-device VTRUST-FLASH 1.1 (c) VTRUST GMBH https://www.vtrust.de/35c3/ READ FLASH: http://10.42.42.42/backup ChipID: XXXX MAC: XXXXX BootVersion: 7 BootMode: normal FlashMode: 1M DOUT @ 40MHz FlashChipId: 144051 FlashChipRealSize: 1024K Active Userspace: user2 0x81000

======================================================

ThaStealth commented 5 years ago

Happy to report: I does work for my Action LSC Smart bulb.

====================================================== Getting Info from IoT-device VTRUST-FLASH 1.1 (c) VTRUST GMBH https://www.vtrust.de/35c3/ READ FLASH: http://10.42.42.42/backup ChipID: XXXX MAC: XXXXX BootVersion: 7 BootMode: normal FlashMode: 1M DOUT @ 40MHz FlashChipId: 144051 FlashChipRealSize: 1024K Active Userspace: user2 0x81000

======================================================

That is strange, I have the same bulbs and they keep posting to the following URL a couple of times (and failing):

POST /d.json?a=tuya.device.config.get&et=1&other={"token":"00000000","region":"US"}&t=14&uuid=some UUID&v=4.1&sign=some hex string

Can you check your logs if you see that string coming by?

Janghou commented 5 years ago

I did a tweak to the HOSTAPD settings (./setup_ap.sh):

cat <<- EOF >/etc/hostapd/hostapd.conf
    interface=$WLAN
    driver=nl80211
    ssid=$AP
    hw_mode=g
    channel=9
    macaddr_acl=0
    auth_algs=1
    ignore_broadcast_ssid=0
    wpa=2
    wpa_passphrase=$PASS
    wpa_key_mgmt=WPA-PSK
    wpa_pairwise=TKIP
    rsn_pairwise=CCMP
    ieee80211n=1
EOF
ieee80211n=1 adds WIFI N channel support, and channel 9 is better then channel 1 I think.
IMHO WIFI N is working better then G.
Janghou commented 5 years ago

@ThaStealth, nope.

URI:/gw.json?a=s.gw.token.get&et=1&gwId=XX&other={"token":"00000000","region":"US","tlinkStat":{"configure":"smartconfig","time":1,"source":"ap","path":"broadcast"}}&t=7&v=3.0&sign=YY

URI:/gw.json?a=s.gw.token.get&et=1&gwId=XX&other={"token":"00000000","region":"US","tlinkStat":{"configure":"smartconfig","time":1,"source":"ap","path":"broadcast"}}&t=7&v=3.0&sign=YY
Answer s.gw.token.get
ESC[32m[I 190920 11:20:49 web:2246]ESC[m^O 200 GET /gw.json?a=s.gw.token.get&et=1&gwId=XX&other={"token":"00000000","region":"US","tlinkStat":{"configure":"smartconfig","time":1,"source":"ap","path":"broadcast"}}&t=7&v=3.0&sign=YY
8686 (127.0.0.1) 9.44ms

URI:/gw.json?a=s.gw.dev.fk.active&et=1&gwId=XX&other={"token":"00000000"}&t=7&v=3.0&sign=yy
92
Answer s.gw.dev.pk.active
READ GW ID xx
ESC[32m[I 190920 11:20:50 web:2246]ESC[m^O 200 POST /gw.json?a=s.gw.dev.fk.active&et=1&gwId=XX&other={"token":"00000000
"}&t=7&v=3.0&sign=YYY (127.0.0.1) 54.88ms

URI:/gw.json?a=tuya.device.dynamic.config.get&et=1&gwId=XX&t=9&v=1.0&sign=YY
Answer tuya.device.dynamic.config.get
READ GW ID XX
ESC[32m[I 190920 11:20:51 web:2246]ESC[m^O 200 POST /gw.json?a=tuya.device.dynamic.config.get&et=1&gwId=XX&t=9&v=1.0&si
gn=YY (127.0.0.1) 5.75ms

URI:/gw.json?a=tuya.device.dynamic.config.ack&et=1&gwId=XX&t=0&v=1.0&sign=YY
Answer tuya.device.dynamic.config.ack
READ GW ID XX
TRIGGER UPGRADE IN 10 SECONDS
Trigger upgrade in 10 seconds
ESC[32m[I 190920 11:20:52 web:2246]ESC[m^O 200 POST /gw.json?a=tuya.device.dynamic.config.ack&et=1&gwId=XX&t=0&v=1.0&sign=YY (127.0.0.1) 62.27ms
Janghou commented 5 years ago

Sonoff-Tasmota v6.5.0 also flashed successfully curl http://10.42.42.42/flash3

Can now switch the bulb on/off with simple requests: ;) curl http://192.168.0.xx/cm?cmnd=Power%20off curl http://192.168.0.xx/cm?cmnd=Power%20on

pandit77 commented 5 years ago

You´re the Män!!

Sucessfully flashed all my SP22 Adapters which all had new tuya FW before!

Thanks a lot!

robtrotter commented 5 years ago

Worked with two Brilliant Smart (Au) plugs. One of them needed the fixes in #226 and the other failed until I pulled those changes out. Really not sure on the difference.

Had some garden lights that seemed to get close.
The trigger_upgrade.sh ran - but then got a disconnect on the MQTT log: "1568993822: Socket error on client P1, disconnecting."

Tollbringer commented 5 years ago

Worked with two Brilliant Smart (Au) plugs. One of them needed the fixes in #226 and the other failed until I pulled those changes out. Really not sure on the difference.

Had some garden lights that seemed to get close. The trigger_upgrade.sh ran - but then got a disconnect on the MQTT log: "1568993822: Socket error on client P1, disconnecting."

I have experienced that also.

SkipperPit commented 5 years ago

It works! I just flashed 12 Teckin SP22 smart sockets. THANK YOU!

Janghou commented 5 years ago

Only Action LSC Smart White bulbs have success, Action Smart Connect RGB bulbs fail to flash here. They only sometimes stop flashing in the initial phase, but then a 404 shows up in ..web.log:


ESC[33m[W 190920 18:20:47 web:2246]ESC[m^O 404 POST /d.json?a=tuya.device.config.get&et=1&other={"token":"00000000","region":"US"}&t=21&u
uid=yy&v=4.1&sign=xx (127.0.0.1) 60.44ms
ESC[33m[W 190920 18:20:49 web:2246]ESC[m^O 404 POST /d.json?a=tuya.device.config.get&et=1&other={"token":"00000000","region":"US"}&t=21&u
uid=yy&v=4.1&sign=xx (127.0.0.1) 54.38ms

FYI Smart Connect RGB bulbs have ID's that are 2 characters longer then white filament bulbs, can that be the cause?

JoshuaSG commented 5 years ago

@haubke I had the same issue you had and was able to fix it by running the following two commands sudo apt-get install python3-dev pip3 install pycrypto

ThaStealth commented 5 years ago

@janghou

I have the issue, didn’t find a solution yet, however if you google tuya.device.config.get you’ll find a page from tuya itself which looks like an MQTT server info request

kueblc commented 5 years ago

@Janghou looks like you have a unique firmware there. It would be great to get a copy of it for research.

As far as your current issue, looks like it's making a request to /d.json instead of /gw.json. Duplicate this line and change one of the gw to d. Let us know how it goes.

ThaStealth commented 5 years ago

@kueblc I have the same bulb, changed the line to test it, but it still doesn't work:

URI:/d.json?a=tuya.device.config.get&et=1&other={"token":"00000000","region":"US"}&t=35&uuid=removed&v=4.1&sign=removed WARN: unknown request: tuya.device.config.get (/d.json?a=tuya.device.config.get&et=1&other={"token":"00000000","region":"US"}&t=35&uuid=removed&v=4.1&sign=removed) ^[[32m[I 190920 19:25:00 web:2246]^[[m^O 200 POST /d.json?a=tuya.device.config.get&et=1&other={"token":"00000000","region":"US"}&t=35&uuid=removed&v=4.1&sign=removed(127.0.0.1) 52.80ms

Note that "tuya.device.config.get" is not implemented in the JSONHandler

kueblc commented 5 years ago

@ThaStealth thanks for testing and following up. Try removing .dynamic from this line

Janghou commented 5 years ago

@ThaStealth @kueblc

URI:/d.json?a=tuya.device.config.get&et=1&other={"token":"00000000","region":"US"}&t=10&uuid=xxx&v=4.1&sign=yy
Answer tuya.device.dynamic.config.get
READ GW ID son?a=tuya.device.co
ESC[32m[I 190920 19:41:18 web:2246]ESC[m^O 200 POST /d.json?a=tuya.device.config.get&et=1&other={"token":"00000000","region":"US"}&t=10&uuid=xx&v=4.1&sign=yy (127.0.0.1) 58.63ms
kueblc commented 5 years ago

@Janghou so, did it work?

digiblur commented 5 years ago

I've tried about 3 or 4 bulb types with 2.0 and they don't work yet. One does this..

RI:/gw.json?a=tuya.device.dynamic.config.get&et=1&gwId=70007862b4e62d5ccf37&t=8&v=1.0&sign=7e4d82fbbf61fb1b0a786abd035778ad Answer tuya.device.dynamic.config.get READ GW ID 70007862b4e62d5ccf37 [I 190919 01:10:22 web:2246] 200 POST /gw.json?a=tuya.device.dynamic.config.get&et=1&gwId=70007862b4e62d5ccf37&t=8&v=1.0&sign=7e4d82fbbf61fb1b0a786abd035778ad (127.0.0.1) 5.47ms

URI:/gw.json?a=tuya.device.dynamic.config.ack&et=1&gwId=70007862b4e62d5ccf37&t=0&v=1.0&sign=9849fa1adcceb0159eaf9ec21b309081 Answer tuya.device.dynamic.config.ack TRIGGER UPGRADE IN 10 SECONDS Trigger upgrade in 10 seconds [I 190919 01:10:22 web:2246] 200 POST /gw.json?a=tuya.device.dynamic.config.ack&et=1&gwId=70007862b4e62d5ccf37&t=0&v=1.0&sign=9849fa1adcceb0159eaf9ec21b309081 (127.0.0.1) 60.66ms b'2.2\x8e\xf6\x14X85542492\xb7\x98\xa7\x08\xc0\xc7\xa5\xc2JB\x9c_9\xed\x91m\x8d\xb2~AK\xa9\xe2\xcf\xe7\x9a\xd8\xd8S\xf25P\xa89\x12\nX\xf1vdR\xd7c\n!\xed\x7fB\x91\xfa\xef\xd4\xa8\xffL\x1a\xef\x01\r\x8e\xbd\xf5EL(\x95\xab\x04\x9c\x1b\xd5\xae=\xaa\xe4\x02\xd5\x0bNl\xcd\xackG\xe0\xf9\x12\x8dT)O\xb0\x9b\x05u\xe6'

ThaStealth commented 5 years ago

@Janghou so, did it work?

I tried it and it repeats a couple of times but it doens't continue (got the same output @Janghou has)