search

codetheweb / tuyapi

🌧 An easy-to-use API for devices that use Tuya's cloud services. Documentation: https://codetheweb.github.io/tuyapi.
MIT License
2.06k stars 339 forks source link

Updating the firmware trough local API #49

Closed engelant closed 6 years ago

engelant commented 6 years ago

As I find it hard to find some information on if sombody managed to update the firmware of a tuya device utilizing any kind of OTA update, I thought this might be the right place to ask.

codetheweb commented 6 years ago

I haven't heard of anyone doing that, although I do know that at least one person has cracked their device open and directly flashed new firmware onto the board.

According to Tuya's docs (scroll/search for section 3.10), it may be possible.

If you manage to, let me know. 😃

drushbrook commented 5 years ago

I was fortunate enough to get a device that needed a firmware update and hence captured the traffic from my router between the device and the outside world. Here is a run down of what it did - I have tried to identify my private information by substituting for the device id and <32 bit AES key> for the signing key of the communication between the atop gateway and the device.

I have attempted to replicate the ota process by running a local mqtt and webserver and redirecting locally - I have got as far as it downloading the 64 byte header of the Sonoff-Tasmota firmware.

MQTT - App Triggered MQTT Publish via Cloud which seems to kick off the OTA update Topic: smart/device/in/ Message: 2.1cc684fc01148c5d8AtUW152t0tPFT2KItF9o+2iU4P+FRyQPpWiAlHQ0FJ+C1LFTkutfz2OboAvWd0ckWRauTKDdRG50K4uIOwyzITqiZgcdwgIVsiYuCks8w5G5V7+bGNnNy7lC3kV7GbQh Decoded: {"data":{"gwId":""},"protocol":15,"s":1523713,"t":1544007190}

HTTP: Post http://a.tuyaeu.com/gw.json?a=tuya.device.upgrade.get&gwId=&t=1544007189&v=4.0&sign=<32 bit aes key> data=99BB3ABB69C97BCA62CCC4B0D50C9EF4

Response: {"result":{"size":"825947","cdnUrl":"http://images.tuyaus.com/smart/firmware/upgrade/201810/1539875023-oem_esp_dltj_ug_1.1.0.bin","originalUrl":"http://s3-us-west-2.amazonaws.com/airtake-public-data/smart/firmware/upgrade/201810/1539875023-oem_esp_dltj_ug_1.1.0.bin","httpsUrl":"https://s3-us-west-2.amazonaws.com/airtake-public-data/smart/firmware/upgrade/201810/1539875023-oem_esp_dltj_ug_1.1.0.bin","version":"1.1.0","url":"http://s3-us-west-2.amazonaws.com/airtake-public-data/smart/firmware/upgrade/201810/1539875023-oem_esp_dltj_ug_1.1.0.bin","md5":"1142247a63a44dd97896c4fe6abd53f0"},"t":1544007191,"e":false,"success":true}

HTTP: Post: http://a.tuyaeu.com/gw.json?a=s.gw.upgrade.updatestatus&gwId=&t=1544007191&sign=<32 bit aes key> data=622AF6DE2E8048A841145B653500E9653CA71ED9200AD1CE8917A58AFA524E76

Response: {"t":1544007193,"e":false,"success":true}

HTTP: Header: Range: bytes=0-63 Get: http://s3-us-west-2.amazonaws.com/airtake-public-data/smart/firmware/upgrade/201810/1539875023-oem_esp_dltj_ug_1.1.0.bin

Partial Response: UªUª1.1.03M#gMGM#'\½ªUªUê @À

... must perform some validation of the first 64 bytes of the firmware probably looking for the version number and it contains the sizing and offset for the next request ... offset in hex: 00064D47 (dec: 412999) and size in hex: 00064D14 (825947 [file length] - 412999 [offset] = 412948 [section])

HTTP: Header: Range: bytes=412999-825946 Get: http://s3-us-west-2.amazonaws.com/airtake-public-data/smart/firmware/upgrade/201810/1539875023-oem_esp_dltj_ug_1.1.0.bin

Partial Response: : MQTT: Topic: smart/device/out/ Message: {"protocol":16,"data":{"gwId":"","devId":"","progress":"0"}}

MQTT: Topic: smart/device/out/ Message: {"protocol":16,"data":{"gwId":"","devId":"","progress":"1"}}

... a number of progress MQTT Publish...

MQTT: Topic: smart/device/out/ Message: {"protocol":16,"data":{"gwId":"","devId":"","progress":"97"}}

... tuya device did a reboot ...

DNS Query: mq.gw.tuyaeu.com

MQTT Connect: Client ID: Will Topic: tuya/smart/will Will Message: {"clientId":"","deviceType":"GATEWAY","message":"11","userName":""} User Name: Password: **

DNS Query: a.tuyaeu.com

HTTP: Post: http://a.tuyaeu.com/gw.json?a=s.gw.update&gwId=&t=9&v=2.0&sign=<32 bit aes key> Form item: "data" = "F45D3AD1A13B58AD14FA7CB50C9CC1467B1812FE0E6DDE0DAE2F46A5F4DCA05AF12517176211F09B175F683D3C63C6A0ECE6726FD7C37634DE32CDBCD741CCDAC399BC7B3D6F6654D554E70B07D26A329ECD2ACB649724A686E54AB97B0098FB"

Response: {"t":1544007229,"e":false,"success":true}

engelant commented 5 years ago

This seems to be exactly what I was looking for, so thanks. You didn't happen to dump the original bin file, did you? I'll have a closer look at this the next weeks, since I'm a little busy these days. afaik the signature is just a hash of some post vars from the request, salted with the device secret key. I'll have to look into my code from back in august, since unfortunatly a few months passed by and I don't remember everything. At first glance it looks like the bin is not encrypted, but X-mas is coming, so I'll find a few days to go into more detail I guess.

drushbrook commented 5 years ago

No I didn't dump the original bin file however I have a second device that is still needs to be updated. When I get time I will make an attempt - recently moved and need to find the soldering iron.

In the meantime I will continue to see if I can get the firmware header correct. Right now the device is downloading my header multiple times.

nijave commented 5 years ago

Here's an older version of the firmware http://s3-us-west-2.amazonaws.com/airtake-public-data/smart/firmware/upgrade/201806/1528276183-esp_bsd_jl_plug_America_ug_1.0.5.bin

I think mine came with 1.0.0 or 1.0.1, though

Zixim commented 5 years ago

anything new on this ?

SynAckFin commented 5 years ago

The 1.1.0 firmware linked above has a header that is 51 bytes long and is then followed by the actual firmware image. Esptool reports the following about the firmware:

Image version: 2
Entry point: 40100004
4 segments
Segment 1: len 0x5b9c0 load 0x00000000 file_offs 0x00000008
Segment 2: len 0x066ac load 0x40100000 file_offs 0x0005b9d8
Segment 3: len 0x004dc load 0x3ffe8000 file_offs 0x0006208c
Segment 4: len 0x02788 load 0x3ffe84e0 file_offs 0x00062570
Checksum: a5 (valid)

What might work for OTA updating is replacing the tuya image with a tasmota one. You might also have to update the firmware version number in the header (located at bytes 4 - 8).

SynAckFin commented 5 years ago

The bad news is I've managed to brick my plug by forcing it to install 1539875023-oem_esp_dltj_ug_1.1.0.bin firmware. The good news is that now I have an excuse to take it apart and try to fix it. I guess the firmware wasn't intended for my plug.

In other news, it turns out the firmware file actually has 2 different firmwares in it. Here is what I'm guessing the header contains:

55 aa 55 aa             - Prefix
31 2e 31 2e 30 00 00    - Version "1.1.0"
00 00 00 00             - Don't know (could be zero padding of version string)
00 00 00 02             - Number of firmwares in file
00 00 00 33             - File offset to first firmware
00 06 4d 14             - Length of first firmware
02 23 00 67             - Sum of all bytes in 1st firmware
00 06 4d 47             - File offset to second firmware
00 06 4d 14             - Length of second firmware
02 23 27 5c             - Sum of all bytes in 2nd firmware
00 00 05 bd             - Sum of all previous header bytes
aa 55 aa 55             - Suffix
SynAckFin commented 5 years ago

I can get the tuya plug to download and install the sonoff-basic firmware by constructing a binary using the above format header with 2 copies of the sonoff-basic appended. Unfortunately the plug no longer boots once its installed. I suspect that I'll have to get it to install some interim firmware and then get this to install sonoff-basic. I've posted in the SonoffUsers group and hopefully somebody will have some ideas on how to proceed.

Zixim commented 5 years ago

I'm on the edge-of-my-seat here 😃 Did you try the minimal version of the firmware ?

SynAckFin commented 5 years ago

The minimal version is the same. At the moment I'm looking at what SonOTA did to see if I can adapt that.

Zixim commented 5 years ago

fwiw, there is also an Espurna core version which is only 299 KB, created to simplify OTA updates. https://github.com/xoseperez/espurna/wiki/TwoStepUpdates

SynAckFin commented 5 years ago

SUCCESS I used the stage 2 firmware from SonOTA (image_user2-0x81000.bin) and when the plug rebooted it acted as an AP with an SSID of "FinalStage". A bit of messing around with setting my wifi to connect to that SSID and setting up a HTTP server to let it download sonoff-basic (renamed to image_arduino.bin) and I now have the sonoff-basic firmware on my plug.

Zixim commented 5 years ago

can you do a detailed write-up ? I'm aching to try this on one of the lightbulbs where I can not reach the serial pins without destroying the lamp.

SynAckFin commented 5 years ago

Will do.

Zixim commented 5 years ago

@SynAckFin , is this you ? https://community.home-assistant.io/t/cheap-uk-wifi-bulbs-with-tasmota-teardown-help-tywe3s/40508/6?u=tommmii

SynAckFin commented 5 years ago

WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING THIS IS EXPERIMENTAL, IT IS POSSIBLE THAT FOLLOWING THESE INSTRUCTIONS WILL TURN YOUR DEVICE INTO A PAPERWEIGHT WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING

Files tuya.zip

Instructions for Updating Tuya device OTA

I did this on a linux machine running openSUSE 15.0 but there is no reason why it shouldn't work on other flavours of linux and should be easily adaptable to windows.

Prerequisites

  1. A web server where you can store the 2 firmwares
  2. A computer with WiFi running some flavour of linux
  3. A method of redirecting HTTP requests to a computer under your control
  4. A computer to receive the redirected HTTP requests.

NOTE Although three computers are mentioned above everything can run on one computer provided that the web server is not listening to either port 80 or port 8080

Setting Everything up Web Server You need store the firmware tuya-sonoff.bin under a directory called ota. Download your target firmware and store it as ota/image_arduino.bin. DO NOT USE sonoff-minimal.bin I recommend using version 6.2.1 of sonoff.bin and if everything works you can then use its OTA mechanism install a different version.

Redirecting HTTP requests (the difficult bit) If you are in Europe then your Tuya device probably makes HTTP request to a.tuyaeu.com. If you are in another part of the world then it could be a.tuyacn.com or a.tuyaus.com. You should be able to find out which one it is by snooping port 80 traffic from your device. Once you know what it is then you need to redirect these request to one of your computers. I did this by using dnsmasq to return my address when the device did a DNS query for a.tuyaeu.com. An alternative would be to have a firewall redirect requests. Whatever method you use, the computer you are redirecting to must be able to lookup the correct address and access the remote site.

Running the Scripts There are 2 scripts, tuyaproxy.pl and finalstage.sh. In tuyaproxy.pl change the $WEBSERVER variable to point to the web server with the firmwares. In finalstage.sh change DEVICE to your WiFi device (usually wlan0) and WEBSERVER and PORT to your web server. tuyaproxy.pl needs to run as root on the computer you are redirecting the http requests to. finalstage.sh needs to run as root on the computer with WiFi

Installing the Firmware Once both scripts are running, http requests are being redirected and both firmwares are on the web server then power off your device, power it on and go and have a cup of coffee.

After everything has finished follow the sonoff getting started instructions.

Acknowledgements Thanks to Mirko Vogt for the original SonOTA. This uses the second stage firmware from that project.

drushbrook commented 5 years ago

Awesome. Will try it with my plugs and let you know. I will also try with my lightbulbs.

digiblur commented 5 years ago

This definitely changes things! Have a plug I can flash the Tuya junk back on it and test with.

Don't have a full blown Linux build going except my media server but I think a wired raspberry Pi should work with this right since it has Wi-Fi capabilities as well?

SynAckFin commented 5 years ago

There is no reason a Pi wouldn't work.

SynAckFin commented 5 years ago

@SynAckFin , is this you ? https://community.home-assistant.io/t/cheap-uk-wifi-bulbs-with-tasmota-teardown-help-tywe3s/40508/6?u=tommmii

No, that's not me

Zixim commented 5 years ago

In tuyaproxy.pl change the $WEBSERVER variable to point to the web server with the firmwares. Shouldn't the port be edited to whatever port the webserver is listening on ?

digiblur commented 5 years ago

I feel like I am close.. I saw one plug request firmware with the log from the pl script but on the webserver I am not seeing any requests for the bin file though. I did a wget from that Pi just to make sure the bin was accessible and that works but I just don't see the plug asking for that bin yet.

Zixim commented 5 years ago

@digiblur did you edit line nr 8 in tuyaproxy.pl ? my $WEBSERVER = "http://YourWebServer:8000/";

SynAckFin commented 5 years ago

If you add: print("URL: $Firmware->{result}{url}\n"); to the pl file just before the while statement (about line 29) then it will output the url it sends to the device. You can then check this is valid using wget on the Pi. Another thing you should check is that the Pi can reach the tuya site. Try: ping a.tuyaeu.com to check it can

digiblur commented 5 years ago

@SynAckFin so the Pi running the PL still needs to reach the real a.tuyaus.com? That would be my issue then as my whole network is redirecting.

digiblur commented 5 years ago

ahhh...I added that print and it was doing a ....server:8000// I removed the trailing / from the variable and the URL looks right now. Now to get it to request a firmware again. Patience...

SynAckFin commented 5 years ago

@SynAckFin so the Pi running the PL still needs to reach the real a.tuyaus.com? That would be my issue then as my whole network is redirecting.

Are you doing that with a firewall or dnsmasq?

digiblur commented 5 years ago

Have PiHole running along with a rule on my firewall that redirects any devices that don't listen to the DHCP DNS config to go through the PiHole for DNS. I can ping that address fine though.

Seeing these in the perl logs... trying to get a plug to ask for a firmware update but it just wants to do these. I'll keep messing with them. It seemed if I did another pairing session it would check for firmware pretty quickly but now it's being a pain.

/gw.json?a=s.gw.dev.pk.active&gwId=685010.......

Zixim commented 5 years ago

for me, the proxy script isn't feeding the firmware URL : 

user@debian:~/tuya$ sudo ./tuyaproxy.pl
URL: http://192.168.11.222:8000/ota/tuya-sonoff.bin
/gw.json?a=atop.online.debug.log&gwId=14801348b4e62d1d5c5e&t=10&sign=3bc1197785bc420d72139535906f0cd2
digiblur commented 5 years ago

I had one device hit twice with this at one time but I had the incorrect URL...now to get it to ask again.

/gw.json?a=tuya.device.upgrade.silent.get&gwId=03200.....

SynAckFin commented 5 years ago

Within about a minute of powering the device off and then on you should see

                Sending Firmware Response...
/gw.json?a=s.gw.upgrade.updatestatus&gwId=

The device only seems to send it one time after powering up. You may see other log entries before the upgrade one.

Zixim commented 5 years ago

@SynAckFin can you add more debug info to the proxy script ? such as the ip of the device that is requesting ?

Zixim commented 5 years ago

The device definately isn't looking for updates any more...perhaps only 1x per day ?

SynAckFin commented 5 years ago

To find tuya devices on your network run the following: nc -u -l -k -p 6666 | strings If strings isn't found (it isn't always installed) then just run: nc -u -l -k -p 6666 The output wont be as pretty but you can find out the IP addresses of the devices

I'm just re- flashing tuya onto one of my devices and once I've done that I'll add more output to the script.

Zixim commented 5 years ago

it's detected alright...it just doesn't want to ask for available update

user@debian:~$ sudo nc -u -l -k -p 6666 | strings
{"ip":"192.168.11.140","gwId":"14801348b4e62d1d5c5e","active":2,"ability":0,"mode":0,"encrypt":true,"productKey":"key4fv3sx8twchhy","version":"3.1"}Z
Zixim commented 5 years ago

errr, i just saw this in the proxy output : 

/gw.json?a=tuya.device.upgrade.silent.get&gwId=14801348b4e62d1d5c5e&t=40&v=4.1&sign=302b82d250cd057342b8175f3774ba85

but the proxy script didn't send the FW response ??

digiblur commented 5 years ago

Same here...just had one ask and saw the Sending firmware response... URL is showing correct though and works. It didn't pull the bin. Hmmm...

Zixim commented 5 years ago

either the device isn't getting the response, or it isn't happy with what it receives and ignores it.

digiblur commented 5 years ago

I noticed I'll see the tuya.device.upgrade.silent.get line but takes several minutes to post the Sending Firmware Response... line.

SynAckFin commented 5 years ago

I've just realised something here. If the url uses a hostname and not an IP then it needs to be fully qualified and resolvable by whatever dns server the device uses (specified in the DHCP response). For example: http://myserver/ota/tuya-sonoff.bin will fail even if other computers on your network can resolve it. On the other hand: http://myserver.somewhere.com/ota/tuya-sonoff.bin will succeed provided the dns server resolves myserver.somewhere.com to your machine. On the other hand: http://192.168.1.10/ota/tuya-sonoff.bin should work.

Zixim commented 5 years ago

my URL to be injected is : URL: http://192.168.11.222:8000/ota/tuya-sonoff.bin

The device is indeed asking at regular interval : 

/gw.json?a=tuya.device.upgrade.silent.get&gwId=14801348b4e62d1d5c5e&t=40&v=4.1&sign=302b82d250cd057342b8175f3774ba85

The answer is very slow, a minute or so : 

Sending Firmware Response...

but nothing happens

SynAckFin commented 5 years ago

I noticed I'll see the tuya.device.upgrade.silent.get line but takes several minutes to post the Sending Firmware Response... line.

That looks like something is going wrong fetching the response from the tuya server.

Zixim commented 5 years ago

You are right...dhcp kicked in and modified the dns... what happens if the device is turned off after tuya-sonoff.bin was flashed , but before sonof-minimal.bin gets flashed ?

digiblur commented 5 years ago

Think I may have missed your response, so the device running the perl script needs to be able to reach the real a.tuyaus.com site?

Zixim commented 5 years ago

yes it does. easy test, right before running the proxy perl script, do a ping to the tuya server, and make sure you are getting the real-world ip address in return.

SynAckFin commented 5 years ago

Here's an updated tuyaproxy.pl script that gives more details tuyaproxy.zip

If its working then the output should be something like:

192.168.31.45 Requesting http://a.tuyaeu.com/gw.json?a=atop.online.debug.log ...
Response: 200 OK
Content: {"result":true,"t":1547053595,"e":false,"success":true}

192.168.31.45 Requesting http://a.tuyaeu.com/gw.json?a=tuya.device.upgrade.silent.get ...
Response: 200 OK
Content: {"t":1547053615,"e":false,"success":true}

                Sending Firmware Response...
192.168.31.45 Requesting http://a.tuyaeu.com/gw.json?a=s.gw.upgrade.updatestatus ...
Response: 200 OK
Content: {"t":1547053615,"e":false,"success":true}
Zixim commented 5 years ago

i've already flashed tuya-sonoff.bin , now struggling with the finalstage... the ssid is there, but 192.168.4.1 isn't pinging , normal ?

SynAckFin commented 5 years ago

Are you running the finalstage script on a computer with wifi?

digiblur commented 5 years ago

Nice..so got that fixed. Saw the sending firmware response go quick... Saw 3 requests for some reason on the webserver for the bin. No Final Stage SSID showing yet. I'll give it time.