Lost communication with TP Link Kasa Plugs after firmware update

[danielsmith89]

WORKAROUND

https://www.tp-link.com/us/support/faq/2707/

The problem

Lost communication with all TP Link Smart Plugs. Smart Lightbulbs still working as expected. However all works fine on TP Link Kasa App, so I know it's on the network and working correctly.

Environment

• Home Assistant Core release with the issue: 0.117.5
• Last working Home Assistant Core release (if known): 0.117.5
• Operating environment (OS/Container/Supervised/Core): Container
• Integration causing this issue: TP Link Kasa
• Link to integration documentation on our website: https://www.home-assistant.io/integrations/tplink/

Problem-relevant configuration.yaml

tplink:
  discovery: false
  light:
    - host: 192.168.1.188
  switch:
    - host: 192.168.1.181
    - host: 192.168.1.176

Traceback/Error logs

2020-11-11 10:18:36 WARNING (SyncWorker_0) [homeassistant.components.tplink.common] Unable to communicate with device 192.168.1.181: Communication error
2020-11-11 10:18:36 WARNING (SyncWorker_0) [homeassistant.components.tplink.common] Unable to communicate with device 192.168.1.176: Communication error

Additional information

[tumm]

Same thing happened to me this morning

[danielsmith89]

Same thing happened to me this morning

It's bizarre. I've even just moved them to running local with static IP's, still nothing. Light bulb is still working flawlessly though, so weird.

[Robbo312]

I'm having the same issue as well on some of my HS110 switches..

Seems to be these don't work, firmware reported in HA; Firmware: 1.0.4 Build 191111 Rel.143903 These do; Firmware: 1.5.10 Build 191125 Rel.094314

I've hit the update firmware in the app, some updated, but im not sure which ones - however looking at a device that has issues it says it's on Firmware Version 1.1.0 (hardware version 4.1) and not 1.0.4 as reported in HA. So it looks like a firmware upgrade borked something. The ones that work are on hardware version 2.1, firmware 1.5.10.

[danielsmith89]

I wonder if TP Link have pushed a firmware update overnight, which has borked some devices...

[Robbo312]

Just done a simple nmap on the devices, the ones that work have port 9999 open, the ones that don't have port 80 open. I don't know if this was the case when they were both working.

Starting Nmap 7.70 ( https://nmap.org ) at 2020-11-11 11:23 GMT Nmap scan report for 192.168.88.51 Host is up (0.00071s latency). Not shown: 999 filtered ports PORT STATE SERVICE 9999/tcp open abyss MAC Address: D8:0D:17:23:47:4E (Unknown)

Starting Nmap 7.70 ( https://nmap.org ) at 2020-11-11 11:23 GMT Nmap scan report for 192.168.88.55 Host is up (0.0023s latency). Not shown: 999 closed ports PORT STATE SERVICE 80/tcp open http MAC Address: B0:95:75:5E:F0:DD (Unknown)

[igiannakas]

Same issue here. TP Link Smart Plug HS110, HW Version 4.1, Firmware Version: 1.1.0, running the latest home assistant release. Using NMAP I see that port 80/TCP is open.

My second TP Link smart plug - HS100, HW Version 2.1, FW version 1.5.10 is working perfectly via HA. Using nmap I see that port 9999/TCP is open

[danielsmith89]

Just done a simple nmap on the devices, the ones that work have port 9999 open, the ones that don't have port 80 open. I don't know if this was the case when they were both working.

nmap 192.168.88.51

Starting Nmap 7.70 ( https://nmap.org ) at 2020-11-11 11:23 GMT Nmap scan report for 192.168.88.51 Host is up (0.00071s latency). Not shown: 999 filtered ports PORT STATE SERVICE 9999/tcp open abyss MAC Address: D8:0D:17:23:47:4E (Unknown)

nmap 192.168.88.55

Starting Nmap 7.70 ( https://nmap.org ) at 2020-11-11 11:23 GMT Nmap scan report for 192.168.88.55 Host is up (0.0023s latency). Not shown: 999 closed ports PORT STATE SERVICE 80/tcp open http MAC Address: B0:95:75:5E:F0:DD (Unknown)

That seems to match up with my devices too. Like you though, I couldn't say if that was the case before we started having problems.

[tombeynon]

I'm seeing this on HS100 devices as well as HS110. Tried restarts, removing integration and re-adding but obviously nothing working. 4 of my 8 devices are working, seems consistently the same devices, but I can't find any difference between those and the missing ones.

[igiannakas]

I'm seeing this on HS100 devices as well as HS110. Tried restarts, removing integration and re-adding but obviously nothing working. 4 of my 8 devices are working, seems consistently the same devices, but I can't find any difference between those and the missing ones.

What is the firmware version for your devices that are working vs the ones not? Im thinking maybe TP Link has pushed a FW update that broke the integration.

[tombeynon]

HS110 working - one on 1.2.6 and one on 1.5.10 HS100 working - 1.0.3 KP303 strip working - 1.0.4 (seems to have combined the entities into one of the plugs as the device)

HS100 not working - 1.1.0

[danielsmith89]

Glad to hear it's not just me 😅 been tearing my hair out this morning thinking it was an issue on my network! Here's hoping it's a simple/easy fix!

[igiannakas]

Seems like the v1.1.0 firmware may be the culprit. My HS110 on V1.1.0 is the one not working as well.

[deosrc]

I thought I had broke my DNS again :joy:

Interestingly it seems the Hardware Version may play into the Firmware Version. I have a few HS110 plugs:

Hardware Version	Firmware Version	Status
2.1	1.5.7	Working (x2)
4.1	1.0.4	Working
4.1	1.1.0	"Unavailable" since 4AM

Guess it's time to add these to the growing list of devices I don't allow access to the internet :disappointed:

EDIT: Updated with status of disconnected plugs

[deosrc]

Also worth noting that the "broken" plug wouldn't connect in Kasa until I updated the app.

[honkerst]

Same symptoms here. One HS110 with HW 2.1 and FW 1.5.7 still working, but another HS110 with HW4.1 and FW1.1.0 stopped working last night.

[SimonWilkinson]

The "broken" devices have a responsive web server on port 80, which claims to be "Server: SHIP 2.0"

[oscarhibbert]

Same symptoms & w/ same timeframe also.

6x UK HS100's Hardware v4.1 Firmware 1.1.0

Working without issue via Kasa app.

[spacegaier]

Interestingly enough, mine still work (both via HA and app): HS100 (HW 4.0, FW 1.1.5) HS110 (HW 4.0, FW 1.0.4)

Looking through here, it seems the issue only/mainly affects hardware version 4.1?

[danielsmith89]

Interestingly enough, mine still work (both via HA and app): HS100 (HW 4.0, FW 1.1.5) HS110 (HW 4.0, FW 1.0.4)

Looking through here, it seems the issue only/mainly affects hardware version 4.1?

I think it's just the firmware version 1.1.0 that is causing the problem.

[drewy-uk]

Any way to revert back to the older firmware?

[YaBoiDan]

I can confirm the same has happened here too, all firmware version 1.1.0.

[Madbeefer]

All of my normal switches (HS200)s are offline, but my dimmers (HS220) are online. Sometimes I will get a couple of the HS200s back after an update to HA comes out, but as soon as it needs to be reset after the update I get only the dimmers back.

[igiannakas]

As I rely on this integration working to automate my “dumb” dehumidifier, I’ve created a workaround via Alexa to control it. I’ve created a helper toggle that is used in a binary sensor value template exposed to Alexa. Then Alexa controls the tp link plug via the Kasa skill when the state of that binary sensor changes. It works the same way and with no real delay, but it’s the long way around and I’ve lost the current and wattage reporting capabilities of the native integration.

[danielsmith89]

@rytilahti or @thegardenmonkey, any chance either of you could have a look into this? Thanks.

[rytilahti]

See the linked python-kasa issue, and also https://github.com/python-kasa/python-kasa/issues/42 . It is all very unclear, but from the looks of it (and due to that changelog entry mentioned in the issue) it may be that tplink is blocking this access on newer firmwares. Only thing I can do is to recommend not updating the devices nor connecting them to the cloud, if not necessary. T

here were some previous reports that some devices will lose the local access when connected to the cloud, although https://www.tp-link.com/de/support/faq/2707/ is still up. Potentially relevant issue (if the device still works when configured statically): https://github.com/python-kasa/python-kasa/issues/105

Reading through the comments wrt. port 80 being open, see https://github.com/python-kasa/python-kasa/issues/113#issuecomment-726263084

[TheGardenMonkey]

All of my HS100's are HW - 2.0 and FW - 1.56 HS105's are HW - 1.0 and FW - 1.56 HS107's are HW - 1.0 and FW - 1.0.10 HS300 is HW - 1.0 and FW - 1.0.19 HS200's are HW - 2.0 and FW - 1.5.7 HS210's are HW - 1.0 and FW - 1.5.8 HS220's are HW - 1.0 and FW - 1.5.11 KL110 is HW - 1.0 and FW - 1.8.11 KP400 is HW - 2.0 and FW - 1.0.6

I have a few other models I should plugin and check, but everything (27 devices in total) is working as expected. I usually just purchase my switches on amazon but I don't know how to get a V4 specifically to help troubleshoot.

[rytilahti]

Potentially related, if someone wants to take a look into potentially downgrading to a previous firmware version (although I'd guess that downgrades are not allowed): https://pceasies.com/blog/2017/12/17/updating-tp-link-smart-plug-firmware-hs110/

[igiannakas]

Potentially related, if someone wants to take a look into potentially downgrading to a previous firmware version (although I'd guess that downgrades are not allowed): https://pceasies.com/blog/2017/12/17/updating-tp-link-smart-plug-firmware-hs110/

This won't work - the download on the attached link is or V1 hardware, and also the tool used to update the firmware needs port 9999 but this is now closed on the device itself.

[rytilahti]

Yes, I just wanted to post in case someone figures out how to communicate with the updated devices over port 80 and wants to try to do a downgrade, if/when the newer firmwares make it much harder to implement local controls.

[deosrc]

I have emailed TP-Link support and requested either instructions for downgrading the firmware or technical details of the protocol but I am not optimistic. I suspect they do not want the support overhead and are also employing some "security by obscurity".

I've managed to run some packet captures on my phone to try work out what is going on but I believe root is required to do this. Steps I took were:

1. Connect phone to computer and run adb shell (usual steps required to ensure phone is in USB debug mode)
2. Elevate to root by running su
3. Force close the kasa app (to ensure a full lifecycle of discovery, connection, communication)
4. Start capture using tcpdump -vv -i any -s 0 -w /sdcard/dump.pcap
5. Copy the file back to the computer and open with Wireshark
6. Filter the results to the IP of the plug ip.addr == <ip-of-plug>

The good news is that the new firmware seems to be using HTTP POST requests on port 80 using application/x-www-form-urlencoded. Unfortunately, Wireshark doesn't seem to understand the form contents.

As @SimonWilkinson says at https://github.com/python-kasa/python-kasa/issues/113#issuecomment-726263084, there seems to be some kind of authentication process of POSTing to /app/handshake1, followed by /app/handshake2 (possibly repeated but could just be a failure to connect). There is a TP_SESSIONID cookie returned in the response from /app/hanshake1 which is probably relevant.

Control of the plug is then done by POSTing to /app/request?seq=<seq-id> where <seq-id> is an incrementing value, possibly based on function being performed as there was an anomalous request which didn't match the incrementing.

I tried sending a few postman requests using the URLs, with and without the cookie with no body. I received a 400 Bad Request every time, or 404 Not Found if the URL is wrong.

[rytilahti]

I have emailed TP-Link support and requested either instructions for downgrading the firmware or technical details of the protocol but I am not optimistic. I suspect they do not want the support overhead and are also employing some "security by obscurity".

Yeah, I wouldn't be holding my breath on that. That being said, tplink uses or has used GPL-licensed software, and some source code is released at https://www.tp-link.com/en/support/gpl-code/ . I have no idea if anything released there is useful though.

The good news is that the new firmware seems to be using HTTP POST requests on port 80 using application/x-www-form-urlencoded. Unfortunately, Wireshark doesn't seem to understand the form contents.

Yes, it is encrypted. See my comment here https://github.com/softScheck/tplink-smartplug/issues/81#issuecomment-726939746 - the question is, is it even possible to use these devices without any internet connectivity? If yes, the encryption key is derived from the device response and if someone figures out how the encryption is done, this would allow local controls. Or maybe the key is static like it used to be? There is currently not simply enough information to say what is happening.

Control of the plug is then done by POSTing to /app/request?seq=<seq-id> where <seq-id> is an incrementing value, possibly based on function being performed as there was an anomalous request which didn't match the incrementing.

The id is likely just to be able to ignore duplicate requests. What you could try is to resend a working request with an incremented value to see if it is accepted. If yes, the encrypted payload itself does not depend on some shared state which would allow re-using the encrypted payloads without figuring out the encryption. If the encryption is not bound to any information from the device, which is unlikely, it would allow simply re-using these requests on other devices.

[probot-home-assistant[bot]]

tplink documentation tplink source _{^{(message by IssueLinks)}}

[deosrc]

the question is, is it even possible to use these devices without any internet connectivity?

I've moved all of my TP-Link smart plugs to an area of my network which should be firewalled from the internet and allow only local access in and out to prevent them updating again. The remaining HW v4.1 plug has not updated so I think the firewall is working correctly are correct and the updated plug is still working locally. Unless it managed to cache something while it was online or if the key is static as you say, I think they can still work offline.

I realised that the discovery packet is not encrypted:

{
    "result": {
        "ip": "192.168.[REDACTED]",
        "mac": "B0-95-75-[REDACTED]",
        "device_id": "57FF5E67DA3326A2[REDACTED]",
        "owner": "",
        "device_type": "IOT.SMARTPLUGSWITCH",
        "device_model": "HS110(UK)",
        "hw_ver": "4.1",
        "factory_default": true,
        "mgt_encrypt_schm": {
            "is_support_https": false,
            "encrypt_type": "KLAP",
            "http_port": 80
        }
    },
    "error_code": 0
}

(output prettied)

I've never heard of KLAP before. I wondered if it would be possible to spoof this discovery packet and possibly trick the app into sending decrypted commands but there's a few raw bytes of prefix which may be important and it was a long shot anyway.

[brendonrogers]

Similar discussion with the folk over at Homebridge homebridge-tplink-smarthome

https://github.com/plasticrake/homebridge-tplink-smarthome/issues/154

[smoke007]

I have three model HS100, hardware version 1.0, with firmware 1.2.6 that all stopped working. All 3 still have port 9999 (abyss) open.

[nathanmay]

Same issues here my HA has lost all connectivity to all my HS100's - all running firmware version 1.1.0

[danerich]

Having the same issue here. My HS100's are under my bed so heard them clicking as if they were turning on and off in the middle of the night so guessing they have performed a firmware update. Hardware version: 4.1 Firmware version: 1.1.0. Both now not showing in HA and still working with the Kassa app.

[Codestian]

Facing same issue of plugs not communicating with HA. However, the issue only applies to 2 of my smart plugs, whereas the other 2 is working well with no issues.

Edit: I have 4 HS100 smart plugs.

[SeamusMcCarthy]

Have 2 plugs bought recently for a college IoT assignment. Had the proposal all written up and prototyped.... and then this happened :/ Hoping people much smarter than I figure out a workaround. Probably goes without saying but both v4.1

[cjdshaw]

I'm getting the same on my two HS100 4.1s, both FW 1.1.0

Trying Stream on my iPhone, it seems the app is POSTing to http://ip:80/app/request?seq=int where int started at 1448106091 and increased by 1 each call. Unfortunately, both the payload and the response are (presumably encrypted) gibberish

[igiannakas]

Have 2 plugs bought recently for a college IoT assignment. Had the proposal all written up and prototyped.... and then this happened :/ Hoping people much smarter than I figure out a workaround. Probably good without saying but both v4.1

If you want an immediate workaround try using an Alexa as I’ve done myself a few posts above.

[SeamusMcCarthy]

Have 2 plugs bought recently for a college IoT assignment. Had the proposal all written up and prototyped.... and then this happened :/ Hoping people much smarter than I figure out a workaround. Probably goes without saying but both on v4.1

If you want an immediate workaround try using an Alexa as I’ve done myself a few posts above.

Cheers, I did manage to use an Alexa but for a more caveman workaround. Instead of using python-kasa directly, my webservice now uses espeak to tell Alexa by voice command to turn on/off the plugs. Not ideal but gets me where I need to be for the moment.

[cjdshaw]

Have 2 plugs bought recently for a college IoT assignment. Had the proposal all written up and prototyped.... and then this happened :/ Hoping people much smarter than I figure out a workaround. Probably goes without saying but both v4.1

There's also the cloud API, of course. In some ways, it's simpler since you can use it from the command line with curl and jq https://itnerd.space/tag/hs100/

[Robbo312]

There's also the cloud API, of course. In some ways, it's simpler since you can use it from the command line with curl and jq https://itnerd.space/tag/hs100/

Cheers for the link, using that method I can also get to the emeter readings for my hs110's using the REST sensor.

I found this node-red flow that also talks to the cloud that some may find useful. https://gist.github.com/adumont/dbc679320ca68f6c1a149c2f4fe9a439

[SeamusMcCarthy]

IFTTT applets are also still communicating with the plugs so can just post requests to those

[conorlap]

Any idea if this is/will be fixable? Very disappointed that I didn't block my HS100's from talking to the internet, could've avoided this mess with no firmware updates!

[conorlap]

As a workaround for now, I've setup multiple routines on my Alexa app to switch on/off the various plugs etc. Then I created multiple scripts within HA to call the routines. At least this way I can get my automations up and running again!

Example script:

alias: Hall Camera Off sequence:

• service: media_player.play_media data: media_content_id: hallcameraoff media_content_type: routine entity_id: media_player.living_room_echo_dot mode: single

[chriswheeldon]

I also suffered from this firmware update breaking my home automation setup last week. Over the weekend I have managed to make some progress on reverse engineering the updated protocol in order to restore interoperability. I can now handshake with my HS110 and issue an encrypted request to get info. Summary:

1. UDP broadcast with fixed payload on port 20002 to discover devices.
2. Requests are plaintext HTTP on TCP port 80 with encrypted contents
3. Two step handshake to determine encryption key and base IV to use for later requests.
4. Key and IV seem to be derived from the device email and password. Presumably this is for the account registered for TP-Link cloud functionality. I haven't registered and so the email and password are empty strings.
5. AES-128-CBC for request and response body encryption.
6. HTTP requests include an incrementing sequence number as a query parameter. This is used to determine the per-request IV.
7. Encrypted request and response bodies are prefixed with a HMAC.
8. Pre-documented requests seem to work e.g. {"system":{"get_sysinfo":null}}

Here is a proof of concept showing how to handshake and send a request to the device. FWIW It seems fairly unreliable: I get a 403 response to the request for system info about 50% of the time. Would be interesting to try and determine why that is...

https://gist.github.com/chriswheeldon/3b17d974db3817613c69191c0480fe55

@plasticrake would be wonderful to get this integrated into tplink-smarthome-api

[deosrc]

@chriswheeldon Just tried out your code. Got a success response from /app/handshake1 :tada: Unfortunately the assert on the seed failed (https://gist.github.com/chriswheeldon/3b17d974db3817613c69191c0480fe55#file-hs110-py-L88). I factory reset the plug and re-configured it without connecting to Kasa but got the same error. Any ideas?

[chriswheeldon]

@deosrc do you have a TP-Link account (so that you can control the plug from outside your local network)? If so then you might need to set an email and password at https://gist.github.com/chriswheeldon/3b17d974db3817613c69191c0480fe55#file-hs110-py-L79

I've not tried this as I do not have an account.