Deco S7 - Fetch keys 401 unauthorized

[Nedevski]

Version of the custom_component

v2.2.3

Deco Model

Deco S7 (1.3.0 Build 20220609 Rel. 64814)

Describe the bug

Just got 2 different Deco networks and I'm trying to add them to 2 different HA instances (one of them is fresh). I get the same result, I am using the settings below. The IP of the root node is correct, the password is correct. At my main HA instance I got some 2-3 second delay the first time I tried to add it and then it was an immediate error afterwards Checked the logs and they don't provide any meaningful info for me, just straight 401's

I saw the other issue, however my error messages are different and that's why I'm opening a new issue.

Debug log

[Warning] Error testing credentials: 401, message='Unauthorized', url=URL('http://192.168.1.2/cgi-bin/luci/;stok=/login?form=keys')
[Error] Fetch keys client response error: 401, message='Unauthorized', url=URL('http://192.168.1.2/cgi-bin/luci/;stok=/login?form=keys')

[amosyuen]

Follow steps in Additional Debugging https://github.com/amosyuen/ha-tplink-deco/blob/master/DEBUGGING.md and find what the request for fetch keys looks like

[Nedevski]

Here is the debug log, nothing of value, I think

2022-12-26 18:30:06.704 DEBUG (MainThread)
[homeassistant.components.websocket_api.http.connection] [140208908303520]
Sending {"id":85,"type":"result","success":true,"result":[{"name":"custom_components.tplink_deco",
"message":["Error testing credentials: 401, message='Unauthorized',
url=URL('http://192.168.1.2/cgi-bin/luci/;stok=/login?form=keys')"],
"level":"WARNING","source":["custom_components/tplink_deco/config_flow.py",81],
"timestamp":1672072193.8828318,"exception":"","count":1,"first_occurred":1672072193.8828318},
{"name":"custom_components.tplink_deco",
"message":["Fetch keys client response error: 401, message='Unauthorized',
url=URL('http://192.168.1.2/cgi-bin/luci/;stok=/login?form=keys')"],"level":"ERROR",
"source":["custom_components/tplink_deco/api.py",137],"timestamp":1672072193.8812113,
"exception":"","count":1,"first_occurred":1672072193.8812113}]}

I logged in to the Deco Web UI with dev console open and these are the only requests I'm getting:

Also when I'm logged in the Web UI and I try to add the integration I am not logged out. Even if I logout of the Web UI and I try again, still no luck.

Maybe it's just incompatible with this firmware?

EDIT: I tried clearing all cache, cookies and website data, same result.

[Nedevski]

Ok, I did the advanced debugging part, I did find some data={something} in one of the requests and I ran it through the command.

Quick note - the command in the documentation did not work: jQuery.encrypt.encryptManager.encryptor.AesDecrypt(decodeURIComponent(data));

However this was successful: jQuery.encrypt.encryptManager.encryptor.aes.decrypt(decodeURIComponent(data));

The output was: {"operation":"read"}

[amosyuen]

Hmm, since you didn't see an http call that looks like login?form=keys then seems like the authentication method is different in that model and firmware.

You should only get logged out if the authentication suceeds, so it's expected that you didn't get logged out since it failed.

It is rather difficult to reverse engineer the API remotely, so we'll probably have to wait for a programmer who has a deco with incompatible firmware to look into it. For now I'll add your model and firmware to the README as incompatible.

[wzaatar]

Same error with a Deco X60 mesh…

[bsimmo]

Still happens with 1.3.3 firmware? I guess more will go this way too.

[Nedevski]

Yes, still happens with 1.3.3 firmware. This is not an issue on the Deco side, they just seem to use different requests with the newer Deco models.

I am a developer myself, but not with Python but with .NET. I am still pretty new to creating/modifying integrations, so if someone that has more experience is willing to help, I can look into this.

[amosyuen]

@Nedevski I can help with the HA integration part, the difficult part is reverse engineering TP-Link's calls and encryption scheme. In the old decos they do some encryption with RSA and AES, generating payloads with matching signs. Honestly I only figured it out from looking at other libraries / blog posts. Hopefully they haven't changed the encryption scheme too much.

[fgsilva12]

Anyone working on a solution for the Deco S7?

[Nedevski]

I just had a small breaktrough. So it looks like after you are logged in, you poll 3 different endpoints - network, device, client.

All requests are in the form of endpoint?form=performance&id=some_long_key and they return some encrypted response string. I just saw my previous comment about using the jquery encryptManager and I modified it a bit.

jQuery.encrypt.encryptManager.encryptor.aes.decrypt("some_encrypted_response_string");

Without any additional parameters I was able to get a full readable JSON with all my devices:

      {
        "linked_device_info": {
          "signal_level": {
            "band2_4": 0,
            "band5": 0
          },
          "connection_type": [
            "wired"
          ],
          "device_id": ""
        },
        "ip": "0.0.0.0",
        "mac": "XX-XX-XX-XX-XX-XX",
        "name": "UGhvbmU=",
        "online": false,
        "owner_id": "1",
        "remain_time": 0,
        "enable_priority": false,
        "interface": "main",
        "client_type": "phone",
        "wire_type": "wired",
        "connection_type": "wired",
        "up_speed": 0,
        "down_speed": 0
      },

I will look into it a bit more later today and will report if I find anything. If someone is a bit more familiar with encryption in general and is willing to waste a bit of time on this - ping me.

[aronkahrs-us]

seems like the authentication method is different in that model and firmware.

As far as I understand the new authentication method has 4 "steps" to login.

Step 1: Some sort of "enable", it sends a POST request with 2 parameters (code and asyn) and in the form data it sends enable code is 16 and asyn is 0

headers = {
        'authority': '<here goes the host>',
        'accept': '*/*',
        'accept-language': 'es-ES,es;q=0.8',
        'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
        'origin': '<here goes the host>',
        'referer': '<here goes the host>',
        'sec-ch-ua': '"Not A(Brand";v="99", "Brave";v="121", "Chromium";v="121"',
        'sec-ch-ua-mobile': '?0',
        'sec-ch-ua-platform': '"Windows"',
        'sec-fetch-dest': 'empty',
        'sec-fetch-mode': 'cors',
        'sec-fetch-site': 'same-origin',
        'sec-gpc': '1',
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36',
        'x-requested-with': 'XMLHttpRequest',
    }

    data = 'enable'

    response = requests.post(
        '<here goes the host>/?code=16&asyn=0',
        headers=headers,
        data=data,
    )

Step 2: Get the RSA ee and nn, it sends a POST request with 3 parameters (code, asyn and id) and in the form data it sends get code is 16, asyn is 0 and id seems to be a random, 32 character, alphanumeric string

headers = {
        'authority': '<here goes the host>',
        'accept': '*/*',
        'accept-language': 'es-ES,es;q=0.8',
        'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
        'origin': '<here goes the host>',
        'referer': '<here goes the host>',
        'sec-ch-ua': '"Not A(Brand";v="99", "Brave";v="121", "Chromium";v="121"',
        'sec-ch-ua-mobile': '?0',
        'sec-ch-ua-platform': '"Windows"',
        'sec-fetch-dest': 'empty',
        'sec-fetch-mode': 'cors',
        'sec-fetch-site': 'same-origin',
        'sec-gpc': '1',
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36',
        'x-requested-with': 'XMLHttpRequest',
    }

    data = 'get'

    response = requests.post(
        f'<here goes the host>/?code=16&asyn=0&id={USER_ID}',
        headers=headers,
        data=data,
    )

Step 3: The actual login, here we send the password (RSA encrypted), it sends a POST request with 3 parameters (code, asyn and id) and in the form data it sends the RSA encrypted password code is 7, asyn is 0 and id seems to be a random, 32 character, alphanumeric string

headers = {
        'authority': '<here goes the host>',
        'accept': '*/*',
        'accept-language': 'es-ES,es;q=0.8',
        'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
        'origin': '<here goes the host>',
        'referer': '<here goes the host>',
        'sec-ch-ua': '"Not A(Brand";v="99", "Brave";v="121", "Chromium";v="121"',
        'sec-ch-ua-mobile': '?0',
        'sec-ch-ua-platform': '"Windows"',
        'sec-fetch-dest': 'empty',
        'sec-fetch-mode': 'cors',
        'sec-fetch-site': 'same-origin',
        'sec-gpc': '1',
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36',
        'x-requested-with': 'XMLHttpRequest',
    }

    ee = keys[1]
    nn = keys[2]
    data = str(rsa_encrypt(int(nn,16),int(ee,16),password.encode()))

    response = requests.post(
        f'<here goes the host>/?code=16&asyn=0&id={USER_ID}',
        headers=headers,
        data=data,
    )

Step 3: This step sets something, but I haven't figured out what, it sends a POST request with 3 parameters (code, asyn and id) and in the form data it sends set followed by something RSA encrypted code is 16, asyn is 0 and id seems to be a random, 32 character, alphanumeric string

headers = {
        'authority': '<here goes the host>',
        'accept': '*/*',
        'accept-language': 'es-ES,es;q=0.8',
        'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
        'origin': '<here goes the host>',
        'referer': '<here goes the host>',
        'sec-ch-ua': '"Not A(Brand";v="99", "Brave";v="121", "Chromium";v="121"',
        'sec-ch-ua-mobile': '?0',
        'sec-ch-ua-platform': '"Windows"',
        'sec-fetch-dest': 'empty',
        'sec-fetch-mode': 'cors',
        'sec-fetch-site': 'same-origin',
        'sec-gpc': '1',
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36',
        'x-requested-with': 'XMLHttpRequest',
    }

    data = "set <Something encrypted>"

    response = requests.post(
        f'<here goes the host>/?code=7&asyn=0&id={USER_ID}',
        headers=headers,
        data=data,
    )

Every step returns 00000 if the request was successful, if not, it returns three, 5 numbers codes(error codes I assume), an encrypted code and 00000 at the end

All requests are in the form of endpoint?form=performance&id=some_long_key and they return some encrypted response string.

That id is the random, 32 character, alphanumeric string used to login

If I manage to get anything else I'll share it here, hope this helps.

[aronkahrs-us]

Update

This step sets something, but I haven't figured out what

It sets the AES Key (i supose to decrypt on the server side?), so basically it sends the key and the iv as a RSA encrypted string.

The string it encrypt is f"k={key}&i={iv}", so the data in the request ends up something like this:

data = "set "+str(rsa_encrypt(int(nn,16),int(ee,16),f"k={key}&i={iv}".encode()))

[aronkahrs-us]

Update 2

id seems to be a random, 32 character, alphanumeric string

Turns out it's not random, it's an encrypted form of the password:

var n = s.su.encrypt(e[3], s.encrypt.MD5(r), e[4])
return localStorage.setItem("id", n)

I don't know which encryption it uses.

[aronkahrs-us]

doLogin function

I'm going to share the controller.js file that has the doLogin function

!function(s) {
    s.su.moduleManager.define("localLogin", {
        services: ["ajax"],
        models: ["localLogin", "localLoginControl"],
        stores: [],
        views: ["localLoginView"],
        deps: ["login", "main"],
        listeners: {
            "ev_on_launch": function(e, n, t, o, r, l, a) {
                s.encrypt.encryptManager.cleanStorage(),
                s.su.encryptor = s.encrypt.encryptManager.genEncryptor(),
                localStorage.removeItem("id"),
                l.main.keyDictionary = "",
                l.main.getTmpKey()
            }
        },
        init: function(n, e, t, o, r, l) {
            this.control({
                "#local-login-pwd": {
                    "keyup": function(e) {
                        13 == e.keyCode && n.doLogin()
                    }
                },
                "#local-login-button": {
                    "ev_button_click": function() {
                        n.doLogin()
                    }
                },
                ".local-login-switch-to-tp": {
                    "click": function(e) {
                        r.login.goToChildModule("tpLogin")
                    }
                },
                ".local-login-forget-pwd": {
                    "click": function() {
                        e.localLoginView.forgetPasswordMsg.show()
                    }
                }
            })
        }
    }, function(l, c, i, e, a, n) {
        return {
            encryptKey: null,
            doLogin: function() {
                if (i.localLoginControl.password.validate()) {
                    var r = i.localLoginControl.password.getValue();
                    c.localLoginView.loginBtn.disable(),
                    a.main.getTmpKey().then(function(e) {
                        return l.enableGDPR(e)
                    }).then(function(e) {
                        var n = s.Deferred();
                        return l.getGDPRKey().then(function() {
                            n.resolve(e)
                        }, function() {}),
                        n
                    }).then(function(e) {
                        var n = s.su.encrypt(e[3], s.encrypt.MD5(r), e[4])
                          , o = s.Deferred()
                          , t = s.encrypt.encryptManager.getEncryptor();
                        return localStorage.setItem("id", n),
                        s.ajax({
                            url: s.su.url("?code=7&asyn=0"),
                            data: t.rsaEncrypt(r)
                        }).then(function(e) {
                            "00000" === e.split("\r\n")[0] ? (i.localLoginControl.password.setValue(null),
                            o.resolve()) : o.reject()
                        }, function(e) {
                            var n = e.responseText.split("\r\n");
                            switch (parseInt(n[1])) {
                            case 0:
                                i.localLoginControl.password.setError(s.su.CHAR.LOGIN.loginFull);
                                break;
                            case 1:
                                i.localLoginControl.password.disable(),
                                c.localLoginView.leftAttemptsMsgContent.setText(s.su.CHAR.LOGIN.loginLock),
                                c.localLoginView.leftAttemptsMsg.show();
                                break;
                            case 2:
                                i.localLoginControl.password.setError(s.su.CHAR.LOGIN.loginTimeout);
                                break;
                            case 3:
                            case 5:
                                var t = n[2] == undefined ? 10 : 10 - parseInt(n[2]);
                                if (10 === t)
                                    break;
                                l.disableButton(c.localLoginView.loginBtn),
                                t <= 5 ? (c.localLoginView.leftAttemptsMsgContent.setText(s.su.CHAR.LOGIN.loginErrorTipH.replace("%s", t)),
                                c.localLoginView.leftAttemptsMsg.show()) : i.localLoginControl.password.setError(s.su.CHAR.LOGIN.loginPwdErr);
                                break;
                            case 6:
                                l.disableButton(c.localLoginView.loginBtn),
                                i.localLoginControl.password.setError(s.su.CHAR.LOGIN.loginPwdErr)
                            }
                            o.reject()
                        }),
                        o
                    }).then(function() {
                        return l.setGDPRKey()
                    }).then(function() {
                        a.main.loadBasicModule("index")
                    }).always(function() {
                        0 < l.countDownSec || c.localLoginView.loginBtn.enable()
                    })
                }
            },
            loginSuccessDealer: function(e, n) {
                var t, o, r, l = e.stok || (t = "12345",
                o = top.location.href,
                0 <= (r = o.indexOf("stok=")) && (t = o.substring(r + 5)),
                t);
                localStorage && (a.main.setToken(l),
                a.main.reload())
            },
            loginFailDealer: function(e, n) {
                var t = e.result;
                switch (c.localLoginView.loginBtn.enable(),
                n) {
                case -5002:
                    var o, r = t.failureCount, l = t.attemptsAllowed;
                    if (r < 7) {
                        var a = String(-5002).replace(/^-/, "E");
                        i.localLoginControl.password.setError(s.su.CHAR.ERROR[a])
                    } else
                        o = s.su.CHAR.LOGIN.LOGIN_FAILED_COUNT.replace("%num1", r).replace("%num2", l),
                        c.localLoginView.leftAttemptsMsgContent.setText(o),
                        c.localLoginView.leftAttemptsMsg.show();
                    break;
                case -5003:
                    c.localLoginView.maxAttemptsMsgContent.setText(s.su.CHAR.ERROR["00000089"]),
                    c.localLoginView.maxAttemptsMsg.show()
                }
            },
            queryRecoveryPasswordMethod: function() {
                var e = s.Deferred();
                return e.resolve(!0),
                e.promise()
            },
            enableGDPR: function(n) {
                var t = s.Deferred();
                return s.ajax({
                    url: s.su.url("?code=16&asyn=0"),
                    data: "enable"
                }).then(function(e) {
                    "00000" === e.split("\r\n")[0] ? t.resolve(n) : t.reject()
                }, function() {
                    t.reject()
                }),
                t
            },
            getGDPRKey: function() {
                var n = s.Deferred()
                  , t = s.encrypt.encryptManager.getEncryptor() || s.encrypt.encryptManager.genEncryptor();
                return s.ajax({
                    url: s.su.url("/?code=16&asyn=0"),
                    data: "get"
                }).then(function(e) {
                    data = e.split("\r\n"),
                    "00000" === data[0] ? (t.setRSAKey(data[2], data[1]),
                    t.setSeq(data[3]),
                    t.genAESKey(),
                    s.encrypt.encryptManager.recordEncryptor(),
                    n.resolve()) : n.reject()
                }, function() {
                    n.reject()
                }),
                n
            },
            setGDPRKey: function() {
                var n = s.Deferred()
                  , e = s.encrypt.encryptManager.getEncryptor();
                return s.ajax({
                    url: s.su.url("/?code=16&asyn=0"),
                    data: "set " + e.getEncodeAESKey()
                }).then(function(e) {
                    "00000" === e.split("\r\n")[0] ? n.resolve() : n.reject()
                }, function() {
                    n.reject()
                }),
                n
            },
            disableButton: function(e) {
                l.countDownSec = 30,
                e.setText(s.su.CHAR.LOGIN.LOGIN_BUTTON_COUNTDOWN.replace("%num%", l.countDownSec)),
                l.intervalId = setInterval(function n() {
                    l.countDownSec--,
                    0 < l.countDownSec ? e.setText(s.su.CHAR.LOGIN.LOGIN_BUTTON_COUNTDOWN.replace("%num%", l.countDownSec)) : (clearInterval(l.intervalId),
                    e.enable(),
                    e.setText(s.su.CHAR.LOGIN.LOGIN_BUTTON_TEXT))
                }, 1e3),
                e.disable()
            }
        }
    })
}(jQuery);