Handle different 4xx-errors

[Olen]

Trying harder to log in on a 4xx Error code. Also add status code to error message.

Should help fixing https://github.com/home-assistant/core/issues/97885

UNTESTED!

[persandstrom]

Hi,

Please bump the version (patch) and add a row to the revision history in README.

let's hope this will bring us closer to understanding the issue...

[Olen]

I am a bit reluctant to just release this as I don't know the code well enough to really trust what will happen if we "ccontinue" on a 4xx-error. But if you believe it is safe and will not cause endless loops of failed connections and basically a ddos of Versiure, I can do that.

[persandstrom]

As I see it the issue with this change might be that a wrong password is used once more, and thus using up allowed retries faster. The "contine" will only do one more Try on the other Endpoint, so not really a risk to start an involuntary ddos attack 😀

[Olen]

Tested it locally, and it works as expected. I added an extra debug print() here to verify that it only runs twice and with different urls:

(Olen-patch-1)]$ python verisure.py --event-log asdg asder
Login failed on https://automation02.verisure.com
Login failed on https://automation01.verisure.com
Traceback (most recent call last):
  File "/home/olen/prog/python-verisure/verisure.py", line 3, in <module>
    __main__.cli()
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/olen/prog/python-verisure/verisure/__main__.py", line 127, in cli
    installations = session.login()
  File "/home/olen/prog/python-verisure/verisure/session.py", line 135, in login
    response = self._post(
  File "/home/olen/prog/python-verisure/verisure/session.py", line 125, in wrapper
    raise last_exception
verisure.session.LoginError: Login error, status code: 401 - Data: {"errorGroup": "UNAUTHORIZED","errorCode": "AUT_00004","errorMessage": "Invalid username/password/authentication method combination"}

Logging in with muy real username and password works fine.

[niro1987]

Can you also show that when the first url return 4** (on valid credentials) that the other url then return a 200?

[Olen]

Adding a couple of print()s more:

                    if response.status_code == 200:
                        print(f"Login ok on {base_url}")
                        print(f"Status code: {response.status_code} - Data: {response.text}")

(Olen-patch-1)]$ python verisure.py --event-log XXXXX ZZZZZZ
Login ok on https://automation01.verisure.com     
Status code: 200 - Data: 

[Olen]

I can't easily replicate the logout problem, so we just need to wait until it happens. I can patch vsure in my HA container and restart it and try to see what happens next time it fails

[niro1987]

I ment, in the issue, for some reason the server return 400 and breaks the try-loop, returning a LoginError as a result. With this change, the other url will be tested and we are expecting it to return a 200 result. So far I did not see proof of that.

[niro1987]

You can make these changes to your local vsure package without having to promote this to pypi.

[Olen]

Yeah. Sorry. I just realized what you ment. I patched session.py in my HA container, adding some more debug info. Will see how it goes next time the problem appears.

[Olen]

After digging through some logs here, it seems like there are two different 401-messages.

If I intentionally use a bad username/password, I get this: Code: 401 - {"errorGroup": "UNAUTHORIZED","errorCode": "AUT_00004","errorMessage": "Invalid username/password/authentication method combination"}

But it seems like the "Cookie Expired" message(?) is misleadingly described as:

Code: 401 - {"errorGroup": "UNAUTHORIZED","errorCode": "AUT_00001","errorMessage": "Username/password does not match any valid login"}

I am still waiting for the issue to show up again. It might be up to 48 hours or so before that happens...

[niro1987]

We've learned that at least...

I've also made the same change in my dev environment and will report the response when I see the issue reappear.

[Olen]

I got one:

Lots of "Finished fetching... success: True", and then:

2023-08-09 13:50:07.140 DEBUG (MainThread) [custom_components.verisure] Finished fetching verisure data in 1.055 seconds (success: True)
2023-08-09 13:51:07.264 WARNING (SyncWorker_14) [verisure.session] Error from https://automation02.verisure.com: 401 - {"errorGroup": "UNAUTHORIZED","errorCode": "AUT_00001","errorMessage": "Username/password does not match any valid login"}
2023-08-09 13:51:07.397 WARNING (SyncWorker_14) [verisure.session] Error from https://automation01.verisure.com: 401 - {"errorGroup": "UNAUTHORIZED","errorCode": "AUT_00001","errorMessage": "Username/password does not match any valid login"}
2023-08-09 13:51:07.399 ERROR (MainThread) [custom_components.verisure] 3 Could not log in to verisure, Login error, status code: 401 - Data: {"errorGroup": "UNAUTHORIZED","errorCode": "AUT_00001","errorMessage": "Username/password does not match any valid login"}
2023-08-09 13:51:07.407 ERROR (MainThread) [custom_components.verisure] Authentication failed while fetching verisure data: Credentials expired for Verisure
2023-08-09 13:51:07.410 DEBUG (MainThread) [custom_components.verisure] Finished fetching verisure data in 0.325 seconds (success: False)
2023-08-09 13:51:07.416 DEBUG (MainThread) [custom_components.verisure] Running async_step_reauth,
2023-08-09 13:51:07.417 DEBUG (MainThread) [custom_components.verisure] Running async_step_reauth_confirm,

But then I tried something else. I logged back in in HA, and then I logged in to the app, and changed the password.

That did first give me another error - Session has expired - but on second try, it gave the same error as above:

2023-08-09 18:01:46.571 WARNING (SyncWorker_12) [verisure.session] Error from https://automation02.verisure.com: 401 - {"errorGroup": "UNAUTHORIZED","errorCode": "AUT_00001","errorMessage": "Session has expired"}
2023-08-09 18:01:46.698 WARNING (SyncWorker_12) [verisure.session] Error from https://automation01.verisure.com: 401 - {"errorGroup": "UNAUTHORIZED","errorCode": "AUT_00001","errorMessage": "Username/password does not match any valid login"}
2023-08-09 18:01:46.706 ERROR (MainThread) [custom_components.verisure] 3 Could not log in to verisure, Login error, status code: 401 - Data: {"errorGroup": "UNAUTHORIZED","errorCode": "AUT_00001","errorMessage": "Username/password does not match any valid login"}
2023-08-09 18:01:46.708 ERROR (MainThread) [custom_components.verisure] Authentication failed while fetching verisure data: Credentials expired for Verisure
2023-08-09 18:01:46.720 DEBUG (MainThread) [custom_components.verisure] Finished fetching verisure data in 0.372 seconds (success: False)
2023-08-09 18:01:46.733 DEBUG (MainThread) [custom_components.verisure] Running async_step_reauth,
2023-08-09 18:01:46.733 DEBUG (MainThread) [custom_components.verisure] Running async_step_reauth_confirm,

I will try to make some logic that might be able to handle this.

[Olen]

Tested the latest changes, and if I log in to verisure and change my password, HomeAssistant also asks me to log in again.

2023-08-09 18:26:02.667 WARNING (SyncWorker_4) [verisure.session] Error from https://automation02.verisure.com: 401 - {"errorGroup": "UNAUTHORIZED","errorCode": "AUT_00001","errorMessage": "
Session has expired"}                                                                          
2023-08-09 18:26:02.672 ERROR (MainThread) [custom_components.verisure] 3 Could not log in to verisure, Login error, status code: 401 - Data: {"errorGroup": "UNAUTHORIZED","errorCode": "AUT_
00001","errorMessage": "Session has expired"}                                                                                                                                                 
2023-08-09 18:26:02.679 ERROR (MainThread) [custom_components.verisure] Authentication failed while fetching verisure data: Credentials expired for Verisure
2023-08-09 18:26:02.680 DEBUG (MainThread) [custom_components.verisure] Finished fetching verisure data in 0.283 seconds (success: False)
2023-08-09 18:26:02.685 DEBUG (MainThread) [custom_components.verisure] Running async_step_reauth,                                                                                            
2023-08-09 18:26:02.685 DEBUG (MainThread) [custom_components.verisure] Running async_step_reauth_confirm,

Now we wait to see what happens next time the timeout occurs.

(I don't really like this text-matching but it is currently the best I have).

[niro1987]

So if the first url returns 401, so does the second. Yes the message is different, but the result is the same. That makes this pr irrelevant.

I think the issue happens because we don't "trust" the device when we login. So after some time we loose trust between us and the server, requiring a new authentication to take place.

[Olen]

The point is that there are different 401s and they should be treated differently.

At least for now, it looks like "Session is expired" and "Invalid username/password/authentication method combination" should raise a "LoginError" (these indicate that the username/password is wrong) whereas "Username/password does not match any valid login" probably should raise a "ResponseError" as we can mos likely log in again with the same username and password (waiting for verification of that).

There might be others, but those are the three different I have seen so far.

[niro1987]

I'll try to figure out how we can 'trust' the device when we login.

If you're interested to try for yourself, install mitmproxy on you machine and install it's certificate on your phone. Set the up of your machine as proxy in your WiFi settings on your phone. Now you should be able to view the traffic.

[persandstrom]

Trusted device was implemented in version 1 of python Verisure. Don't know if it helps. I guess it's quite a quick fix and I think you mighty be on to something here.

https://github.com/persandstrom/python-verisure/blob/2df6979bd0c7ca0e879bbac723fc10a66a85458e/verisure/session.py#L159C12-L159C12

[Olen]

That only seems to be related to mfa. I have disabled mfa for the account that I use in HA, so I am not sure trusting a device is possible then?

[niro1987]

Let's find out, trust is similar to [ ] remember me.

[Olen]

I think now that maybe that is the way to go. If "trust" can be used also without MFA. I have misunderstood something about how HA handles the different exceptions. I understand that after a LoginError in Verisure - which then triggers a ConfigEntryAuthFailed in HA - HA will show the dialog to enter username and password again. But then I thought that after a ResponseError from Verisure, HA would try to do a reauth automatically with the same credentials. But that does not seem to be the case.

What is needed is either the "trust" fix, OR a way to make HA retry a login with the same username/password after a certain 401. I made a naïve patch to config_flow.py that passes the existing username and password as "user_input" to async_step_reauth_confirm from async_step_reauth, and that works as long as the password has not been changed.

[niro1987]

@Olen and everyone who reads this

Can you please assist in testing #172 on home-assistant/core#98258

To do so, you have to:

1. setup a dev environment from https://github.com/niro1987/homeassistant-core/tree/verisure_trust, or you can just open a Github COdespace from home-assistant/core#98258
2. run pip install git+https://github.com/niro1987/python-verisure.git@trust#vsure==2.6.5 --target ~/.homeassistant/deps in the dev environment, in accordance with the developer docs
3. run pip install git+https://github.com/niro1987/python-verisure.git@trust#vsure==2.6.5 (without the --target bit).
4. Start Home Assistant in your dev environment with hass --debug -c config --skip-pip-packages vsure
5. Login to Verisure and leave it running for a few days

Optionally, you can add some extra print statements to /home/vscode/.local/lib/python3.11/site-packages/verisure/session.py, for example by adding this below line 105.

print(f"{response.request.url} {response.status_code} {response.text}")

If the changes work as intended, you should see a request to /auth/login every few hours. If you've added the extra print statement that is.

[Olen]

But does it need to be related to mfa? I don't have mfa (and are reluctant to add it as I also use that account for another script that might not play well with mfa).

[niro1987]

Good question, I'll try to grab some traffic with MFA disabled.

[niro1987]

The change to the integration could still fix it for you. As the integration now attempts to reauthenticate on 4** responses.