AWS new authorization (Worx connection blocked, App operates normal, waiting does not resolve)

[LukasA83]

Hi,

Since around 1 week I got the blocked message in the binding. I tried to wait 48 hours and also remove and add back the mower to my worx account without success. The Worx app operates normally.

Here is the log: 2022-10-21 10:20:42.915 [ERROR] [d.internal.WorxLandroidBridgeHandler] - Connection blocked from Worx, please try again in 24h 2022-10-21 10:22:50.060 [DEBUG] [nternal.webapi.request.WebApiRequest] - URI: https://id.eu.worx.com/oauth/token 2022-10-21 10:22:50.692 [DEBUG] [nternal.webapi.request.WebApiRequest] - Worx Landroid WebApi Response: {"token_type":"Bearer","expires_in":3600,"access_token":"***hidden for debug log***","refresh_token":"***hidden for debug log***"} 2022-10-21 10:22:50.696 [DEBUG] [nternal.webapi.request.WebApiRequest] - URI: https://api.worxlandroid.com/api/v2/users/me 2022-10-21 10:22:50.858 [DEBUG] [nternal.webapi.request.WebApiRequest] - Worx Landroid WebApi Response: {"id":267139,"user_type":"customer","push_notifications":true,"location":null,"mqtt_endpoint":"a1optpg91s0ydf-ats.iot.eu-west-1.amazonaws.com","actions_on_google_pin_code":null,"created_at":"2020-03-27 20:33:54","updated_at":"2022-10-21 04:45:35"} 2022-10-21 10:22:50.874 [INFO ] [d.internal.WorxLandroidBridgeHandler] - Start retrieving AWS certificate 2022-10-21 10:22:50.877 [DEBUG] [nternal.webapi.request.WebApiRequest] - URI: https://api.worxlandroid.com/api/v2/users/certificate 2022-10-21 10:22:51.024 [DEBUG] [nternal.webapi.request.WebApiRequest] - Worx Landroid WebApi Response: {"id":160983,"pkcs12":"***hidden for debug log***","active":false,"created_at":"2020-04-10 11:21:16","updated_at":"2020-04-10 11:21:16"} 2022-10-21 10:22:51.034 [ERROR] [d.internal.WorxLandroidBridgeHandler] - Connection blocked from Worx, please try again in 24h 2022-10-21 10:23:43.735 [DEBUG] [nternal.webapi.request.WebApiRequest] - URI: https://id.eu.worx.com/oauth/token 2022-10-21 10:23:44.169 [DEBUG] [nternal.webapi.request.WebApiRequest] - Worx Landroid WebApi Response: {"token_type":"Bearer","expires_in":3600,"access_token":"***hidden for debug log***","refresh_token":"***hidden for debug log***"} 2022-10-21 10:23:44.175 [DEBUG] [nternal.webapi.request.WebApiRequest] - URI: https://api.worxlandroid.com/api/v2/users/me 2022-10-21 10:23:44.236 [DEBUG] [nternal.webapi.request.WebApiRequest] - Worx Landroid WebApi Response: {"id":267139,"user_type":"customer","push_notifications":true,"location":null,"mqtt_endpoint":"a1optpg91s0ydf-ats.iot.eu-west-1.amazonaws.com","actions_on_google_pin_code":null,"created_at":"2020-03-27 20:33:54","updated_at":"2022-10-21 04:45:35"} 2022-10-21 10:23:44.251 [INFO ] [d.internal.WorxLandroidBridgeHandler] - Start retrieving AWS certificate 2022-10-21 10:23:44.254 [DEBUG] [nternal.webapi.request.WebApiRequest] - URI: https://api.worxlandroid.com/api/v2/users/certificate 2022-10-21 10:23:44.387 [DEBUG] [nternal.webapi.request.WebApiRequest] - Worx Landroid WebApi Response: {"id":160983,"pkcs12":"***hidden for debug log***","active":false,"created_at":"2020-04-10 11:21:16","updated_at":"2020-04-10 11:21:16"} 2022-10-21 10:23:44.459 [ERROR] [d.internal.WorxLandroidBridgeHandler] - Connection blocked from Worx, please try again in 24h

Any idea?

[sihui62]

It could be that you are blocked for 24 hours.

You were right, after having the Bridge Thing disabled for 2 days it came online again today. Thx a lot.

[sihui62]

@nibi79

Is there any chance to make the binding work with openHAB 4.x?

I found the following parameters which should make the binding compatible with 3.2 through 4.0:

https://community.openhab.org/t/openhab-4-0-milestone-discussion/145133/46

If the developer compiles the jar with the following options it will work on 3.2.0 through 4.0.0 -Dohc.version=3.2.0 -Dmaven.compiler.source=11 -Dmaven.compiler.target=11 -Dkaraf.version=4.3.7 EDIT: you also need to readd the binding.xml file (dont delete the addon.xml)

[nibi79]

Is there any chance to make the binding work with openHAB 4.x?

Sorry for the late reply. I tried it briefly, but didn't get it to work quickly. Unfortunately, I currently have too little time, but I will keep trying.

[maho-cpu]

@nibi79 v3.4.1-beta binding works for OpenHAB 3.4.0 too - just to let you know

[foobar26]

Please keep monitoring and report if it occurs again.

Unfortunately it stopped working completely. It does not help do disable/reenable the Bridge Thing or create a new Bridge Thing. :-(

2023-03-11 15:21:19.228 [DEBUG] [d.internal.WorxLandroidBridgeHandler] - try to connect to AWS...
2023-03-11 15:21:19.637 [ERROR] [worxlandroid.internal.mqtt.AWSClient] - Exception: software.amazon.awssdk.crt.mqtt.MqttException: The connection was closed unexpectedly.
2023-03-11 15:21:19.637 [DEBUG] [d.internal.WorxLandroidBridgeHandler] - AWS connected: false

For me the same happens yesterday. After 2-3 days of normal operation it stopped working. Also app doesn't get any connection now. I will disable the binding and see what happens after two days. With the old version and the pollinginterval of 3600s I never got blocked. I also didn't use the refresh so often. Therefore it's unclear for me why I got blocked this time :-( Hope this doesn't happen too often.

[sihui62]

After 2-3 days of normal operation it stopped working. Also app doesn't get any connection now.

Same here again. I remember having beeing blocked last year once with the "old" binding and connection method, the Android app still worked. So maybe this time it was a problem on the server side? Everything is working again at least here.

[PilotIoTSystems]

I looked at it now and the binding was offline. I disabled it, then re-enabled it, and now it's online again.

[nibi79]

My binding has been running continuously for 2 weeks now. Again, the problem is that it uses a non-public and undocumented interface. In addition, Worx/Positec is not helpful and unfortunately not interested in supporting me (and others). I don't know the reason for the blocking and what is happening in the backend at Positec. Therefore, for now, I am just glad that the binding is working at all.

[sihui62]

Therefore, for now, I am just glad that the binding is working at all.

Thx for all the time you are investing.

[foobar26]

Thank you very much for putting so much time and effort into continuously supporting the binding! My ban was lifted after I disabled the binding for 24 hours. Now it's working again.

[nibi79]

My binding has been running continuously for 2 weeks now.

Said it and now it stopped working last night.

Unfortunately I don't see the reason in the logs anymore, but after I simply reactivated the bridge, it worked again. So I think I wasn't blocked.

Currently, I don't know what triggers the blocking and how to prevent it, but perhaps I can implement another workaround and catch other exceptions (beside software.amazon.awssdk.crt.mqtt.MqttException: The connection was closed unexpectedly.) and simply try to reconnect one time.

[mitch-geht-ab]

@nibi79 when you reconnect, do you do a new name resoltution to the mqtt endpoint? I could imagine that AWS is using microservices, containers and global loadbalancing (DNS) very excessive (of course, it's AWS! ;)) That would explain why for some the connections are stable for ~24h and for others ~2w. Depends of if the container is still running.

An indicator could be the small TTL in the DNS:

$ dig a1optpg91s0ydf-ats.iot.eu-west-1.amazonaws.com
...
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22574
;; flags: qr rd ra; QUERY: 1, ANSWER: 8, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;a1optpg91s0ydf-ats.iot.eu-west-1.amazonaws.com.        IN A

;; ANSWER SECTION:
a1optpg91s0ydf-ats.iot.eu-west-1.amazonaws.com. 57 IN A 52.18.50.213
a1optpg91s0ydf-ats.iot.eu-west-1.amazonaws.com. 57 IN A 52.209.219.92
a1optpg91s0ydf-ats.iot.eu-west-1.amazonaws.com. 57 IN A 52.16.209.178
a1optpg91s0ydf-ats.iot.eu-west-1.amazonaws.com. 57 IN A 34.240.135.192
a1optpg91s0ydf-ats.iot.eu-west-1.amazonaws.com. 57 IN A 52.212.12.89
a1optpg91s0ydf-ats.iot.eu-west-1.amazonaws.com. 57 IN A 52.212.188.45
a1optpg91s0ydf-ats.iot.eu-west-1.amazonaws.com. 57 IN A 34.241.133.245
a1optpg91s0ydf-ats.iot.eu-west-1.amazonaws.com. 57 IN A 34.249.234.130

Is there a lifetime reported in any of the response bodys that could give you a hint of how often do a reconnect? Maybe you have to do a reconnect in a proactive way (including a new nameresolution) maybe every 1d.

BE mitch

[sihui62]

Unfortunately I had to stop using the binding: after around two days of correct operation my account gets blocked and also the Android app isn't working anymore ("Connection error").

[Incom0de]

Same behaviour on my side. Binding Was working for some days. Today binding was offline. Reactivation isn't working, stays on "initialize". Android App is also not working."Connection error". I disabled the bridge and will check if android app will work again tomorrow.

[PilotIoTSystems]

"Today binding was offline. Reactivation isn't working, stays on "initialize"." The same thing happened to me...

[nibi79]

My binding is still running. But now I'm really at a loss. Sorry!

[cinadr]

Me too. Binding stays at initializing state and android app says "Connection error". Are we banned for 3rd party software usage? I opened a ticket at Worx support.

@nibi79 is this related? (Duplicate client IDs?) https://github.com/aws/aws-iot-device-sdk-cpp-v2/issues/110

@Incom0de:

Same behaviour on my side. Binding Was working for some days. Today binding was offline. Reactivation isn't working, stays on "initialize". Android App is also not working."Connection error". I disabled the bridge and will check if android app will work again tomorrow.

Is your app started to working or stays at Connection Error? Does your mower do its job?

[Incom0de]

@cinadr I deactivated the bridge and on next day App was working again. Since then I have not activated the bridge. I opened a ticket at Worx support last year when the binding stopped working. My experience was that Worx will close the ticket because the do not support the API and doesn't care that Customer want to use other software than the App to manage the mower...

Me too. Binding stays at initializing state and android app says "Connection error". Are we banned for 3rd party software usage? I opened a ticket at Worx support.

Same behaviour on my side. Binding Was working for some days. Today binding was offline. Reactivation isn't working, stays on "initialize". Android App is also not working."Connection error". I disabled the bridge and will check if android app will work again tomorrow.

Is your app started to working or stays at Connection Error? Does your mower do its job?

[cinadr]

@Incom0de: Thank you. I'll disable the binding while this is going to be resolved.

I found some info on the DesktopApp forum which explains this somehow but .net isn't JAVA so I'm not sure if this is even related (Google Translate):

There is a parallel exchange between the few dotnet developers in the Positec beta github. With .Net framework 4.8 it only works for 15 minutes and no proper disconnect. With dotnet 6 I have it running with 1h (token duration) and reconnect in a test app. I am now trying to switch the DeskApp to dotnet 6. I don't need any help with that. If there is anything to share or problems arise, I'll get back to you.

Original post is here.

[PilotIoTSystems]

Hi cinadr (Kedves Zsolt!), I will summarize to the best of my knowledge what happened, that the binding is currently not working...

1. If you query the Worx MQTT server (AWS) too often (too much), the server will be disabled for 24 hours. In this case, all queries must be stopped, because the ban will not be lifted!
2. Last year, Positec (which develops the Worx software) changed the method of connection (authorization) to the Worx MQTT server (AWS)
3. For questions about the software, you should contact Positec, not Worx. In principle, they do not ban Worx developments, but lately they have not been very helpful.
4. niby79 is working hard to get the binding working again, but he is not getting help from Positech.

Please, nibi79, correct me if I summarized what happened incorrectly.

A few days ago (after the binding stopped working) I restarted it, but days later it stopped working again...

[nibi79]

I found a bug in my implementation that triggered too many reconnections. I will provide you a new version for testing later.

[nibi79]

@PilotIoTSystems you are correct, maybe the new version will improve point 1

[cinadr]

1. If you query the Worx MQTT server (AWS) too often (too much), the server will be disabled for 24 hours. In this case, all queries must be stopped, because the ban will not be lifted!

@PilotIoTSystems Do I have to log out from the Android App as well or is it OK to disable the binding in openHAB only? Thank you for your clarification.

@nibi79 I'm waiting for the test version. Thank you. 🥇

[PilotIoTSystems]

@PilotIoTSystems Do I have to log out from the Android App as well or is it OK to disable the binding in openHAB only? Thank you for your clarification.

Disabling the binding was enough for me (but I rarely use the Android app)

[foobar26]

@PilotIoTSystems Do I have to log out from the Android App as well or is it OK to disable the binding in openHAB only? Thank you for your clarification.

Disabling the binding was enough for me (but I rarely use the Android app)

Same for me. Just disabling the binding is enough. @nibi79 great to hear you found a bug and many thanks for providing a fix. After the comment about Positec supporting Iobroker I was wonderig why they would disallow other home automizition apps like Openhab. But if there are too many requests by the binding to Amazon, it's clear why they block access completely as this might get expensive for them.

[foobar26]

Just saw that on Iobroker GitHub they are discussing s similar issue (unfortunately only German): https://github.com/iobroker-community-adapters/ioBroker.worx/issues/656

[nibi79]

Maybe this version is a little bit more stable: https://github.com/nibi79/worxlandroid/releases/tag/v3.4.1-beta

Let me know!

[cinadr]

Maybe this version is a little bit more stable: https://github.com/nibi79/worxlandroid/releases/tag/v3.4.1-beta

Let me know!

Do we need to wait 24h to expire after former ban to test this?

[nibi79]

Do we need to wait 24h to expire after former ban to test this?

yes definitely

[nibi79]

and as I said

Maybe this version is a little bit more stable:

so please keep in mind that this version will not solve all problems. Don't expect too much and don't be disappointed!

[PilotIoTSystems]

I uploaded the new beta binding. Now online! So far everything is OK!

[nibi79]

unfortunately i got blocked again

[cinadr]

I've got this from worx support:

Hello,

Thank you for contacting us. Yes, probably you are blocked, because you are using 3rd party software, we don't recommend using it. Have a nice day.

Regards, Elena from WorxTeam.

[sihui62]

unfortunately i got blocked again

Mine is still running after around 18 hours of operation. I implemented an ugly workaround now: the Bridge and Mower Things are disabled via cron rule when the mower is not in operation, so these will only be enabled in the afternoon between 13 and 17 o'clock. Maybe this way extends the time until I got blocked again ....

[Incom0de]

unfortunately i got blocked again

Me Too. Disabled the binding 5 Minuten ago because it was offline :(

[PilotIoTSystems]

Online since yesterday (about 30 hours ago)... OH ver.3.3.0

[PilotIoTSystems]

Unfortunately, the binding is offline after 3 days... "AWS connection closed" I restarted it and it's online now...

[nibi79]

I restarted it and it's online now...

So you weren't blocked? Did you receive an MQTT exception or something else?

[PilotIoTSystems]

No, there isn't, and I don't think it was disabled either! I just looked at it today. Also online now. I only got the message "AWS connection closed"!

[mitch-geht-ab]

With the new binding, I'm disabled after ~18h . 1.4. 20:00 enabled... 2.4. ~14:00 killed with:

2023-04-02 13:58:47.229 [ERROR] [worxlandroid.internal.mqtt.AWSClient] - Exception: software.amazon.awssdk.crt.mqtt.MqttException: The connection was closed unexpectedly.

My mower is still in the cellar. Is there maybe a relation between active mower and mower with no new status, because of winter sleep?

[PilotIoTSystems]

My binding is online now! refreshStatusInterval=1200, pollingInterval=599

[foobar26]

For me even with the new version I get blocked more or less after 24/36h. Happened just again. Then have to disable the plugin for 24h. After that it works again. On iobroker they implemented an automated disconnect/reconnect to make it at least a little more convenient. Would appreciate a similar feature. Thanks again for your support.

[nibi79]

On iobroker they implemented an automated disconnect/reconnect to make it at least a little more convenient. Would appreciate a similar feature.

Do you mean that after the exception/lock, the binding waits for 24 hours and then restores the connection itself? I had this idea too, I will take a look at it.

[foobar26]

On iobroker they implemented an automated disconnect/reconnect to make it at least a little more convenient. Would appreciate a similar feature.

Do you mean that after the exception/lock, the binding waits for 24 hours and then restores the connection itself? I had this idea too, I will take a look at it.

Yes exactly. It somehow detects the temporary blockage and disables the connection retry for a configruable time span, eg 24h and afterwards connects again. Don't know how easy it is to detect the issue, because there could also some temporary internet loss or other reasons why you can't reach AWS.

[cinadr]

The Android App itself is not blocked so the binding should follow it's behavior as close as possible. I've made no research on this but what comes to me is that maybe the app identifies itself with some private id's that it is internal to it. (Some WireShark?) So this ID should be used if it exists (probably different for every user, or generated from the serial number of the mower, etc?)
Another option is to limit updates seriously and probably make queries/updates only by manual refresh or by every hour or so. And it should only update its status more often at next scheduled run of the mower. A manual start of the mower should trigger this too for an hour or two. Is there any information how much query can happen during an hour or for the next 24 hours? If it is known the binding should display a current counter how many queries have left. I'm not a java developer. I'm just thinking of some possible solutions here.

[PilotIoTSystems]

Another option is to limit updates seriously and probably make queries/updates only by manual refresh or by every hour or so.

@cinadr We are over that, these are adjustable values! For example, I have: refreshStatusInterval=1200, pollingInterval=599

[cinadr]

@PilotIoTSystems Hi! I'm aware of that from your previous post. I'm talking of a dynamic update system that follows a pattern of rare updates while the mower is not in use but a more regular update when the mowing is scheduled. Similar to the mobile app behavior.

[PilotIoTSystems]

@PilotIoTSystems Hi! I'm aware of that from your previous post. I'm talking of a dynamic update system that follows a pattern of rare updates while the mower is not in use but a more regular update when the mowing is scheduled. Similar to the mobile app behavior.

It's the same now. If you change any parameter, it will be synchronized on the AWS server. If there is no change, the status query is done, for example, every 20 minutes.

[foobar26]

@nibi79 maybe a dumb question: why is the max for polling interval 599 now? In a previous version of the binding it could be set to a far larger value. Is the connection lost or dropped from AWS side after that time and you cannot detect it otherwise?

[foobar26]

@nibi79 maybe a dumb question: why is the max for polling interval 599 now? In a previous version of the binding it could be set to a far larger value. Is the connection lost or dropped from AWS side after that time and you cannot detect it otherwise?

Ok this was dumb. In the config it's stated. Why not just opening a new connection after 10 minutes have passed instead of polling?