Only sporadically receiving activity updates from locks

[spraot]

When I restart the service using yalexs, the correct state is found. After that lots of pubnub activity can be seen in the logs, but my service mostly doesn't receive any notification of lock activity, though the rare event does slip through (at most 1 per day).

This has been ongoing for a couple months and may have started when the new Yale app came out. There haven't been any changes to code in the meantime and I've tried downgrading yalexs to a version I used last spring when I believe things were working.

Has this issue popped up anywhere else? Any tips on where to look to debug this? Thanks!

[bdraco]

Yale added some type of keep alive requirement fro the global servers only, but we don't know how to fulfill it

https://github.com/home-assistant/core/issues/100798

[spraot]

I see! I assume someone has already tried analyzing the traffic from the app?

Brute force checking for new activity every 30s would work failing anything else, wouldn't it?

[bdraco]

I see! I assume someone has already tried analyzing the traffic from the app?

Nope, the user who was helping me with this is no longer available so its waiting for someone who has a device outside of North America to analyze the traffic

Brute force checking for new activity every 30s would work failing anything else, wouldn't it?

That seems like it would result in Yale cutting of access to everyone for abusing their servers

[spraot]

I have a rooted Android I use for intercepting traffic, but I'm not very experienced at this. Any tips?

[Dejniel]

I think this issue may not be related to any API changes. I haven't been able to load newer lock logs since my token unexpectedly expired on November 15, 2023 There appear to be no corresponding changes in the official app. I hypothesize that they may be implementing a client shadowban in response to unusual query patterns or certain types of headers ;/

[spraot]

I notice that the actual event subscription happens through pubnub, so it could be that any changes (or shadowban) is a documented part of the pubnub API. I had a quick look at their documentation, but I haven't spotted anything relevant yet.

[Dejniel]

I tried updating the API key and other headers, but to no avail. There are no new API queries in the app related to client validation or sync. Queries to the endpoints /augustappversionok/android/{version} and /appfeatures/android/{version} don't make any difference.

[illuzn]

@bdraco I think I have found the relevant request....

approximately every 30s my device is sending a keepalive request as follows.

HTTP/2.0 PUT https://api.aaecosystem.com/remoteoperate/[32 DIGIT BASE16 LOCK ID]/STATUS?type=async&connection=persistent&intent=keepalive&v=2.4.0&sn=[mfgBridgeID]

x-api-key: [8 base16 digits]-[4 base16 digits]-[4 base16 digits]-[4 base16 digits]-[12 base16 digits]
user-agent: yale/Luna-2023.8.0 (Android; SDK33; [Phone Identifier]
x-branding: yale
x-country AU
accept-version: 2.0.0
x-access-token: [Access Token]
content-type: application/json; charset=UTF-8
content length: 84
accept-encoding: gzip

The json payload is:

{
  "iv": "[32 base16 digits]"
  "token": "[32 base16 digits]"

I have no idea what iv and token are (hoping you do) but they seem to be unique every time. They may not actually matter because replaying old requests still nets me with the same response.

A successful request results in a HTTP/2.0 202 response a failed request results in a HTTP/2.0 401 response.

Hopefully I haven't just stuffed up and doxxed myself or worse by not sanitising this properly.

[bdraco]

Sadly I don't know what the iv and token are. I'll see if I can find out

[illuzn]

If anyone wants a temporary workaround, this automation hits the wake button at XX:XX:50 (basically once a minute). I've found that the API disconnects after 1 minute if 2 keep alive windows I've mentioned above pass without a keep alive request.

This also doesn't hammer the API if the device is unknown or unavailable to hopefully avoid you getting banned.

This may impact battery life (too early to tell). Will report back with battery life statistics.

alias: Door keep alive
description: ""
trigger:
  - platform: time_pattern
    seconds: "50"
condition: []
action:
  - if:
      - condition: not
        conditions:
          - condition: state
            entity_id: binary_sensor.front_door_open_august
            state: unavailable
          - condition: state
            entity_id: binary_sensor.front_door_open_august
            state: unknown
    then:
      - service: button.press
        target:
          entity_id: button.front_door_wake
        data: {}
mode: single

[kmdm]

@bdraco

So this APPEARS to be the close correct implementation on a quick-pass over the decompiled Yale Home APK:

import binascii
import secrets
import sys
import time

from Cryptodome.Cipher import AES
from Cryptodome.Util.Padding import pad

# ...<<SNIP yalexs imports and authenticating, etc...>>...

locks = api.get_locks(authentication.access_token)
mylock = locks[0]
lock_id = mylock.device_id
mylock_details = api.get_lock_detail(authentication.access_token, lock_id)

ros = bytes(mylock_details.raw.get('remoteOperateSecret'), "ascii")

# iv = "RAND_16_BYTES"
# encrypt currentTimeMillis() with ROS

iv = secrets.token_bytes(16)
ms = bytes(f"{int(time.time() * 1000 // 1)}", "ascii")

cipher = AES.new(ros, AES.MODE_CBC, iv)
token = cipher.encrypt(pad(ms, 16))

str_token = str(binascii.hexlify(token))
str_iv = str(binascii.hexlify(iv))

print(f"ros={ros}, iv={str_iv}, token={str_token}")

# send the remoteoperate keepalive
API_STATUS_ASYNC_KEEPALIVE_URL = '/remoteoperate/{lock_id}/STATUS?type=async&connection=persistent&intent=keepalive&v=2.4.0'
req = {
    "method": "put",
    "url": api.get_brand_url(API_STATUS_ASYNC_KEEPALIVE_URL.format(lock_id=lock_id)),
    "access_token": authentication.access_token,
    "timeout": 5,
    "json": {
        "iv": str_iv,
        "token": str_token,
    }
}

res = api._dict_to_api(req)
print(f"resp_status_code={res.status_code}\n")

However: I get a 202 response if I don't send any json payload in the request so that may well be sufficient (although this could also mean operations may require a valid token in future :shrug:)

NB: I've spent very little time on this - apologies for any ugly code :joy:

[kmdm]

Also, tentatively, running this request in a loop every approx 30s (like @illuzn said the app was doing) seems to be enough to fix updates to HA without updating the library/component in HA.

[illuzn]

This is probably the missing link given we didn't know how to derive iv and token before. Great work.

[kmdm]

@illuzn

For the moment at least, it does seem to work without iv/token.

Probably the best way to verify the method is correct is if you can decrypt the token you captured (or I'm happy to try if you're happy to DM the details of your capture and the remoteOperateSecret (+iv & token)

[bdraco]

It appears the keep alive was put in place to reduce load on the Yale servers. If we start sending them its likely going to have a negative impact for Yale which is something we want to avoid. I am pursing an alternate solution. It will likely be a few weeks before the next update.

[Dejniel]

For the moment at least, it does seem to work without iv/token.

Probably the best way to verify the method is correct is if you can decrypt the token you captured (or I'm happy to try if you're happy to DM the details of your capture and the remoteOperateSecret (+iv & token)

@kmdm Are we talking about the same issue? My problem is that since November 15, 2023, there has been no new activity on the get_house_activities list. The STATUS doesn't seem to have anything to do with the lack of updates to the activity list. The STATUS as a keepalive was not added in the same period when the activity list stopped being updated. Nevertheless, I have tested sending the STATUS request with the correct remoteOperateSecret in various configurations, and it changes nothing; the activity list is still empty. I should mention that on the same account, the activity list is visible in the official Yale Home app. I've analyzed the code changes in the Yale Home app and don't see any changes that could affect such behavior, so I suspect a shadowban.

[Kaajink]

Team,

I am not sure whether the problem stated above exhibits itself in the scenario below. If so please let me know how I can assist. I am based in NZ and have to Yale Y226 locks. One with August WIFI module and the other one with Yale WIFI module. Last year we were forced to upgrade the IOS APP from August to Yale and I recall the experience was pretty bad. I have an automation setup to disarm the alarm if it is armed when I unlock the door from outside using the pinpad. I also have another automation to turn the lights on when I unlock the door. I noticed both of them were working although I am sure they worked last year but I really dir not pay attention. I got some time yesterday to test and if I was to unlock the door using the Yale App or via HA dashboard the automation works however manually doing it with the pinpad the automation does not work. Looking at the Developer Tools -Entity status I noticed that the status never changes when manually unlocking the door however it does change when doing it via the App. Is this scenario related to the issue in this thread?

[bdraco]

Its a known issue with the servers outside of North America.

We are getting closer to a solution but will probably still be a few months before its rolled out.

[Bogdis]

Any progress on this issue? having same problem to with Yale Linus in Sweden and August integration HA 2024.6.4

[bdraco]

Yes progress is being made

[lesgrebe]

+1 same behavior with Yale Linus in Austria/Europe and August integration in HA 2024.8.1

It was working fine until a few days ago. One solution was the wakeup, which I´ve only used when the door gets closed for 2 seconds to keep the traffic to yale/august low (I´m using an additional door sensor from Bosch Smart Home to check the state from the door). Now the wakeup "feature" no longer works and the state keeps only updated in Home Assistant if the Yale app remains open on the phone (iPhone 13Pro).

Thx. lot for helping us with this problem and I hope there will be an update to use my be loved automations in HA again ;)