search

JanLoebel / eufy-node-client

Experiment to talk to eufy security
43 stars 6 forks source link

Push notification no longer works correctly #9

Closed bropat closed 3 years ago

bropat commented 3 years ago

As of today, I have noticed that the push notification no longer works correctly. Before i received the payload correctly and now it is missing:

{"notification":{"from":"348804314802","priority":"normal"},"persistentId":"<id>"}

Do you have the same behavior?

JanLoebel commented 3 years ago

Same behaviour here.. very strange.. normally that means that they have changed the APP_ID oder APP_SENDER_ID. I will have to check that.

JanLoebel commented 3 years ago

Okay, they have changed the APP_ID now the part after the "1:" match the SENDER_ID, but even with this new APP_ID I will receive only that mini-notification. They seems also to have changed the code regarding push notifications. I will have to look further.

bropat commented 3 years ago

Okay, they have changed the APP_ID now the part after the "1:" match the SENDER_ID, but even with this new APP_ID I will receive only that mini-notification. They seems also to have changed the code regarding push notifications. I will have to look further.

The strange thing is that i have an older version of the EufySecurity App in my test environment (Bluestacks) where i receive the push notifications as before, instead on nodejs i see the mini-notification at the same time. This makes me think that the code change isn't on EufySecurity App side, but maybe at FCM side that the push-receiver module doesn't support. It's a hypothesis. Maybe we can work together to find a solution?

JanLoebel commented 3 years ago

Yeah I think so too, I've tested other APP_IDs (random/small modifications/etc..) and on each I only got the messages without any data. Of course we could work on that together! Currently I have a lot to do with other private things, so I would need help to get further here. A first step could be to setup a small test setup by creating a firebase test application and use push-receiver to see if that would receive all information.

bropat commented 3 years ago

I have set up a test firebase app and tested the notification with push-receiver. I have successfully received the notification with push-receiver:

Notification received
{
  data: {
    'gcm.n.e': '1',
    'google.c.a.ts': '1603952956',
    'google.c.a.udt': '0',
    'google.c.a.e': '1',
    'google.c.a.c_id': '<id>'
  },
  from: '<from>',
  priority: 'high',
  notification: {
    title: 'Test notification',
    body: 'This is a test notification.',
    tag: 'campaign_collapse_key_<id>'
  },
  collapse_key: 'campaign_collapse_key_<id>'
}

So the problem lies elsewhere...

bropat commented 3 years ago

I have dug deeper and I think Eufy have migrated to the new FCM HTTP v1 API (https://firebase.google.com/docs/cloud-messaging/migrate-v1).

The module "push-receiver" does not support the new API, because it uses:

https://fcm.googleapis.com/fcm/send

The new endpoint is:

https://fcm.googleapis.com/v1/projects/myproject-b5ae1/messages:send

If you read the documentation of the new API at https://firebase.google.com/docs/cloud-messaging/migrate-v1, they say:

More efficient customization of messages across platforms For the message body, the HTTP v1 API has common keys that go to all targeted instances, plus platform-specific keys that let you customize the message across platforms. This allows you to create "overrides" that send slightly different payloads to different client platforms in a single message.

The API key was correctly restricted by Eufy on their Android app. I checked this by creating a test project and configuring it with messagingSenderId, apiKey, projectId and appId from Eufy (taken from the APK). When fetching the token I correctly get this error:

FirebaseError: Installations: Create Installation request failed with error "403 PERMISSION_DENIED: Requests from this Android client application are blocked. (installations/requestst-failed).

To work around this we need to add the following headers to the request:

X-Android-Package: <android_package_name> X-Android-Cert: <sha1_fingerprint_apk>

Both values can also be taken from the APK.

If the push receiver is rewritten to support the new API and contains these headers (which can be set as parameters somewhere), we should get the correct payload again.

What do you think?

JanLoebel commented 3 years ago

Great catch @bropat ! We could of course try to add the headers to push-receiver. Can you guide me how to find the sha1_fingerprint_apk? I could then try it at the weekend. Currently I have too much to do for a client, sorry :(

bropat commented 3 years ago

The simplest way is to use JADX, select the APK and then scroll down to APK signature and get the SHA-1 Fingerprint. The value for X-Android-Cert must be without ":" or spaces. Don't worry, I understand ;) Let me know if you need anything else :)

JanLoebel commented 3 years ago

@bropat okay, I took my break and tried what you've found.

I've added following values to the header (file: 'eufy-node-client/node_modules/push-receiver/src/fcm/index.js'): 'X-Android-Package': 'com.oceanwing.battery.cam', 'X-Android-Cert': 'F051262F9F99B638F3C76DE349830638555B4A0A',

Furthermore I've changed the 'FCM_ENDPOINT' to 'https://fcm.googleapis.com/v1/projects/batterycam-3250a/messages:send' in the same file.

Afterwards the registration seems to work, but I could not get any push-notification afterwards. I've reverted the 'FCM_ENDPOINT' to 'https://fcm.googleapis.com/fcm/send' and tried again, same problem as before, I just got the mini notifications without any data. So far no luck.. I could not find any other library that receives push notifications from fcm, so its hard to say how it should work. So perhaps we have to reengineer the new FCM mechanism.

bropat commented 3 years ago

@JanLoebel would have been too easy ;)

In the meantime I managed to get a push notification with Firebase in the browser. Unfortunately always only the shortened version. But here I used the official Firebase Javascript libraries (an old version 6.0.1, that uses the old fcm legacy calls).

image

The firebase application was initialized with the following values (see also attached project):

firebase.initializeApp({
    messagingSenderId: "348804314802",
    apiKey: "AIzaSyCSz1uxGrHXsEktm7O3_wv-uLGpC9BvXR8",
    projectId: "batterycam-3250a",
    appId: "1:348804314802:android:440a6773b3620da7",
});

To test the attached Firebase project (testfirebase.zip) do the following:

  1. Install the Firebase CLI as described here: https://firebase.google.com/docs/cli
  2. Extract the Firebase project somewhere
  3. Go to the extracted Firebase project and execute:
firebase emulators:start --only hosting
  1. Open a browser an go to http://localhost:5000 (enable notification for this page in the browser!)
  2. Register the FCM Token on Eufy (https://security-app-eu.eufylife.com/v1/apppush/register_push_token)
  3. Generate a notification with EufySecurity App.

PS: Same behaviour with latest Firebase version (8.0.0).

bropat commented 3 years ago

I did some more research, and the new FCM API calls I found are these:

  1. Register the new Firebase Installation (register the device):

    Request:

    POST /v1/projects/batterycam-3250a/installations HTTP/1.1
Content-Type: application/json
Accept: application/json
X-Android-Package: com.oceanwing.battery.cam
x-firebase-client: fire-abt/17.1.1 fire-installations/16.3.1 fire-android/ fire-analytics/17.4.2 fire-iid/20.2.0 fire-rc/17.0.0 fire-fcm/20.2.0 fire-cls/17.0.0 fire-cls-ndk/17.0.0 fire-core/19.3.0
x-firebase-client-log-type: 3
X-Android-Cert: F051262F9F99B638F3C76DE349830638555B4A0A
x-goog-api-key: AIzaSyCSz1uxGrHXsEktm7O3_wv-uLGpC9BvXR8
User-Agent: Dalvik/2.1.0 (Linux; U; Android 7.1.1; ONEPLUS A5000 Build/NMF26X)
Host: firebaseinstallations.googleapis.com
Connection: close
Accept-Encoding: gzip, deflate
Content-Length: 129

{"fid":"<REDACTED:generated_somehow_fid>","appId":"1:348804314802:android:440a6773b3620da7","authVersion":"FIS_v2","sdkVersion":"a:16.3.1"}

    Response:

    HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Vary: Origin
Vary: X-Origin
Vary: Referer
Date: Sun, 01 Nov 2020 15:49:23 GMT
Server: ESF
Cache-Control: private
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
Alt-Svc: h3-Q050=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Connection: close
Content-Length: 573

{
  "name": "projects/348804314802/installations/<generated_somehow_fid>",
  "fid": "<REDACTED:generated_somehow_fid>",
  "refreshToken": "<REDACTED:refresh_token>",
  "authToken": {
    "token": "<REDACTED:fi_token>",
    "expiresIn": "604800s"
  }
}

  2. Register the Firebase Installation on FCM to get a token:

    Request:

    POST /c2dm/register3 HTTP/1.1
Authorization: AidLogin <REDACTED:androidid>:<REDACTED:securityToken>
app: com.oceanwing.battery.cam
gcm_ver: 201216023
User-Agent: Android-GCM/1.5 (OnePlus5 NMF26X)
Content-Length: 1061
content-type: application/x-www-form-urlencoded
Host: android.clients.google.com
Connection: close
Accept-Encoding: gzip, deflate

X-subtype=348804314802&sender=348804314802&X-app_ver=741&X-osv=25&X-cliv=fiid-20.2.0&X-gmsv=201216023&X-appid=<REDACTED:generated_somehow_fid>&X-scope=*&X-Goog-Firebase-Installations-Auth=<REDACTED:fi_token>&X-gmp_app_id=1%3A348804314802%3Aandroid%3A440a6773b3620da7&X-Firebase-Client=fire-abt%2F17.1.1+fire-installations%2F16.3.1+fire-android%2F+fire-analytics%2F17.4.2+fire-iid%2F20.2.0+fire-rc%2F17.0.0+fire-fcm%2F20.2.0+fire-cls%2F17.0.0+fire-cls-ndk%2F17.0.0+fire-core%2F19.3.0&X-firebase-app-name-hash=R1dAH9Ui7M-ynoznwBdw01tLxhI&X-Firebase-Client-Log-Type=1&X-app_ver_name=v2.2.2_741&app=com.oceanwing.battery.cam&device=<REDACTED:androidid>&app_ver=741&info=g3EMJXXElLwaQEb1aBJ6XhxiHjPTUxc&gcm_ver=201216023&plat=0&cert=f051262f9f99b638f3c76de349830638555b4a0a&target_ver=28

    Response:

    HTTP/1.1 200 OK
Content-Type: text/plain; charset=UTF-8
Date: Sun, 01 Nov 2020 15:49:24 GMT
Expires: Sun, 01 Nov 2020 15:49:24 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Content-Security-Policy: frame-ancestors 'self'
X-XSS-Protection: 1; mode=block
Server: GSE
Alt-Svc: h3-Q050=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Connection: close
Content-Length: 169

token=<REDACTED:generated_somehow_fid>:<REDACTED:some_other_token>

    The received token is the FCM token that the EufySecurity App now uses to register with its web services to receive the push notifications:

    /v1/apppush/register_push_token

    What I noticed is that the new FCM token is a token that is composed of the Firebase installation ID and a new token. I think that could be the reaseon for filtering some data.

If we could somehow implement this registration "sequence", I think we could get the "full notification" again. :)

In the next few days I have some stressful days at work, so I will not be able to continue working on it. Maybe you can? :)

JanLoebel commented 3 years ago

@bropat I've tried to implement it and was able to get the first request working. But I've a question regarding the second request:

And short question regarding the first request:

The error I get for the second request is: "Error=PHONE_REGISTRATION_ERROR"

bropat commented 3 years ago

@JanLoebel What i found is that before the second request there must be some type of authentication where we got the androidid and securitytoken. i found this repo. Look at this file.

And yes the given "REDACTED:generated_somehow_fid" is the same as in the result of the http-request.

bropat commented 3 years ago

@JanLoebel The authentication part could be done with Google protobuf. Here the checkin.proto.

Here a protobuf typescript library: protobuf-typescript Here a sample: sample

I think we are close to the goal. :)

JanLoebel commented 3 years ago

@bropat So far so good, after some more work I got the registration process working so we get the GCM-Token to register at eufy. But now we have to reimplement the listening to the push messages. As far as I've seen the push-receiver-library is using some more information with private/public keys etc.. so let's see how that is going. So far I've pushed my results to this branch:

https://github.com/JanLoebel/eufy-node-client/tree/feature/push_rework

JanLoebel commented 3 years ago

@bropat I've added the listening process but sadly I could not receive any push messages. Furthermore some connection tries failed, a retry worked than.. any ideas?

JanLoebel commented 3 years ago

@bropat I got it working :) At least a working version with logging of the events and the data is back :) I will have to do some cleanup in the next days :)

bropat commented 3 years ago

@JanLoebel Great Work 💯 :) I am sorry that I could not help you the last 2 days, but I had 2 very exhausting days at work.

bropat commented 3 years ago

Another thing to understand is how to refresh the token with the received "refreshToken" before it expires.

JanLoebel commented 3 years ago

No worries you've found all the needed information, thanks for that! ;)

A) I have restructured and cleared some code. Could you give the branch a test run and see if it also works for you?

B) Next step would be to include a try mechanism for the push-client to connect to the server, sometimes it just gets closed.

C) For the refresh token I would suggest a new ticket, I have a lot of other topics on my agenda so perhaps somebody could work on that.

bropat commented 3 years ago

A) It works after a minor fix #12 :)

B) I added the error argument to the onSocketClose event to understand the cause. The cause could be this.

C) Ok :)

bropat commented 3 years ago

A) It seems that the received push notification must be aknowledged somehow, because if we restart the appl. within the valid push notification TTL, we receive it again, else not.

B) I received this error after some time:

onSocketError:  Error: read ECONNRESET
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:213:20) {
  errno: -104,
  code: 'ECONNRESET',
  syscall: 'read'
}

I'm tracing now with wireshark to better understand. It could be that "mtalk.google.com" closes the connection after some idle time (maybe the tls keep alive is not enough).

Update: Wireshark result after about 25 min.:

image

JanLoebel commented 3 years ago

A) that are the persistendIds, we have to send them in our first request to avoid getting old messages. We would have to save the persistendIds till the ttl expires and send them on reconnect.

B) Could you do the same with the push-reciever? I think that should happen there too, they have implemented a reconnect-mechanism that should handle that.

bropat commented 3 years ago

A) Sounds good :)

B) I'm reading the Chromium source and the microg source. In both i found the hearbeat mechanism. Unfortunately I have to babysit now :P, but I think I will find a solution by the end of the week ;) Otherwise we can use the ECONNRESET event as trigger to reconnect.

bropat commented 3 years ago

B) Done ;) see pull request: #14

bropat commented 3 years ago

A) Done: #15 B) Fixed reconnect

bropat commented 3 years ago

@JanLoebel C) Here how a token refresh works:

POST https://firebaseinstallations.googleapis.com/v1/projects/batterycam-3250a/installations/<fid>/authTokens:generate

With headers:

X-Android-Package: com.oceanwing.battery.cam
X-Android-Cert: F051262F9F99B638F3C76DE349830638555B4A0A
x-goog-api-key: AIzaSyCSz1uxGrHXsEktm7O3_wv-uLGpC9BvXR8
Authorization: FIS_v2 <refreshToken>

JSON Response:

{
    "token": "<newToken>",
    "expiresIn": "604800s"
}

The refreshToken remains valid for future token refreshs :)

Pull request: #16

JanLoebel commented 3 years ago

Awesome :) Have merged your pull requests, but refactored a little bit and added "expiresAt" which is a unix timestamp when the token will expire, so it will be easier to check if the token is expired and renew it before doing other calls. Do you have any information if that means, that after the token expires we have to start all over again: 1) Refresh the token.. 2) Register at the gcm 3) Reconnect with the push client with updated securityToken and androidId?

bropat commented 3 years ago

You're welcome :) At the moment I have no information about this, but I think so.

JanLoebel commented 3 years ago

@bropat I will close this issue, because push is working again :) And better than ever ;) For the refresh-token we can create a new issue if we get some more information there. My next step would be to make the library available via npm and create a new repo with examples/tutorials to provide a better understanding for new users.