EphEmber Integration not working

[nefelodamon]

The problem

Hello,

The EPH Ember integration is no longer working. I am receiving the error "Cannot connect to EphEmber" in the logs.

The connection to the endpoint "eu-https.topband-cloud.com" seems to be OK on my machine and other machines.

Also the API seems to work OK, as I have tested that with POSTMAN:

POST https://eu-https.topband-cloud.com/ember-back/appLogin/login 200 85 ms POST /ember-back/appLogin/login HTTP/1.1 Accept: application/json Content-Type: application/json;charset=utf-8 User-Agent: PostmanRuntime/7.26.10 Postman-Token: cb1bfed9-bf8b-4fc0-8216-d69912078761 Host: eu-https.topband-cloud.com Accept-Encoding: gzip, deflate, br Connection: keep-alive Content-Length: 117 { "password": "EPHPASSWORD", "model": "iPhone 7", "os": "13.1.3", "userName": "email@gmail.com" } HTTP/1.1 200 OK Date: Mon, 08 Feb 2021 14:10:40 GMT Content-Type: application/json;charset=UTF-8 Content-Length: 173 Connection: keep-alive { "data":{ "refresh_token":"refresh_token", "token":"access_token" }, "message":null, "status":0, "timestamp":1612793440983 }

What is version of Home Assistant Core has the issue?

core-2021.2.1

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

EPH Controls

Link to integration documentation on our website

https://www.home-assistant.io/integrations/ephember/

Example YAML snippet

# Put your YAML below this line
climate:
  - platform: ephember
    username: !secret ember_username
    password: !secret ember_password

Anything in the logs that might be useful for us?

# Put your logs below this line
2021-02-08 14:18:37 INFO (SyncWorker_1) [homeassistant.loader] Loaded ephember from homeassistant.components.ephember
2021-02-08 14:18:38 INFO (MainThread) [homeassistant.components.climate] Setting up climate.ephember
2021-02-08 14:18:40 ERROR (SyncWorker_2) [homeassistant.components.ephember.climate] Cannot connect to EphEmber

[probot-home-assistant[bot]]

ephember documentation ephember source _{^{(message by IssueLinks)}}

[probot-home-assistant[bot]]

Hey there @ttroy50, mind taking a look at this issue as its been labeled with an integration (ephember) you are listed as a codeowner for? Thanks! _{^{(message by CodeOwnersMention)}}

[walshtj]

I have a little more information which will hopefully help @ttroy50.

I don't think the issue is with the 'logging in' part. This line seems to executes fine: t = EphEmber(args.email, password, cache_home=args.cache_home)

The problem seems to be the with def get_home(self, gateway_id=None): method. The response text (stepping through with a debugger) is "'WiFi device is not online, please check and try again.'"

All my EPH kit, including the app, are working fine. So my gut instinct tells me they have probably changed something subtle in the /zones/polling endpoint.

Other (non HA) info: On 30th Dec 2020 Ember emailed users explaining that a "major release of EMBER will be available to download on January 4th. It is important that you update the EMBER App to this latest version as the old version of the App will no longer control your heating system in the coming weeks". I think most users, like myself, were expecting some fancy new UI experience. By the 6th I wasn't sure if something was meant to have changed because I didn't notice any difference even though the app was updated. I asked the EPH people who told me that ... "The update is to facilitate the change of all customers to a new server – simple bug fixes – nothing major will change."

I later learned that they were migrating users to the new server in batches rather than all at once. I was seemingly moved to the new server on the 29th January. (I was getting 'socket errors' in the native app mid January and these problems seemed to go away after my migration to the new server.)

I can't be 100% sure of the exact date but I think sometime last week (say between 1st to 4th Feb) is when ember integration in HA stopped working. What I mean is that I'm pretty sure that everything was still working in HA after EPH said they migrated me to their new server. It would have made more sense if it broke the moment I was moved to the new server - so I'm a bit confused.

Hope that helps. Happy to answer any more questions if needed as I really loved the integration.

[ttroy50]

Thanks for the detailed info @walshtj

I'd see the warning about the app update but haven't had a chance to look at the new API yet. I'll try prioritise it over the weekend and see what I can get done.

[walshtj]

You are very welcome @ttroy50. I forgot to add that the response code for the /zones/polling code is OK (200).

As you said all those years ago "..this API may be subject to change. Use of this API is at your own risk." so I wasn't really expecting a reply - certainly not so quickly! I really enjoyed your blog post about the whole reverse engineering task from 2017 - very educational.

thanks again and if I can help in any way please shout.

[nefelodamon]

@ttroy50 and @walshtj , thanks for looking into that! If I can do anything to help please let me know!

[ttroy50]

I've run some tests against the API from my library it is working fine for me. I've run it checking the home every 30 seconds and it always gets a response.

I also captured some traffic from the app and the API seems to be almost the exact same to me. There are a few extra fields in the login request but they seem to be optional. However, It may be that I haven't been moved to the new server.

@walshtj or @nefelodamon would you be able to capture some traffic from your app and send me the examples of the requests and responses with your personal information removed?

[walshtj]

@ttroy50 I'm proxying my wifi connection on Android phone to Charles proxy on my mac and I am seeing the app talk to the https://eu-https.topband-cloud.com but all the calls are showing up as 'unknown'. I've installed the charles proxy cert on my phone for "vpn and apps" and "wifi" because I wasn't sure which.

I don't think that has worked though because I seem to get logged out of the Ember app and can't log back in with the proxy in place. The (I assume) API call to login fails. In Charles I see a failure field with "SSL handshake with client failed: An unknown issue occurred processing the certificate (certificate_unknown)" and Notes field with "You may need to configure your browser or application to trust the Charles Root Certificate. See SSL Proxying in the Help menu."
Can you help me get the cert working on my phone pls?

EDIT: I also included eu-https.topband-cloud.com in the SSL proxying settings in Charles. EDIT2: I'm added a random url (reddit.com) to Charles also and I can see its calls and data. So I guess its the actual Ember app which isn't playing ball in this case? I saw someone on stackoverflow mentioning adding something to the manifest - but thats not possible without the source (or is it?)

[ttroy50]

thanks for trying. Unfortunately android changed their security policy in a recent version so you can't do this kind of man in the middle proxying unless you control the app. See the android section of https://www.charlesproxy.com/documentation/using-charles/ssl-certificates/

When capturing traffic now, I have to use the iPhone version of the app.

[walshtj]

@ttroy50 yeah I just read that... Android Nougat changed things...so I guess you originally did your investigation before this. Let me see if I can get an iOS device

[walshtj]

@ttroy50 I found an old android phone running lollipop (ie pre-noughat) and its also didn't work. Odd.

Anyway I got it working on an iPad and I can see the traffic. I see calls to login, reportToken, selectUser, list, details, weathInfo. Which would you like to see? (and which parts - overview, contents, summary...)?

btw I did notice one thing... in your original tutorial your traffic was going to https://ember.ephcontrols.com while mine is going to https://eu-https.topband-cloud.com. When you were conducting successful tests earlier today were you still pointing to the first url?

The ember guys did say "......we are only moving customers to the new server in lots of 100 so it will take until March before all are over,...."

[ttroy50]

The library is currently using the topband-cloud address. https://github.com/ttroy50/pyephember/blob/master/API.md specifies the current API which has been in place since late 2019

If you could compare your responses to the linked API doc and see if it looks similar, then we can start to see what has changed.

If the API is updated, I might need someone to document the new API or give me access to a temporary (non-admin) account so I can get the network traces myself.

[walshtj]

/login. ( think this also works on HA still) POST {"password":"xxxxxx","model":"iPhone","os":"14.3","type":2,"appVersion":"2.0.4","userName":"xxxxxxx"}

RESPONSE

{
    "data": {
        "refresh_token": "long_refresh_token",
        "token": "long_token"
    },
    "message": null,
    "status": 0,
    "timestamp": 1613237316835
}

Actually before I start pasting in piles of json.... a few thoughts: I noticed that the sequence of calls when I login to the app(see attached image) is login, reportToken, selectUser, list (x2), details......etc. I think some of those are new endpoints that are not referenced in your list of endpoints. While your example.py code does the/login first, and then/zonePolling (in the getHome() method). Its the 2nd call where we get the "WiFi device is not online, please check and try again." message.

Anyway earlier today, since there is no chat in github, I sent you a linkedin request and I offered to add you to my EPH ember. As much as I enjoy learning about this stuff the reality is that its still all new to me. By adding you (@ttroy50) then you have a system which works (yours) and a system which doesn't (mine) and you'll probably be able to spot the problem quicker. If this works for you can you accept my linkedin request and we can swap emails.

[ttroy50]

I got some testing done over the weekend with the new API (thanks @walshtj ) and it looks like they have split their new API into 2 parts.

1. A JSON / HTTPs API for authentication and returning some information about the home / zones. Some of the JSON naming isn't as clear as the old API but it's understandable enough to read the basic zone information, set times, current temp, target temp.
2. When changing data (e.g. setting boost), I don't see anything over the HTTP API. Doing a packet capture I can see that the app is also sending encrypted TLS data to a server like eu-mqtt.topband-cloud.com on port 18883. Which seems to me to imply that they have split some of their control data out to an MQTT interface.

To proceed with keeping the same functionality we currently have I would need to capture and decrypt the MQTT traffic to see the data they are sending there. I'm not very familiar with MQTT so will need to do some investigation on how to do that.

[nefelodamon]

I was able to capture the traffic, send to eu-mqtt.topband-cloud.com, from the app installed in an Android simulator. Unfortunately I was not able to decrypt it. Trying some MIM apps with VPN doesnt seem to work either.

Will keep looking for a way to decrypt the data.

[nefelodamon]

I have spend several hours looking at that today.

The app will not work with any man in the middle proxy apps, as this method requires a user certificate. To make the app able to use user certificates, it needs to be repackaged with a process similar to this one: https://hurricanelabs.com/blog/modifying-android-apps-to-allow-tls-intercept-with-user-cas/

Unfortunately, the developers at EPH used the Jiagu packer (https://jiagu.360.cn/#/global/index) making the re-package/re-signing process very hard.

Simply trying to re-sign the app breaks it and gives these errors when installed in a device:

02-16 17:17:06.457 31055 31055 E AndroidRuntime: FATAL EXCEPTION: main 02-16 17:17:06.457 31055 31055 E AndroidRuntime: Process: com.ephcontrols.ember, PID: 31055 02-16 17:17:06.457 31055 31055 E AndroidRuntime: java.lang.UnsatisfiedLinkError: JNI_ERR returned from JNI_OnLoad in "/data/data/com.ephcontrols.ember/.jiagu/libjiagu_64.so"

At this point, unless someone is very good with Android development, the decrypting of the data looks very improbable.

[walshtj]

Fair play everyone for all the effort. I, and I'm sure lots of other people, really appreciate your efforts.

I can't think of a reason why EPH decided to change so much. Perhaps there were reasonable security concerns, perhaps there are upcoming products which plan to compete against Nest and Hive... I've no idea.

However I really liked being able to add my Ember central heating to Home Assistant as I just don't like their App. It's very limited - for example to 3 programs a day. With HA I can just boost a single zone until it reaches my desired temperature ..it was so cool (pardon the pun). But I also understand them trying to keep it simple for general use.

When I was getting my boiler replaced I was asking my installer about Nest and Hive.. but the reality was that Ember are a 'central heating' company first and foremost...and boiler installers just understand EPH systems. They know the products well and trust and use them.

Nest is just another a 'Google acquisition" which they overpaid for but could shut down tomorrow. If I was a boiler installer I'd stick with the dedicated engineering company rather than someone testing the waters like Google.

However Ember make (lets be honest) pretty basic controllers. The gateway thing is a bit of a hardware hack to get the main controller 'on the internet'. They are a central heating company who are respected by boiler engineers for their complete package of controllers, motorised values, plumbing bits-n-pieces but tech is not their forte. By contrast Nest/Hive sell a little computer.... they don't also make the 'wet' components...they are blow-ins without the underlying domain expertise.

This was all a VERY VERY long winded way of me to suggest the following..... Instead of trying to hack their system - I could just reach out to Ember and say 'hey guys... this is what we're doing...can we work together and just allow this since IoT stuff is the future'.

It's surely in their interest to essentially let other people augment their tech? The people I've emailed on EPH support seem very reasonable and are based in Cork I believe. I think an open dialogue might be the most potentially fruitful here?

Thoughts and comments welcome.

[ttroy50]

I will try to contact them but I wouldn't hold out much hope. From what I can gather they outsource the gateway / API and internet controls to a Chinese company and I think getting a spec / support for it may be hard. But then again maybe they will be helpful and it will make things easier.

@nefelodamon I think the packet capture may be easier on iOS but I haven't had a chance to test again this week. From what I can gather it will involve

• installing a root cert on the phone
• modifying the dns entry on the router to point to my "intercept device"
• Hosting a TCP (or MQTT) MITM proxy to decrypt the TLS, dump it and then forward the packets to the "real server"

The first 2 steps are easy enough with my setup. It's finding what I want to use for the last one that I'm not sure of. If you have any suggestions then let me know.

[nefelodamon]

@ttroy50, I am afraid I am not much much of an iOS guy but if you have a rooted phone perhaps is possible? I just hope they don't do any certificate pining. I think they do in the android app as fiddler decrytps the data ok with a custom cert for http traffic.

For a proxy app perhaps the mitmproxy will work? I believe it can capture tcp traffic.

On another note, I was playing a bit with some android emulators today. Not much progress there but will keep trying.

Regarding contacting the company, I suppose we have nothing to lose. I guess if we ask nicely they may help :-)

Small update: looks that burp proxy is a good candidate? Maybe these 2 articles can help? https://medium.com/cybersecurityservices/getting-started-with-traffic-analysis-of-ios-applications-part-1-application-traffic-and-burp-84313e1334ff

And

https://portswigger.net/burp/documentation/desktop/tools/proxy/options/invisible

[github-actions[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.