Invesitgate issues with API calls failing

[magico13]

Noticed in https://github.com/magico13/ha-emporia-vue/issues/27 and https://github.com/jertel/vuegraf/discussions/14.

Feels like a change in the API meaning I'll have some work to do to sniff out the changes and update. Hopefully just a simple fix and not a totally new auth scheme or something that would prevent our access. The app updated recently and is still working so presumably that update included the changes to whatever is different with the API.

[aviadoffer]

Same issue for me just started last night :

requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://api.emporiaenergy.com/usage/d8802Z&end=2021-02-09T17:55:39.288802Z&scale=1m&unit=WATTS&customerGid=....

And thank for working on it!

[drewstreib]

The payload of the 403 response for several endpoints is now: "Missing Authentication Token", but the "/customers" endpoint is still working with the same auth header (for me).

Note that for a period this morning, I was getting 400 responses to at least one endpoint, although that has changed to 403 now (so things were changing). For a period the response was 400 and:

"message": "Unhandled path '/USAGE/DEVICES' and method 'GET' for input '{resource=/usage/devices, path=/usage/devices, httpMethod=GET, headers=...

[magico13]

So the bad news is that the API definitely had some changes but the good news is that it looks like they might be prepping for a public API. All the calls moved to "/AppAPI" and some of the units have changed. I haven't had time to actually go make any updates yet, just watched some of the traffic.

[aviadoffer]

Thanks for looking into it!

[drewstreib]

I'd be happy to probe around a bit but don't have a sniffer setup. I have implemented the API in nodered through and am happy to poke around and report results. Did the auth header change? I'm still getting "Missing Authentication Token" when using "authtoken:" with the cognito idtoken for the call.

(Also if anyone has any interest in the nodered flow using function-npm nodes for the cognito part, let me know.)

[magico13]

It looks the same to me but I haven't thrown it into postman to check out yet. Definitely still using authtoken and the token itself looks like the typical AWS ones I've seen. I've got another few hours of work today and then I'll dig into it more after work, should hopefully not take too long to get this updated and I'll try to document the updated API as I go.

[drewstreib]

The token error could just be a red herring for changed api endpoints too. Cloudfront isn't always the greatest when it comes to accuracy of error messages.

[magico13]

Here's a quick sample of updated calls if you're interested in poking around with them:

GET /remoteconfig?appVersion=2.4.35.2435 (Not sure if this is going to be used to lock the API down if you're not the app)

GET /customers/thermostats?customerGid=0000 (Related to the new thermostat integration, haven't messed with it yet but I do have an Ecobee so I can look into it eventually)

GET /Notifications/GetAll?customerGid=0000

GET /AppAPI?apiMethod=getDevicesUsage&deviceGids=%5B7466%2C+16072%5D&instant=2021-02-09T18%3A42%3A53.598068Z&scale=1S&energyUnit=KilowattHours

GET /AppAPI?apiMethod=getChartUsage&deviceGid=7466&channel=1%2C2%2C3&start=2021-02-09T17%3A42%3A51.000Z&end=2021-02-09T18%3A42%3A48.000Z&scale=1S&energyUnit=KilowattHours

[drewstreib]

Tested /Notifications/GetAll and /AppAPI?apiMethod=getDevicesUsage and they do appear to work for me, although I had to now grab my deviceID and pass that in rather than customerGid. Of course the actual return values/scales are completely different now and require some new easy math. The api mappings aren't quite 1:1.

Edit: Not sure how to programmatically get deviceGids. I grabbed mine because they appeared in the /Notifications/GetAll endpoint, although that probably isn't proper way to get the list.

Edit2: Also not sure the valid scales. I'm using "1S" per your example, but "60S" and "1M" return things like "scaleName '1M' is not a valid scale; defined scales are [Lcom.emporiaenergy.common.pojo.Scale;@19ac267e" on the getDevicesUsage endpoint.

[drewstreib]

I've been polling every minute for ~3 hours now and using getDevicesUsage to try to recreate my prior use of /usage/devices. With some multiplication to match the scale, the values seem sane. Using "instant" and the current timestamp, these values do seem to update at least once per minute. I'd rather know that I'm getting 1 minute averages rather than point in time values, but the differences are minor for me.

[magico13]

Alright I just upload 0.12.1 that should have the new methods implemented. I've removed the old methods as well so it's definitely a breaking change code-wise. Let me know if there are any unexpected issues with it. I'm gonna try to get the Home Assistant integration updated really quick as well.

[aviadoffer]

Sweet, I will try now. Thanks man!

[aviadoffer]

so what happened to WATTS ? How do I do vue.get_recent_usage(scale="1S",unit="WATTS") ?

[drewstreib]

You grab KilowattHours, and if you're getting "1S" unit, multiply by (60 60 1000), which is (secs mins kilo), to get point in time watts. If you get "1MIN" unit, multiply by (60 * 1000), etc. '

Edit: Don't get me started on the weird choice to throw out a float and then make the unit have a built in prefix (rather than just reporting "WattHours"), but meh, just chasing whatever they've implemented.

[magico13]

What @drewstreib said, it's all Kilowatt-Hours now so to get a rate you have to scale it manually. AmpHours is also an option now though the app still appears to give me the wrong value for that since my furnace (which doesn't have its own sensors attached) makes my house-wide current almost double what it should be.

[aviadoffer]

Thanks, I get an error

channel_usage_list = vue.get_devices_usage(deviceGids, None, scale=pyemvue.enums.Scale.DAY.value, unit=pyemvue.enums.Unit.KWH.value)

Traceback (most recent call last): File "", line 1, in File "c:\Python38-64\lib\site-packages\pyemvue-0.12.1-py3.8.egg\pyemvue\pyemvue.py", line 88, in get_devices_usage File "c:\Python38-64\lib\site-packages\pyemvue-0.12.1-py3.8.egg\pyemvue\device.py", line 93, in from_json_dictionary TypeError: argument of type 'NoneType' is not iterable

[magico13]

Could be an issue like #8? I would presume the json dictionary is empty/null for that channel. I can make a quick fix for it probably to just null check that.

[aviadoffer]

Yes, I have 2 devices in your list, the second one is empty. I actually only have one device in reality.

[magico13]

Just uploaded 0.12.2, let me know if that fixes it.

[aviadoffer]

OK, but notice this weird before I updated to 0.12.2

devices = vue.get_devices() deviceGids = [] for device in devices: ... deviceGids.append(device.device_gid) ... deviceGids [14388, 14388]

[magico13]

I saw something similar in my testing and it seemed to just ignore the duplicate when I passed it so I didn't think too much of it. When updating the Home Assistant integration I modified the append to check if the id was already in the list. The get_devices call might need refactored a bit.

[aviadoffer]

Cool. seems to be working fine now. I will do some more testing. You are the man thanks!

[cooldil]

The getChartUsage API call doesn't seem to work for the individual circuits (channels). It pulls the usage over time correctly for the Vue device itself (main CTs) but it returns an error 400 when the individual channel information is attempted to be pulled. Seems like the channel=xxx parameter doesn't like any value other than "1,2,3"....

400 Client Error: Bad Request for url: https://api.emporiaenergy.com/AppAPI?apiMethod=getChartUsage&deviceGid=&channel=1&start=2021-02-10T07:15:13+00:00Z&end=2021-02-10T07:21:13Z&scale=1S&energyUnit=KilowattHours

[magico13]

Did you manually pull the device gid value out from that message or is that exactly how it was? Might be a bug with passing the channel object in and it not having the device gid set on it for some reason. You could new up a VueDeviceChannel with the device gid and channel set and pass that in rather than passing one from get_devices and see if that works. Or I might have a bug with pulling that data out.

I'll try to test that with other channels in a bit. The HA integration only uses the getDevicesUsage call so I haven't done too much with the getChartUsage call past some basic tests.

[DarwinData]

Thanks to everyone working on this. I currently use the Vue HA integration, which of course is not working. Any eta on getting that fixed? No pressure at all as this data is 'nice' to have and not a 'must' have for me. cheers and thanks again

[magico13]

@DarwinData I did push out an update for that about 12 hours ago (v0.3.0) though there are some people seeing issues with it still. https://github.com/magico13/ha-emporia-vue/issues/28

[cooldil]

Bummer... I had masked the devicegid parameter value with brackets and they got removed... Short answer, there IS a valid value being passed for devicegid....

I did try to cheat :-) and took a look at your HA code last night to see if you were pulling the individual circuit data and saw that you were not before posting about the "bug" here.....

[DarwinData]

magico13, just want to let you know that I updated to your v0.3.0 and it is working! fyi, I am using the 1st gen with 8port expansioin and no other smart plugs. thanks!

[magico13]

@cooldil if you run python -m pyemvue keys.json with a proper keys.json file does that error out? My guess is it will but there's specifically a call for each channel to getChartUsage after the "Total usage for yesterday in kwh:" message that on my machine is working fine and gives the same results as what I see in the app.

Actually I think the issue with that call is that the date has the timezone data in it (+00:00Z) which the API doesn't like. Pull the timezone data out with time = time.replace(tzinfo=None) and it might work. I probably should be doing that in the code automatically and automatically swap over to UTC if a local time is passed in.

[aviadoffer]

Confirmed with support that they now assume voltage is 120v therefore the amp readings are inaccurate. They are working on fixing it as they actually do measure voltage but they don't store it yet.

[cooldil]

magico13, you nailed it. My code was changing the start time AFTER the initial call and it was adding the TZ information. I did figure it out eventually... :-) Came here to to say "my bad..." only to find out that you had come to the same conclusion....

[drewstreib]

How long before they realize that they send duplicate "value" and "Value" entries in the same object and remove the offending one? (Although with their capitalization inconsistency, there's no telling which one fits the intended scheme).

[magico13]

Yeah I saw that and arbitrarily took one of them. Might update to try to grab the other if the one is missing because they might drop one of them. That whole object felt weird to me, like in the Timestamp they have a nanosecond property that is always zero. Someone serialized an object without really looking at the result I think.

[magico13]

Cleaning up old issues, this definitely seems outdated now. I think the API has changed even further from this particular change.