Opower endpoint no longer works

[yuna9]

>>> from coned import Meter
>>>
>>> meter = Meter("123456789")
>>> energy_usage_kWh = meter.last_read()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "coned/meter.py", line 39, in last_read
    jsonResponse['error'])
coned.meter.MeterError: ('Error in getting the meter data: %s', {'details': 'HTTP 404 Not Found (43dd6f3a-d2eb-4d6a-acde-1ed6b74021a9)'})

The opower endpoint used in this project no longer works. The correct endpoint to use now is:

https://cned.opower.com/ei/edge/apis/cws-real-time-ami-v1/cws/cned/accounts/<account_id>/meters/<meter_id>/usage

This endpoint only works after opower authenticates via Okta SSO. If they are not, you may get this response:

{"error":{"details":"Customer is not logged in. (ee9d0f9a-e52f-4840-9c35-c975a08ecbb1)"}}

[bvlaicu]

@yuna9 Thanks, yes, I am aware of the change but unfortunately I didn't get a chance to dig into the Okta auth apis.

[jpeeler]

Are these opower APIs documented anywhere?

[bvlaicu]

Unfortunately I could not find any API documentation. My initial implementation was based solely on reverse engineering. That broke when they changed the API and added oauth authentication. I didn't have time to dig into that (yet). Help would be appreciated.

[Martino15]

after a little digging i found this. https://developer.okta.com/docs/reference/api/authn/#request-example-for-primary-authentication-with-public-application it seem coned use them for authentication

[mhoran]

I reached out to ConEd to see if it would be possible to get access to the API. They responded quickly, but the answer was no.

I did try to hit the authenticated API from a private browsing window and it redirected me to the ConEd sign-in page. I'm wondering if that could be scripted, or if cookies could be used to authenticate the API request.

I haven't looked into this much further, but I'd love to be able to use this API. It's unfortunate that ConEd won't provide end users with access, but I think that's because the big companies that provide utility management services make a lot of money from this.

[jpoles1]

Hi all! I was sad to see that this method for fetching the data no longer works, given it was so clean and easy. I was nonetheless determined to get at the data, so I took an alternative approach using Node.js and puppetry, a headless browser automation library by Google. Using my repo you should once again have access to automated data fetches!

You can give it a look here, hopefully it might help some of y'all with your projects! https://github.com/jpoles1/coned-fetch

[bvlaicu]

Thanks, @jpoles1 Using puppetry is a good idea. I can use pyppeteer and implement something similar in python.

[yuna9]

Sorry, I forgot to mention similar work I did back in February. It uses selenium in python. Feel free to check it out. I can contribute something similar to this if you want.

https://github.com/yuna9/coned-rtu

[jpoles1]

@yuna9 you've got a very interesting alternate approach to handling the 2FA problem! I don't 100% understand how it uses the OTP method to generate useable codes though, can you explain a bit how it works?

[yuna9]

@jpoles1 TOTP uses a shared secret and the current time to generate codes. Sites communicate the secret to the user via the QR code that you scan with an authenticator app, etc. If you decode the QR code or you have an authenticator that will show you its stored secret, you can generate valid codes like an authenticator.

[mhoran]

@yuna9 great work, thanks! To run on Debian I had to add --no-sandbox to the webdriver.ChromeOptions() but otherwise I was able to get at my data.

Before I spend a bunch of time on it myself, I'm wondering if anyone has come up with a good way to store and chart the data? I just want to be able to look at my historical usage and figure out where my money is going, so anything is better than what ConEd provides.

[bvlaicu]

I switched the implementation to use pyppeteer, similar to @jpoles1's node.js implementation. Thanks again for the suggestion.

@mhoran I store my data in InfluxDB and use Grafana on top.

[yuna9]

@bvlaicu Seems that folks with TOTP 2FA would be unable to use this. Can you add that method of authentication? You can see an example of this in the code that I linked before.

[bvlaicu]

@yuna9 Done. Check new usage in the updated readme.