search

tronikos / opower

A Python library for getting historical and forecasted usage/cost from utilities that use opower.com such as PG&E
Apache License 2.0
66 stars 59 forks source link

City of Austin Utilities fetching from "customers" endpoint fails with 403 (Forbidden) #73

Closed philipflesher closed 8 months ago

philipflesher commented 8 months ago

Running latest in main branch in a dev container, using my own credentials. I believe prior calls are working (e.g., call to the "connectedaccounts" endpoint seems to succeed).

Happy to run again and try various things to help run this down!

From debugger, full stack:


  File "/workspaces/opower/src/opower/opower.py", line 309, in _async_get_customers
    async with self.session.get(
  File "/workspaces/opower/src/opower/opower.py", line 229, in async_get_forecast
    for customer in await self._async_get_customers():
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/opower/src/demo.py", line 89, in _main
    for forecast in await opower.async_get_forecast():
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/opower/src/demo.py", line 170, in <module>
    asyncio.run(_main())
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://dss-coa.opower.com/webcenter/edge/apis/multi-account-v1/cws/coa/customers?offset=0&batchSize=100&addressFilter=')```
max2697 commented 8 months ago

Sorry for delay. Could you also describe your account please? Do you have electricity only or water/trash/gas too? Are you log in from https://coautilities.com/wps/wcm/connect/occ/coa/home page and then it redirect you to https://dss-coa.opower.com/dss/landing page?

philipflesher commented 8 months ago

@max2697 In looking at the structure, I think I may be realizing an edge case in my accounts that is probably just not handled yet in code. I have an account with no premises associated with it, due to the premises being transferred to a tenant's account. The account is still active, though.

"No premises found for this account."

RE: the redirect, yes, it does redirect to that landing page at that domain.

If you have a suggestion on what I should attempt to modify in my own branch, I could try to make the change and see if it works for me. Let me know.

wcorrigan commented 8 months ago

I set up my integration per the instructions and onboarding went smoothly, but I get no data to the Energy Dashboard in Lovelace. Just a notice there that it takes a couple of hours for data to show up. I've had this set up for a month now. Please advise on how I might find useful data to troubleshoot this issue.

tronikos commented 8 months ago

You need to add the statistics to the energy dashboard and not any sensors the integration might have created.

wcorrigan commented 8 months ago

I followed the instructions here: https://www.home-assistant.io/integrations/opower#energy My configuration looks like the example provided. Given that, I don't understand how your suggestion applies to my issue. When I go to: Developer Tools > Statistics and search for “opower”, the 2 statticial ids configured show up with no issues. when I click on the adjust button at the right, then click on outliers in the modal that pops up, it reports data going back 5 years. The logs are not reporting anything to me. Please advise.

tronikos commented 8 months ago

In the energy dashboard if you select "this year" from the top right corner do you see data per month? Some utilities only report data every month. And all report data with at least one day delay.

wcorrigan commented 8 months ago

Ah there you go! Yes it must be only reporting by month. I see January and February. I will review the instructions on usage of the Energy usage dashboard to see if there was something I missed or if some clarification is needed. Thank you very much.

max2697 commented 8 months ago

@philipflesher Okay, let's check a few details. When you run the demo, add the -v flag for logging requests and change the constant DEBUG_LOG_RESPONSE (https://github.com/tronikos/opower/blob/main/src/opower/opower.py#L20) to True. Check the first request and its response: https://dss-coa.opower.com/webcenter/edge/apis/dss-invite-v1/cws/v1/utilities/connectedaccounts?pageOffset=0&pageLimit=100 My script expects to get at least one account in the accounts list.

By the way, if you have no premises, what data do you want to get from Opower? Can you still see your electricity data on the Opower website?

UPD: You can also open the https://dss-coa.opower.com/dss/overview page in browser and check requests to the server. It should be a https://dss-coa.opower.com/webcenter/edge/apis/multi-account-v1/cws/coa/customers request presented with the current account ID in response.

philipflesher commented 8 months ago

Hi @max2697 -- to clarify, I do have a premises in one of my two accounts. The other account I do not have any premises under. And by "account," I mean the customer account data structure within my login/user account:

I ran the demo again and got the following output:

DEBUG:/home/vscode/.local/lib/python3.12/site-packages/opower/opower.py:Fetching: https://dss-coa.opower.com/webcenter/edge/apis/dss-invite-v1/cws/v1/utilities/connectedaccounts?pageOffset=0&pageLimit=100
DEBUG:/home/vscode/.local/lib/python3.12/site-packages/opower/opower.py:Fetching: https://dss-coa.opower.com/webcenter/edge/apis/multi-account-v1/cws/coa/customers?offset=0&batchSize=100&addressFilter=
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "/home/vscode/.vscode-server/extensions/ms-python.debugpy-2024.0.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module>
    cli.main()
  File "/home/vscode/.vscode-server/extensions/ms-python.debugpy-2024.0.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
    run()
  File "/home/vscode/.vscode-server/extensions/ms-python.debugpy-2024.0.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 317, in run_module
    run_module_as_main(options.target, alter_argv=True)
  File "/home/vscode/.vscode-server/extensions/ms-python.debugpy-2024.0.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 238, in _run_module_as_main
    return _run_code(code, main_globals, None,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.vscode-server/extensions/ms-python.debugpy-2024.0.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "/home/vscode/.local/lib/python3.12/site-packages/demo.py", line 170, in <module>
    asyncio.run(_main())
  File "/usr/local/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 685, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.12/site-packages/demo.py", line 89, in _main
    for forecast in await opower.async_get_forecast():
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.12/site-packages/opower/opower.py", line 229, in async_get_forecast
    for customer in await self._async_get_customers():
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.12/site-packages/opower/opower.py", line 309, in _async_get_customers
    async with self.session.get(
  File "/home/vscode/.local/lib/python3.12/site-packages/aiohttp/client.py", line 1194, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.12/site-packages/aiohttp/client.py", line 693, in _request
    resp.raise_for_status()
  File "/home/vscode/.local/lib/python3.12/site-packages/aiohttp/client_reqrep.py", line 1060, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://dss-coa.opower.com/webcenter/edge/apis/multi-account-v1/cws/coa/customers?offset=0&batchSize=100&addressFilter=')

Browsing and looking at network requests, there is no request to the customers endpoint itself, but there is this, which is made every time the drop-down selector is changed between my two account IDs:

https://dss-coa.opower.com/webcenter/edge/apis/multi-account-v1/cws/coa/customers/current

And what's interesting is that when switching to the "empty" account, I see an error come back for that request, at least the second time I select it in the drop-down (the first, the request seems to succeed with no error, when the page first loads and the "empty" account is default selected):

{
  "error": {
    "details": "Request forbidden without list of customer UUIDs",
    "serviceErrorCode": "EMPTY_AUTHORIZED_CUSTOMERS_LIST",
    "correlationId": "41886d1e-63b3-4b9d-8a7e-35c3d1826b87"
  }
}
max2697 commented 8 months ago

@philipflesher Could you please change the constant DEBUG_LOG_RESPONSE (https://github.com/tronikos/opower/blob/main/src/opower/opower.py#L20) to True and show (with masking private info) response for the first request? It should be something like:

"accounts": [
    {
      "accountId": "***",
      "cisDivision": null,
      "nickName": "",
      "multiplePremiseExist": false,
      "premises": [
        {
          "country": "USA",
          "addressLine1": "***",
          "addressLine2": "",
          "addressLine3": "",
          "addressLine4": "",
          "houseType": "  ",
          "number1": "",
          "number2": "",
          "inCityLimit": ***,
          "city": "AUSTIN",
          "geographic": "",
          "county": "",
          "state": "TX",
          "postal": "***",
          "premiseId": "***"
        }
      ],
      "invitedUsers": [],
      "guestUsers": []
    }
  ],
  "totalRecords": 1
}
philipflesher commented 8 months ago

I did change that constant. I got no output like you're showing. I added -v also... maybe I'm overlooking something.

philipflesher commented 8 months ago

Yeah it's just the output I already pasted in. The two DEBUG lines I see are the "Fetching" lines, as shown above.

max2697 commented 8 months ago

That's weird... Could you add break point at line https://github.com/tronikos/opower/blob/main/src/opower/opower.py#L337 and check content of result variable? It should show user accounts...

philipflesher commented 8 months ago

@max2697 here you go. No clue why this isn't being written out via the logger... it looks nearly the same as the "Fetching" line above...

{
  "accounts": [
    {
      "accountId": "(removed)",
      "cisDivision": null,
      "nickName": "",
      "multiplePremiseExist": false,
      "premises": [],
      "invitedUsers": [],
      "guestUsers": []
    },
    {
      "accountId": "(removed)",
      "cisDivision": null,
      "nickName": "",
      "multiplePremiseExist": false,
      "premises": [
        {
          "country": "USA",
          "addressLine1": "(removed)",
          "addressLine2": "",
          "addressLine3": "",
          "addressLine4": "",
          "houseType": "  ",
          "number1": "",
          "number2": "",
          "inCityLimit": false,
          "city": "(removed)",
          "geographic": "",
          "county": "",
          "state": "(removed)",
          "postal": "(removed)",
          "premiseId": "(removed)"
        }
      ],
      "invitedUsers": [],
      "guestUsers": []
    }
  ],
  "totalRecords": 2
}
max2697 commented 8 months ago

Created draft PR: https://github.com/tronikos/opower/pull/75

@philipflesher Could you please checkout my branch https://github.com/max2697/opower/tree/dss-multi-accounts-fix and run demo again? I hope the issue will be resolved. Please note that I enabled debugging, so it should show you all responses from the server too.

philipflesher commented 8 months ago

Appears to work at that branch.

Is the library not designed to work with multiple accounts, I suppose? As written, if a user did have multiple accounts, each with a premises, I guess there would be no way to use anything other than the first?

max2697 commented 8 months ago

Yeah, it's a small WA to handle cases like yours. The DSS website structure is more complicated than regular opower: it has connected accounts, customer accounts on each, and utility accounts for them. It's possible to support cases of multiple accounts, but I prefer not to refactor the main part without a good reason, like a user with multiple accounts, customers, and utilities. Let's see if anybody from Austin has a case like this, and then I could do such refactoring.

tronikos commented 8 months ago

Note the non DSS implementation (currently only the City of Austin Utilities is using DSS) supports multiple accounts at least at the opower side.

philipflesher commented 8 months ago

@tronikos I think it might be a different kind of "account" than what you're talking about. There's a login, and that login can have user accounts, and each of those user accounts has customers (??), each of which can have multiple accounts (??). Those last two I'm assuming from browsing the code quickly, you might know better than I do. :)

max2697 commented 8 months ago

@tronikos Yep, you are right. But DSS is adding a new account on top of it: DSS account -> customer -> utility account. It's possible to update the main Opower code to support multiple DSS accounts for one user (update async_get_accounts function). However, I want to avoid modifying shared code in order to not possibly break all other providers to support a hypothetical case for DSS accounts. I suggest waiting until someone encounters such a case, and then we could test it on real data.

@philipflesher So basically, it's DSS accounts and utility accounts. This should also be renamed in the code for possible future refactoring.

tronikos commented 8 months ago

Fixed with #75 @max2697 can you please bump the version on HA to v0.4.2?

max2697 commented 8 months ago

Sure, PR: https://github.com/home-assistant/core/pull/114608

nnayda commented 7 months ago

I had this HA integration working before but am now receiving a slightly different error than OP. When running the demo I get:

Traceback (most recent call last):
  File "/Users/ntnayda/local/GitHub/opower/src/opower/opower.py", line 193, in async_login
    self.access_token = await self.utility.async_login(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ntnayda/local/GitHub/opower/src/opower/utilities/coautilities.py", line 136, in async_login
    async with session.post(
  File "/Users/ntnayda/opt/anaconda3/envs/opower/lib/python3.11/site-packages/aiohttp/client.py", line 1197, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "/Users/ntnayda/opt/anaconda3/envs/opower/lib/python3.11/site-packages/aiohttp/client.py", line 696, in _request
    resp.raise_for_status()
  File "/Users/ntnayda/opt/anaconda3/envs/opower/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1070, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 500, message='Internal Server Error', url=URL('https://dss-coa.opower.com/webcenter/edge/apis/identity-management-v1/cws/v1/auth/coa/saml/ott/confirm')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ntnayda/local/GitHub/opower/src/demo.py", line 172, in <module>
    asyncio.run(_main())
  File "/Users/ntnayda/opt/anaconda3/envs/opower/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/Users/ntnayda/opt/anaconda3/envs/opower/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ntnayda/opt/anaconda3/envs/opower/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/ntnayda/local/GitHub/opower/src/demo.py", line 88, in _main
    await opower.async_login()
  File "/Users/ntnayda/local/GitHub/opower/src/opower/opower.py", line 201, in async_login
    raise CannotConnect(err)
opower.exceptions.CannotConnect: 500, message='Internal Server Error', url=URL('https://dss-coa.opower.com/webcenter/edge/apis/identity-management-v1/cws/v1/auth/coa/saml/ott/confirm')
grutamu commented 4 months ago

Hi there, I think i am running into this same issue. is there some one available to help me? running on the latest version of opower pulled down from github. I am also receiving this error on the latest HA build. 

opower-main % python -m opower --utility coautilities --username *** --password *** -v
DEBUG:/Users/user/Downloads/opower-main/.venv/lib/python3.12/site-packages/opower/opower.py:Fetching: https://dss-coa.opower.com/webcenter/edge/apis/dss-invite-v1/cws/v1/utilities/connectedaccounts?pageOffset=0&pageLimit=100
DEBUG:/Users/user/Downloads/opower-main/.venv/lib/python3.12/site-packages/opower/opower.py:Fetching: https://dss-coa.opower.com/webcenter/edge/apis/multi-account-v1/cws/coa/customers?offset=0&batchSize=100&addressFilter=
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/user/Downloads/opower-main/.venv/lib/python3.12/site-packages/opower/__main__.py", line 214, in <module>
    asyncio.run(_main())
  File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/user/Downloads/opower-main/.venv/lib/python3.12/site-packages/opower/__main__.py", line 97, in _main
    for forecast in await opower.async_get_forecast():
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/Downloads/opower-main/.venv/lib/python3.12/site-packages/opower/opower.py", line 249, in async_get_forecast
    for customer in await self._async_get_customers():
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/Downloads/opower-main/.venv/lib/python3.12/site-packages/opower/opower.py", line 343, in _async_get_customers
    async with self.session.get(
  File "/Users/user/Downloads/opower-main/.venv/lib/python3.12/site-packages/aiohttp/client.py", line 1197, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "/Users/user/Downloads/opower-main/.venv/lib/python3.12/site-packages/aiohttp/client.py", line 696, in _request
    resp.raise_for_status()
  File "/Users/user/Downloads/opower-main/.venv/lib/python3.12/site-packages/aiohttp/client_reqrep.py", line 1070, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://dss-coa.opower.com/webcenter/edge/apis/multi-account-v1/cws/coa/customers?offset=0&batchSize=100&addressFilter=')
max2697 commented 3 months ago

Hi, is it still failing? On my end, it's working fine. If it is still an issue, could you please open a new one, since this one has already been closed?