search

tronikos / opower

A Python library for getting historical and forecasted usage/cost from utilities that use opower.com such as PG&E
Apache License 2.0
65 stars 59 forks source link

Requests to ComEd fail with too large of a date range #32

Open bjackson opened 1 year ago

bjackson commented 1 year ago

In reference to home-assistant/core#97814.

The HA integration requests day aggregate information from ComEd from 2022-06-13T00:00:00-05:00 to 2023-06-11T00:00:00-05:00. This fails with the following response:

{
    "error": {
        "httpStatus": 500,
        "serviceErrorCode": "UPSTREAM_ERROR",
        "details": "Could not get rated costs and usages for utility account UUID: XXX"
    }
}

Here's what I ran in the terminal:

➜ python src/demo.py --utility comed --username "brett@brettjackson.org" --password 'XXX' --start_date 2022-06-13T00:00:00-05:00 --end_date 2023-06-11T00:00:00-05:00

Current bill forecast: Forecast(account=Account(customer=Customer(uuid='9e967f23-9e95-11eb-bf8b-0200170058ac'), uuid='a3ba62b5-9e95-11eb-bf8b-0200170058ac', utility_account_id='XXX', meter_type=<MeterType.ELEC: 'ELEC'>, read_resolution=None), start_date=datetime.date(2023, 8, 1), end_date=datetime.date(2023, 8, 30), current_date=datetime.date(2023, 8, 10), unit_of_measure=<UnitOfMeasure.KWH: 'KWH'>, usage_to_date=309.0, cost_to_date=0.0, forecasted_usage=1337.0, forecasted_cost=0.0, typical_usage=1099.0, typical_cost=0.0)

Getting historical data: account= Account(customer=Customer(uuid='9e967f23-9e95-11eb-bf8b-0200170058ac'), uuid='a3ba62b5-9e95-11eb-bf8b-0200170058ac', utility_account_id='XXX', meter_type=<MeterType.ELEC: 'ELEC'>, read_resolution=<ReadResolution.HALF_HOUR: 'HALF_HOUR'>) aggregate_type= day start_date= 2022-06-13 00:00:00-05:00 end_date= 2023-06-11 00:00:00-05:00
Traceback (most recent call last):
  File "/Users/brett/pcode/opower/src/demo.py", line 151, in <module>
    asyncio.run(_main())
  File "/Users/brett/.pyenv/versions/3.10.1/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/brett/.pyenv/versions/3.10.1/lib/python3.10/asyncio/base_events.py", line 641, in run_until_complete
    return future.result()
  File "/Users/brett/pcode/opower/src/demo.py", line 121, in _main
    cost_data = await opower.async_get_cost_reads(
  File "/Users/brett/pcode/opower/src/opower/opower.py", line 327, in async_get_cost_reads
    reads = await self._async_get_dated_data(
  File "/Users/brett/pcode/opower/src/opower/opower.py", line 431, in _async_get_dated_data
    reads = await self._async_fetch(
  File "/Users/brett/pcode/opower/src/opower/opower.py", line 475, in _async_fetch
    raise err
  File "/Users/brett/pcode/opower/src/opower/opower.py", line 463, in _async_fetch
    async with self.session.get(
  File "/Users/brett/.virtualenvs/opower-quohth9L/lib/python3.10/site-packages/aiohttp/client.py", line 1141, in __aenter__
    self._resp = await self._coro
  File "/Users/brett/.virtualenvs/opower-quohth9L/lib/python3.10/site-packages/aiohttp/client.py", line 643, in _request
    resp.raise_for_status()
  File "/Users/brett/.virtualenvs/opower-quohth9L/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 1005, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 500, message='Internal Server Error', url=URL('https://cec.opower.com/ei/edge/apis/DataBrowser-v1/cws/cost/utilityAccount/a3ba62b5-9e95-11eb-bf8b-0200170058ac?aggregateType=day&startDate=2022-06-14T00:00:00-05:00&endDate=2023-06-12T00:00:00-05:00')

This data is available on ComEd's website.

However, if I change the time frame with opower's demo.py, keeping the start date the same, but decreasing the end date, the request succeeds. I think that opower may be limiting the number of days for which data can be requested, and possibly may be configured per utility.

➜ python src/demo.py --utility comed --username "brett@brettjackson.org" --password 'XXX' --start_date 2022-06-13T00:00:00-05:00 --end_date 2022-10-11T00:00:00-05:00

Current bill forecast: Forecast(account=Account(customer=Customer(uuid='9e967f23-9e95-11eb-bf8b-0200170058ac'), uuid='a3ba62b5-9e95-11eb-bf8b-0200170058ac', utility_account_id='XXX', meter_type=<MeterType.ELEC: 'ELEC'>, read_resolution=None), start_date=datetime.date(2023, 8, 1), end_date=datetime.date(2023, 8, 30), current_date=datetime.date(2023, 8, 10), unit_of_measure=<UnitOfMeasure.KWH: 'KWH'>, usage_to_date=309.0, cost_to_date=0.0, forecasted_usage=1337.0, forecasted_cost=0.0, typical_usage=1099.0, typical_cost=0.0)

Getting historical data: account= Account(customer=Customer(uuid='9e967f23-9e95-11eb-bf8b-0200170058ac'), uuid='a3ba62b5-9e95-11eb-bf8b-0200170058ac', utility_account_id='XXX', meter_type=<MeterType.ELEC: 'ELEC'>, read_resolution=<ReadResolution.HALF_HOUR: 'HALF_HOUR'>) aggregate_type= day start_date= 2022-06-13 00:00:00-05:00 end_date= 2022-10-11 00:00:00-05:00
start_time  end_time    consumption provided_cost   start_minus_prev_end    end_minus_prev_end
2022-06-13 00:00:00-05:00   2022-06-14 00:00:00-05:00   28.9075 4.626645375 None    None
2022-06-14 00:00:00-05:00   2022-06-15 00:00:00-05:00   41.215  6.59646075  0:00:00 1 day, 0:00:00
2022-06-15 00:00:00-05:00   2022-06-16 00:00:00-05:00   59.275  9.48696375  0:00:00 1 day, 0:00:00
2022-06-16 00:00:00-05:00   2022-06-17 00:00:00-05:00   51.265  8.20496325  0:00:00 1 day, 0:00:00
2022-06-17 00:00:00-05:00   2022-06-18 00:00:00-05:00   36.6475 5.865432375 0:00:00 1 day, 0:00:00
2022-06-18 00:00:00-05:00   2022-06-19 00:00:00-05:00   19.6875 3.150984375 0:00:00 1 day, 0:00:00
2022-06-19 00:00:00-05:00   2022-06-20 00:00:00-05:00   20.095  3.21620475  0:00:00 1 day, 0:00:00
...

My account has been active since sometime in April/May 2021 and had a smart meter before activation.

If you'd like access to my utility account, let me know an email and I can share credentials or a HAR file of requests on ComEd's usage page with you.

tronikos commented 1 year ago

Can you try lowering max_request_days at https://github.com/tronikos/opower/blob/main/src/opower/opower.py#L416 ? Try also with --aggregate_type hour to see if you need to lower the value two lines below.

bjackson commented 1 year ago

It appears the max max_request_days that works is dependent on the times I set.

For example, I run with --start_date 2022-10-13T00:00:00-05:00 --end_date 2023-03-12T00:00:00-06:00 and max_request_days of 140, then it fails. It starts working when max_request <= 126. The URL that fails when max_request_days = 127 is https://cec.opower.com/ei/edge/apis/DataBrowser-v1/cws/cost/utilityAccount/a3ba62b5-9e95-11eb-bf8b-0200170058ac?aggregateType=day&startDate=2022-11-06T00:00:00-05:00&endDate=2023-03-13T00:00:00-05:00'

But if I run it with --start_date 2022-10-13T00:00:00-05:00 --end_date 2023-*05*-12T00:00:00-06:00 and keep it at 140 days, it succeeds.

Running --start_date 2022-10-13T00:00:00-05:00 --end_date 2023-03-12T00:00:00-06:00 --aggregate_type hour works fine with the default value.

Here are the debug logs for day aggregate requests:

`max_req = 120` 
/Users/brett/.virtualenvs/opower-quohth9L/bin/python /Users/brett/pcode/opower/src/demo.py --utility comed --username brett@brettjackson.org --password XXX --start_date 2022-10-13T00:00:00-05:00 --end_date 2023-03-12T00:00:00-06:00 --aggregate_type day --verbose 
DEBUG:/Users/brett/pcode/opower/src/opower/opower.py:Fetching: https://cec.opower.com/ei/edge/apis/multi-account-v1/cws/cec/customers?offset=0&batchSize=100&addressFilter=
DEBUG:/Users/brett/pcode/opower/src/opower/opower.py:Fetching: https://cec.opower.com/ei/edge/apis/bill-forecast-cws-v1/cws/cec/customers/9e967f23-9e95-11eb-bf8b-0200170058ac/combined-forecast
DEBUG:/Users/brett/pcode/opower/src/opower/opower.py:Fetching: https://cec.opower.com/ei/edge/apis/DataBrowser-v1/cws/cost/utilityAccount/a3ba62b5-9e95-11eb-bf8b-0200170058ac?aggregateType=day&startDate=2022-11-13T00%3A00%3A00-06%3A00&endDate=2023-03-13T00%3A00%3A00-05%3A00

Current bill forecast: Forecast(account=Account(customer=Customer(uuid='9e967f23-9e95-11eb-bf8b-0200170058ac'), uuid='a3ba62b5-9e95-11eb-bf8b-0200170058ac', utility_account_id='0855043257', meter_type=, read_resolution=None), start_date=datetime.date(2023, 8, 1), end_date=datetime.date(2023, 8, 30), current_date=datetime.date(2023, 8, 10), unit_of_measure=, usage_to_date=309.0, cost_to_date=0.0, forecasted_usage=1337.0, forecasted_cost=0.0, typical_usage=1099.0, typical_cost=0.0)

Getting historical data: account= Account(customer=Customer(uuid='9e967f23-9e95-11eb-bf8b-0200170058ac'), uuid='a3ba62b5-9e95-11eb-bf8b-0200170058ac', utility_account_id='0855043257', meter_type=, read_resolution=) aggregate_type= day start_date= 2022-10-13 00:00:00-05:00 end_date= 2023-03-12 00:00:00-06:00
DEBUG:/Users/brett/pcode/opower/src/opower/opower.py:Fetching: https://cec.opower.com/ei/edge/apis/DataBrowser-v1/cws/cost/utilityAccount/a3ba62b5-9e95-11eb-bf8b-0200170058ac?aggregateType=day&startDate=2022-10-13T00%3A00%3A00-05%3A00&endDate=2022-11-12T00%3A00%3A00-06%3A00
start_time  end_time    consumption provided_cost   start_minus_prev_end    end_minus_prev_end
2022-10-13 00:00:00-05:00   2022-10-14 00:00:00-05:00   12.9    1.899138    None    None
2022-10-14 00:00:00-05:00   2022-10-15 00:00:00-05:00   12.9325 1.90392265  0:00:00 1 day, 0:00:00
2022-10-15 00:00:00-05:00   2022-10-16 00:00:00-05:00   9.9675  1.46741535  0:00:00 1 day, 0:00:00
2022-10-16 00:00:00-05:00   2022-10-17 00:00:00-05:00   15.535  2.2870627   0:00:00 1 day, 0:00:00
2022-10-17 00:00:00-05:00   2022-10-18 00:00:00-05:00   12.9175 1.90171435  0:00:00 1 day, 0:00:00
2022-10-18 00:00:00-05:00   2022-10-19 00:00:00-05:00   17.7925 2.61941185  0:00:00 1 day, 0:00:00
2022-10-19 00:00:00-05:00   2022-10-20 00:00:00-05:00   22.71   3.3433662   0:00:00 1 day, 0:00:00
2022-10-20 00:00:00-05:00   2022-10-21 00:00:00-05:00   19.55   2.878151    0:00:00 1 day, 0:00:00
2022-10-21 00:00:00-05:00   2022-10-22 00:00:00-05:00   33.99   5.0040078   0:00:00 1 day, 0:00:00
2022-10-22 00:00:00-05:00   2022-10-23 00:00:00-05:00   38.21   5.6252762   0:00:00 1 day, 0:00:00
2022-10-23 00:00:00-05:00   2022-10-24 00:00:00-05:00   42.1425 6.20421885  0:00:00 1 day, 0:00:00
2022-10-24 00:00:00-05:00   2022-10-25 00:00:00-05:00   18.815  2.7699443   0:00:00 1 day, 0:00:00
2022-10-25 00:00:00-05:00   2022-10-26 00:00:00-05:00   21.21   3.1225362   0:00:00 1 day, 0:00:00
2022-10-26 00:00:00-05:00   2022-10-27 00:00:00-05:00   16.5975 2.44348395  0:00:00 1 day, 0:00:00
2022-10-27 00:00:00-05:00   2022-10-28 00:00:00-05:00   11.545  1.6996549   0:00:00 1 day, 0:00:00
2022-10-28 00:00:00-05:00   2022-10-29 00:00:00-05:00   11.9325 1.75670265  0:00:00 1 day, 0:00:00
2022-10-29 00:00:00-05:00   2022-10-30 00:00:00-05:00   11.5725 1.70370345  0:00:00 1 day, 0:00:00
2022-10-30 00:00:00-05:00   2022-10-31 00:00:00-05:00   12.6625 1.86417325  0:00:00 1 day, 0:00:00
2022-10-31 00:00:00-05:00   2022-11-01 00:00:00-05:00   15.7175 2.31393035  0:00:00 1 day, 0:00:00
2022-11-01 00:00:00-05:00   2022-11-02 00:00:00-05:00   18.41   2.7103202   0:00:00 1 day, 0:00:00
2022-11-02 00:00:00-05:00   2022-11-03 00:00:00-05:00   20.6025 3.03310005  0:00:00 1 day, 0:00:00
2022-11-03 00:00:00-05:00   2022-11-04 00:00:00-05:00   8.7375  1.28633475  0:00:00 1 day, 0:00:00
2022-11-04 00:00:00-05:00   2022-11-05 00:00:00-05:00   5.1475  0.75781495  0:00:00 1 day, 0:00:00
2022-11-05 00:00:00-05:00   2022-11-06 00:00:00-05:00   5.115   0.7530303   0:00:00 1 day, 0:00:00
2022-11-06 00:00:00-05:00   2022-11-07 00:00:00-06:00   5.92    0.8715424   0:00:00 1 day, 1:00:00
2022-11-07 00:00:00-06:00   2022-11-08 00:00:00-06:00   10.69   1.5737818   0:00:00 1 day, 0:00:00
2022-11-08 00:00:00-06:00   2022-11-09 00:00:00-06:00   11.07   1.6297254   0:00:00 1 day, 0:00:00
2022-11-09 00:00:00-06:00   2022-11-10 00:00:00-06:00   10.63   1.5649486   0:00:00 1 day, 0:00:00
2022-11-10 00:00:00-06:00   2022-11-11 00:00:00-06:00   17.445  2.5682529   0:00:00 1 day, 0:00:00
2022-11-11 00:00:00-06:00   2022-11-12 00:00:00-06:00   8.2025  1.20757205  0:00:00 1 day, 0:00:00
2022-11-12 00:00:00-06:00   2022-11-13 00:00:00-06:00   4.7275  0.69598255  0:00:00 1 day, 0:00:00
2022-11-13 00:00:00-06:00   2022-11-14 00:00:00-06:00   8.2725  1.21787745  0:00:00 1 day, 0:00:00
2022-11-14 00:00:00-06:00   2022-11-15 00:00:00-06:00   12.04   1.7725288   0:00:00 1 day, 0:00:00
2022-11-15 00:00:00-06:00   2022-11-16 00:00:00-06:00   10.28   1.5134216   0:00:00 1 day, 0:00:00
2022-11-16 00:00:00-06:00   2022-11-17 00:00:00-06:00   10.3075 1.51747015  0:00:00 1 day, 0:00:00
2022-11-17 00:00:00-06:00   2022-11-18 00:00:00-06:00   12.945  1.9057629   0:00:00 1 day, 0:00:00
2022-11-18 00:00:00-06:00   2022-11-19 00:00:00-06:00   14.4625 2.12916925  0:00:00 1 day, 0:00:00
2022-11-19 00:00:00-06:00   2022-11-20 00:00:00-06:00   15.3175 2.25504235  0:00:00 1 day, 0:00:00
2022-11-20 00:00:00-06:00   2022-11-21 00:00:00-06:00   14.2175 2.09310035  0:00:00 1 day, 0:00:00
2022-11-21 00:00:00-06:00   2022-11-22 00:00:00-06:00   13.4975 1.98710195  0:00:00 1 day, 0:00:00
2022-11-22 00:00:00-06:00   2022-11-23 00:00:00-06:00   8.1225  1.19579445  0:00:00 1 day, 0:00:00
2022-11-23 00:00:00-06:00   2022-11-24 00:00:00-06:00   7.1575  1.05372715  0:00:00 1 day, 0:00:00
2022-11-24 00:00:00-06:00   2022-11-25 00:00:00-06:00   7.1225  1.04857445  0:00:00 1 day, 0:00:00
2022-11-25 00:00:00-06:00   2022-11-26 00:00:00-06:00   7.125   1.0489425   0:00:00 1 day, 0:00:00
2022-11-26 00:00:00-06:00   2022-11-27 00:00:00-06:00   6.885   1.0136097   0:00:00 1 day, 0:00:00
2022-11-27 00:00:00-06:00   2022-11-28 00:00:00-06:00   9.865   1.4523253   0:00:00 1 day, 0:00:00
2022-11-28 00:00:00-06:00   2022-11-29 00:00:00-06:00   11.1425 1.64039885  0:00:00 1 day, 0:00:00
2022-11-29 00:00:00-06:00   2022-11-30 00:00:00-06:00   12.615  1.8571803   0:00:00 1 day, 0:00:00
2022-11-30 00:00:00-06:00   2022-12-01 00:00:00-06:00   14.2875 2.10340575  0:00:00 1 day, 0:00:00
2022-12-01 00:00:00-06:00   2022-12-02 00:00:00-06:00   12.81   1.9126611   0:00:00 1 day, 0:00:00
2022-12-02 00:00:00-06:00   2022-12-03 00:00:00-06:00   23.1825 3.461379075 0:00:00 1 day, 0:00:00
2022-12-03 00:00:00-06:00   2022-12-04 00:00:00-06:00   15.8025 2.359471275 0:00:00 1 day, 0:00:00
2022-12-04 00:00:00-06:00   2022-12-05 00:00:00-06:00   14.065  2.10004515  0:00:00 1 day, 0:00:00
2022-12-05 00:00:00-06:00   2022-12-06 00:00:00-06:00   15.295  2.28369645  0:00:00 1 day, 0:00:00
2022-12-06 00:00:00-06:00   2022-12-07 00:00:00-06:00   12.665  1.89101115  0:00:00 1 day, 0:00:00
2022-12-07 00:00:00-06:00   2022-12-08 00:00:00-06:00   15.57   2.3247567   0:00:00 1 day, 0:00:00
2022-12-08 00:00:00-06:00   2022-12-09 00:00:00-06:00   15.5525 2.322143775 0:00:00 1 day, 0:00:00
2022-12-09 00:00:00-06:00   2022-12-10 00:00:00-06:00   25.0375 3.738349125 0:00:00 1 day, 0:00:00
2022-12-10 00:00:00-06:00   2022-12-11 00:00:00-06:00   21.65   3.2325615   0:00:00 1 day, 0:00:00
2022-12-11 00:00:00-06:00   2022-12-12 00:00:00-06:00   34.99   5.2243569   0:00:00 1 day, 0:00:00
2022-12-12 00:00:00-06:00   2022-12-13 00:00:00-06:00   18.7275 2.796203025 0:00:00 1 day, 0:00:00
2022-12-13 00:00:00-06:00   2022-12-14 00:00:00-06:00   17.775  2.65398525  0:00:00 1 day, 0:00:00
2022-12-14 00:00:00-06:00   2022-12-15 00:00:00-06:00   27.7075 4.137006825 0:00:00 1 day, 0:00:00
2022-12-15 00:00:00-06:00   2022-12-16 00:00:00-06:00   19.1275 2.855927025 0:00:00 1 day, 0:00:00
2022-12-16 00:00:00-06:00   2022-12-17 00:00:00-06:00   22.4825 3.356862075 0:00:00 1 day, 0:00:00
2022-12-17 00:00:00-06:00   2022-12-18 00:00:00-06:00   25.695  3.83652045  0:00:00 1 day, 0:00:00
2022-12-18 00:00:00-06:00   2022-12-19 00:00:00-06:00   44.885  6.70177935  0:00:00 1 day, 0:00:00
2022-12-19 00:00:00-06:00   2022-12-20 00:00:00-06:00   34.26   5.1153606   0:00:00 1 day, 0:00:00
2022-12-20 00:00:00-06:00   2022-12-21 00:00:00-06:00   29.68   4.4315208   0:00:00 1 day, 0:00:00
2022-12-21 00:00:00-06:00   2022-12-22 00:00:00-06:00   12.1925 1.820462175 0:00:00 1 day, 0:00:00
2022-12-22 00:00:00-06:00   2022-12-23 00:00:00-06:00   8.6 1.284066    0:00:00 1 day, 0:00:00
2022-12-23 00:00:00-06:00   2022-12-24 00:00:00-06:00   10.6275 1.586792025 0:00:00 1 day, 0:00:00
2022-12-24 00:00:00-06:00   2022-12-25 00:00:00-06:00   13.125  1.95969375  0:00:00 1 day, 0:00:00
2022-12-25 00:00:00-06:00   2022-12-26 00:00:00-06:00   9.6775  1.444947525 0:00:00 1 day, 0:00:00
2022-12-26 00:00:00-06:00   2022-12-27 00:00:00-06:00   9.2525  1.381490775 0:00:00 1 day, 0:00:00
2022-12-27 00:00:00-06:00   2022-12-28 00:00:00-06:00   9.3825  1.400901075 0:00:00 1 day, 0:00:00
2022-12-28 00:00:00-06:00   2022-12-29 00:00:00-06:00   12.45   1.8589095   0:00:00 1 day, 0:00:00
2022-12-29 00:00:00-06:00   2022-12-30 00:00:00-06:00   44.51   6.6457881   0:00:00 1 day, 0:00:00
2022-12-30 00:00:00-06:00   2022-12-31 00:00:00-06:00   19.4275 2.900720025 0:00:00 1 day, 0:00:00
2022-12-31 00:00:00-06:00   2023-01-01 00:00:00-06:00   8.6325  1.288918575 0:00:00 1 day, 0:00:00
2023-01-01 00:00:00-06:00   2023-01-02 00:00:00-06:00   8.0925  1.208291175 0:00:00 1 day, 0:00:00
2023-01-02 00:00:00-06:00   2023-01-03 00:00:00-06:00   24.2625 3.622633875 0:00:00 1 day, 0:00:00
2023-01-03 00:00:00-06:00   2023-01-04 00:00:00-06:00   43.3025 6.465496275 0:00:00 1 day, 0:00:00
2023-01-04 00:00:00-06:00   2023-01-05 00:00:00-06:00   38.46   3.9192405   0:00:00 1 day, 0:00:00
2023-01-05 00:00:00-06:00   2023-01-06 00:00:00-06:00   23.795  2.2676012   0:00:00 1 day, 0:00:00
2023-01-06 00:00:00-06:00   2023-01-07 00:00:00-06:00   26.275  2.8827161   0:00:00 1 day, 0:00:00
2023-01-07 00:00:00-06:00   2023-01-08 00:00:00-06:00   24.9875 2.43896185  0:00:00 1 day, 0:00:00
2023-01-08 00:00:00-06:00   2023-01-09 00:00:00-06:00   17.8925 1.705321225 0:00:00 1 day, 0:00:00
2023-01-09 00:00:00-06:00   2023-01-10 00:00:00-06:00   20.3    1.8965251   0:00:00 1 day, 0:00:00
2023-01-10 00:00:00-06:00   2023-01-11 00:00:00-06:00   24.695  3.2049756   0:00:00 1 day, 0:00:00
2023-01-11 00:00:00-06:00   2023-01-12 00:00:00-06:00   17.77   1.725693375 0:00:00 1 day, 0:00:00
2023-01-12 00:00:00-06:00   2023-01-13 00:00:00-06:00   23.73   2.122299975 0:00:00 1 day, 0:00:00
2023-01-13 00:00:00-06:00   2023-01-14 00:00:00-06:00   18.8025 2.04648485  0:00:00 1 day, 0:00:00
2023-01-14 00:00:00-06:00   2023-01-15 00:00:00-06:00   22.1425 2.5160867   0:00:00 1 day, 0:00:00
2023-01-15 00:00:00-06:00   2023-01-16 00:00:00-06:00   15.25   1.286492075 0:00:00 1 day, 0:00:00
2023-01-16 00:00:00-06:00   2023-01-17 00:00:00-06:00   32.0    2.8796108   0:00:00 1 day, 0:00:00
2023-01-17 00:00:00-06:00   2023-01-18 00:00:00-06:00   41.02   3.688363075 0:00:00 1 day, 0:00:00
2023-01-18 00:00:00-06:00   2023-01-19 00:00:00-06:00   34.15   3.247907975 0:00:00 1 day, 0:00:00
2023-01-19 00:00:00-06:00   2023-01-20 00:00:00-06:00   27.4875 2.4340504   0:00:00 1 day, 0:00:00
2023-01-20 00:00:00-06:00   2023-01-21 00:00:00-06:00   39.535  3.677085675 0:00:00 1 day, 0:00:00
2023-01-21 00:00:00-06:00   2023-01-22 00:00:00-06:00   33.88   3.33415245  0:00:00 1 day, 0:00:00
2023-01-22 00:00:00-06:00   2023-01-23 00:00:00-06:00   31.07   2.961562125 0:00:00 1 day, 0:00:00
2023-01-23 00:00:00-06:00   2023-01-24 00:00:00-06:00   28.7175 2.60095755  0:00:00 1 day, 0:00:00
2023-01-24 00:00:00-06:00   2023-01-25 00:00:00-06:00   26.1475 2.39680135  0:00:00 1 day, 0:00:00
2023-01-25 00:00:00-06:00   2023-01-26 00:00:00-06:00   34.9    3.124541125 0:00:00 1 day, 0:00:00
2023-01-26 00:00:00-06:00   2023-01-27 00:00:00-06:00   25.605  2.37536605  0:00:00 1 day, 0:00:00
2023-01-27 00:00:00-06:00   2023-01-28 00:00:00-06:00   30.2125 2.486395825 0:00:00 1 day, 0:00:00
2023-01-28 00:00:00-06:00   2023-01-29 00:00:00-06:00   28.9925 2.50918945  0:00:00 1 day, 0:00:00
2023-01-29 00:00:00-06:00   2023-01-30 00:00:00-06:00   26.84   2.352525275 0:00:00 1 day, 0:00:00
2023-01-30 00:00:00-06:00   2023-01-31 00:00:00-06:00   33.9575 3.1569132   0:00:00 1 day, 0:00:00
2023-01-31 00:00:00-06:00   2023-02-01 00:00:00-06:00   33.1475 3.23518315  0:00:00 1 day, 0:00:00
2023-02-01 00:00:00-06:00   2023-02-02 00:00:00-06:00   30.65   2.835511375 0:00:00 1 day, 0:00:00
2023-02-02 00:00:00-06:00   2023-02-03 00:00:00-06:00   26.235  2.22147565  0:00:00 1 day, 0:00:00
2023-02-03 00:00:00-06:00   2023-02-04 00:00:00-06:00   38.46   3.433606275 0:00:00 1 day, 0:00:00
2023-02-04 00:00:00-06:00   2023-02-05 00:00:00-06:00   29.635  2.4040409   0:00:00 1 day, 0:00:00
2023-02-05 00:00:00-06:00   2023-02-06 00:00:00-06:00   24.015  1.987782425 0:00:00 1 day, 0:00:00
2023-02-06 00:00:00-06:00   2023-02-07 00:00:00-06:00   36.415  3.08948345  0:00:00 1 day, 0:00:00
2023-02-07 00:00:00-06:00   2023-02-08 00:00:00-06:00   30.1375 2.520288825 0:00:00 1 day, 0:00:00
2023-02-08 00:00:00-06:00   2023-02-09 00:00:00-06:00   28.425  2.48002685  0:00:00 1 day, 0:00:00
2023-02-09 00:00:00-06:00   2023-02-10 00:00:00-06:00   29.945  2.521370325 0:00:00 1 day, 0:00:00
2023-02-10 00:00:00-06:00   2023-02-11 00:00:00-06:00   30.6175 2.545089425 0:00:00 1 day, 0:00:00
2023-02-11 00:00:00-06:00   2023-02-12 00:00:00-06:00   23.1925 1.936913725 0:00:00 1 day, 0:00:00
2023-02-12 00:00:00-06:00   2023-02-13 00:00:00-06:00   16.815  1.439623075 0:00:00 1 day, 0:00:00
2023-02-13 00:00:00-06:00   2023-02-14 00:00:00-06:00   24.5225 2.086496675 0:00:00 1 day, 0:00:00
2023-02-14 00:00:00-06:00   2023-02-15 00:00:00-06:00   24.565  1.990961875 0:00:00 1 day, 0:00:00
2023-02-15 00:00:00-06:00   2023-02-16 00:00:00-06:00   30.2725 2.655019775 0:00:00 1 day, 0:00:00
2023-02-16 00:00:00-06:00   2023-02-17 00:00:00-06:00   29.025  2.792119325 0:00:00 1 day, 0:00:00
2023-02-17 00:00:00-06:00   2023-02-18 00:00:00-06:00   30.1525 2.911893425 0:00:00 1 day, 0:00:00
2023-02-18 00:00:00-06:00   2023-02-19 00:00:00-06:00   23.6075 1.86678915  0:00:00 1 day, 0:00:00
2023-02-19 00:00:00-06:00   2023-02-20 00:00:00-06:00   13.82   1.0698706   0:00:00 1 day, 0:00:00
2023-02-20 00:00:00-06:00   2023-02-21 00:00:00-06:00   33.2675 2.888854125 0:00:00 1 day, 0:00:00
2023-02-21 00:00:00-06:00   2023-02-22 00:00:00-06:00   29.4825 2.43592425  0:00:00 1 day, 0:00:00
2023-02-22 00:00:00-06:00   2023-02-23 00:00:00-06:00   29.3575 2.6474878   0:00:00 1 day, 0:00:00
2023-02-23 00:00:00-06:00   2023-02-24 00:00:00-06:00   37.54   3.15816995  0:00:00 1 day, 0:00:00
2023-02-24 00:00:00-06:00   2023-02-25 00:00:00-06:00   31.11   2.71141405  0:00:00 1 day, 0:00:00
2023-02-25 00:00:00-06:00   2023-02-26 00:00:00-06:00   13.7875 1.2186579   0:00:00 1 day, 0:00:00
2023-02-26 00:00:00-06:00   2023-02-27 00:00:00-06:00   13.62   1.119590325 0:00:00 1 day, 0:00:00
2023-02-27 00:00:00-06:00   2023-02-28 00:00:00-06:00   43.655  3.46832725  0:00:00 1 day, 0:00:00
2023-02-28 00:00:00-06:00   2023-03-01 00:00:00-06:00   39.095  3.550899725 0:00:00 1 day, 0:00:00
2023-03-01 00:00:00-06:00   2023-03-02 00:00:00-06:00   24.9425 2.156150875 0:00:00 1 day, 0:00:00
2023-03-02 00:00:00-06:00   2023-03-03 00:00:00-06:00   29.665  2.88610135  0:00:00 1 day, 0:00:00
2023-03-03 00:00:00-06:00   2023-03-04 00:00:00-06:00   21.8275 2.252724925 0:00:00 1 day, 0:00:00
2023-03-04 00:00:00-06:00   2023-03-05 00:00:00-06:00   23.81   2.1958558   0:00:00 1 day, 0:00:00
2023-03-05 00:00:00-06:00   2023-03-06 00:00:00-06:00   19.9425 1.654511275 0:00:00 1 day, 0:00:00
2023-03-06 00:00:00-06:00   2023-03-07 00:00:00-06:00   28.05   2.56680775  0:00:00 1 day, 0:00:00
2023-03-07 00:00:00-06:00   2023-03-08 00:00:00-06:00   23.1575 1.9953014   0:00:00 1 day, 0:00:00
2023-03-08 00:00:00-06:00   2023-03-09 00:00:00-06:00   27.77   2.430727    0:00:00 1 day, 0:00:00
2023-03-09 00:00:00-06:00   2023-03-10 00:00:00-06:00   25.0575 2.24815015  0:00:00 1 day, 0:00:00
2023-03-10 00:00:00-06:00   2023-03-11 00:00:00-06:00   27.81   2.5086128   0:00:00 1 day, 0:00:00
2023-03-11 00:00:00-06:00   2023-03-12 00:00:00-06:00   18.685  1.617906075 0:00:00 1 day, 0:00:00
2023-03-12 00:00:00-06:00   2023-03-13 00:00:00-05:00   32.9625 2.75841355  0:00:00 23:00:00
2023-03-13 00:00:00-05:00   2023-03-14 00:00:00-05:00   31.2025 2.617630725 0:00:00 1 day, 0:00:00
`max_req = 127` 

/Users/brett/.virtualenvs/opower-quohth9L/bin/python /Users/brett/pcode/opower/src/demo.py --utility comed --username brett@brettjackson.org --password XXX --start_date 2022-10-13T00:00:00-05:00 --end_date 2023-03-12T00:00:00-06:00 --aggregate_type day --verbose 
DEBUG:/Users/brett/pcode/opower/src/opower/opower.py:Fetching: https://cec.opower.com/ei/edge/apis/multi-account-v1/cws/cec/customers?offset=0&batchSize=100&addressFilter=
DEBUG:/Users/brett/pcode/opower/src/opower/opower.py:Fetching: https://cec.opower.com/ei/edge/apis/bill-forecast-cws-v1/cws/cec/customers/9e967f23-9e95-11eb-bf8b-0200170058ac/combined-forecast

Current bill forecast: Forecast(account=Account(customer=Customer(uuid='9e967f23-9e95-11eb-bf8b-0200170058ac'), uuid='a3ba62b5-9e95-11eb-bf8b-0200170058ac', utility_account_id='0855043257', meter_type=, read_resolution=None), start_date=datetime.date(2023, 8, 1), end_date=datetime.date(2023, 8, 30), current_date=datetime.date(2023, 8, 10), unit_of_measure=, usage_to_date=309.0, cost_to_date=0.0, forecasted_usage=1337.0, forecasted_cost=0.0, typical_usage=1099.0, typical_cost=0.0)

Getting historical data: account= Account(customer=Customer(uuid='9e967f23-9e95-11eb-bf8b-0200170058ac'), uuid='a3ba62b5-9e95-11eb-bf8b-0200170058ac', utility_account_id='0855043257', meter_type=, read_resolution=) aggregate_type= day start_date= 2022-10-13 00:00:00-05:00 end_date= 2023-03-12 00:00:00-06:00
DEBUG:/Users/brett/pcode/opower/src/opower/opower.py:Fetching: https://cec.opower.com/ei/edge/apis/DataBrowser-v1/cws/cost/utilityAccount/a3ba62b5-9e95-11eb-bf8b-0200170058ac?aggregateType=day&startDate=2022-11-06T00%3A00%3A00-05%3A00&endDate=2023-03-13T00%3A00%3A00-05%3A00
Traceback (most recent call last):
  File "/Users/brett/pcode/opower/src/demo.py", line 151, in 
    asyncio.run(_main())
  File "/Users/brett/.pyenv/versions/3.10.1/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/brett/.pyenv/versions/3.10.1/lib/python3.10/asyncio/base_events.py", line 641, in run_until_complete
    return future.result()
  File "/Users/brett/pcode/opower/src/demo.py", line 121, in _main
    cost_data = await opower.async_get_cost_reads(
  File "/Users/brett/pcode/opower/src/opower/opower.py", line 327, in async_get_cost_reads
    reads = await self._async_get_dated_data(
  File "/Users/brett/pcode/opower/src/opower/opower.py", line 431, in _async_get_dated_data
    reads = await self._async_fetch(
  File "/Users/brett/pcode/opower/src/opower/opower.py", line 475, in _async_fetch
    raise err
  File "/Users/brett/pcode/opower/src/opower/opower.py", line 463, in _async_fetch
    async with self.session.get(
  File "/Users/brett/.virtualenvs/opower-quohth9L/lib/python3.10/site-packages/aiohttp/client.py", line 1141, in __aenter__
    self._resp = await self._coro
  File "/Users/brett/.virtualenvs/opower-quohth9L/lib/python3.10/site-packages/aiohttp/client.py", line 643, in _request
    resp.raise_for_status()
  File "/Users/brett/.virtualenvs/opower-quohth9L/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 1005, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 500, message='Internal Server Error', url=URL('https://cec.opower.com/ei/edge/apis/DataBrowser-v1/cws/cost/utilityAccount/a3ba62b5-9e95-11eb-bf8b-0200170058ac?aggregateType=day&startDate=2022-11-06T00:00:00-05:00&endDate=2023-03-13T00:00:00-05:00')

I notice that the first request in each of these has the startDate greater than the date passed in as a parameter to the script.

tronikos commented 1 year ago

The code fetches batches backwards see https://github.com/tronikos/opower/blob/main/src/opower/opower.py#L420 so that it can terminate early when there is no more data in the past. For my PG&E account, daily and hourly data is only available for the past 3 years. If the provided date range was 10 years and since, at least for PG&E, I can request data in ~1 year ranges, going backwards we only have to make 4 requests. If we were going forward we would have to make 10 requests.

tronikos commented 1 year ago

It's strange that the max_request_days depends on the start and end date. Can you try with --aggregate_type bill and then experiment with different start and end dates and see if there is any correlation to the bill start and end dates? On my utility website opower always requests daily aggregated data between the bill range, ~30 days, and hourly aggregated data within a single day. By trial and error I set the current values of max_request_days in an attempt to limit outgoing HTTP requests.

warren1201 commented 1 year ago

I am also having an issue with Comed. See below. I am not a programmer so I can't really tell why. The two factor is turned off and there is no other option.

Logger: homeassistant.components.opower.coordinator Source: helpers/update_coordinator.py:290 Integration: Opower (documentation, issues) First occurred: November 1, 2023 at 11:47:41 PM (3 occurrences) Last logged: 2:03:25 AM

Unexpected error fetching Opower data: 400, message='Bad Request', url=URL('https://secure.comed.com/api/Services/OpowerService.svc/GetOpowerToken') Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/opower/opower.py", line 193, in async_login self.access_token = await self.utility.async_login( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/opower/utilities/exelon.py", line 209, in async_login async with session.post( File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 1141, in aenter self._resp = await self._coro ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 643, in _request resp.raise_for_status() File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1005, in raise_for_status raise ClientResponseError( aiohttp.client_exceptions.ClientResponseError: 400, message='Bad Request', url=URL('https://secure.comed.com/api/Services/OpowerService.svc/GetOpowerToken')

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 290, in _async_refresh self.data = await self._async_update_data() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/opower/coordinator.py", line 79, in _async_update_data await self.api.async_login() File "/usr/local/lib/python3.11/site-packages/opower/opower.py", line 201, in async_login raise CannotConnect(err) opower.exceptions.CannotConnect: 400, message='Bad Request', url=URL('https://secure.comed.com/api/Services/OpowerService.svc/GetOpowerToken')

tronikos commented 1 year ago

@warren1201 your issue is different than this. Yours fail while getting the Opower access token. Most likely the request now requires Referer similar to https://github.com/tronikos/opower/pull/48 Without access to a ComEd account I unfortunately cannot fix this.