firstof9 / ha-openei

OpenEI integration for Home Assistant
MIT License
15 stars 2 forks source link

[BUG] OpenEI API rate limit exceeded #65

Closed DanaGoyette closed 1 year ago

DanaGoyette commented 2 years ago

Version of ha-openei

0.1.8

Describe the bug

My OpenEI integration keeps dropping back to a state of Error fetching openei data:.

The fact that there's no content or explanation after that dangling colon seems like a bug to me; it should have exception text or something.

When I looked at the logs before I restarted homeassistant, I saw that the API had replied with a message about having exceeded the rate limit.

Debug log


2022-10-27 17:07:36.596 INFO (MainThread) [custom_components.openei]
-------------------------------------------------------------------
OpenEI
Version: 0.1.8
This is a custom integration!
If you have any issues with this you need to open an issue here:
https://github.com/firstof9/ha-openei/issues
-------------------------------------------------------------------
2022-10-27 17:07:36.596 DEBUG (MainThread) [custom_components.openei] config_entry: {'api_key': <REDACTED>, 'location': <ZIP>, 'radius': 0, 'utility': 'Pacific Gas & Electric Co', 'rate_plan': '62d05f87d2612e35a469bb87', 'manual_plan': ''}
2022-10-27 17:07:36.596 DEBUG (MainThread) [custom_components.openei] updated_config: {'api_key': <REDACTED>, 'location': <ZIP>, 'radius': 0, 'utility': 'Pacific Gas & Electric Co', 'rate_plan': '62d05f87d2612e35a469bb87', 'manual_plan': ''}
2022-10-27 17:07:36.596 DEBUG (MainThread) [custom_components.openei] Data will be updated at the top of every hour.
2022-10-27 17:07:36.597 DEBUG (MainThread) [custom_components.openei] Next update in 3144 seconds.
2022-10-27 17:07:37.133 ERROR (MainThread) [custom_components.openei] Error fetching openei data:
2022-10-27 17:07:37.136 DEBUG (MainThread) [custom_components.openei] Finished fetching openei data in 0.539 seconds (success: False)
2022-10-27 17:07:37.136 WARNING (MainThread) [homeassistant.config_entries] Config entry 'Pacific Gas & Electric Co' for openei integration not ready yet: None; Retrying in background
2022-10-27 17:07:40.863 DEBUG (MainThread) [custom_components.openei] config_entry: {'api_key': <REDACTED>, 'location': <ZIP>, 'radius': 0, 'utility': 'Pacific Gas & Electric Co', 'rate_plan': '62d05f87d2612e35a469bb87', 'manual_plan': ''}
2022-10-27 17:07:40.864 DEBUG (MainThread) [custom_components.openei] updated_config: {'api_key': <REDACTED>, 'location': <ZIP>, 'radius': 0, 'utility': 'Pacific Gas & Electric Co', 'rate_plan': '62d05f87d2612e35a469bb87', 'manual_plan': ''}
2022-10-27 17:07:40.864 DEBUG (MainThread) [custom_components.openei] Data will be updated at the top of every hour.
2022-10-27 17:07:40.864 DEBUG (MainThread) [custom_components.openei] Next update in 3140 seconds.
2022-10-27 17:07:41.398 ERROR (MainThread) [custom_components.openei] Error fetching openei data:
2022-10-27 17:07:41.399 DEBUG (MainThread) [custom_components.openei] Finished fetching openei data in 0.535 seconds (success: False)
2022-10-27 17:08:07.733 DEBUG (MainThread) [custom_components.openei] config_entry: {'api_key': <REDACTED>, 'location': <ZIP>, 'radius': 0, 'utility': 'Pacific Gas & Electric Co', 'rate_plan': '62d05f87d2612e35a469bb87', 'manual_plan': ''}
2022-10-27 17:08:07.733 DEBUG (MainThread) [custom_components.openei] updated_config: {'api_key': <REDACTED>, 'location': <ZIP>, 'radius': 0, 'utility': 'Pacific Gas & Electric Co', 'rate_plan': '62d05f87d2612e35a469bb87', 'manual_plan': ''}
2022-10-27 17:08:07.734 DEBUG (MainThread) [custom_components.openei] Data will be updated at the top of every hour.
2022-10-27 17:08:07.734 DEBUG (MainThread) [custom_components.openei] Next update in 3113 seconds.
2022-10-27 17:08:08.188 ERROR (MainThread) [custom_components.openei] Error fetching openei data:
2022-10-27 17:08:08.189 DEBUG (MainThread) [custom_components.openei] Finished fetching openei data in 0.455 seconds (success: False)
2022-10-27 17:08:08.189 WARNING (MainThread) [homeassistant.config_entries] Config entry 'Pacific Gas & Electric Co' for openei integration not ready yet: None; Retrying in background

Next refresh in 3140 seconds? But then it retries in way less time than that.

At some point, all the retries also blow up the rate limit from the API. From before I restarted:

2022-10-27 12:58:03.570 ERROR (MainThread) [custom_components.openei] Error fetching openei data: 
2022-10-27 12:59:24.073 ERROR (SyncWorker_6) [openeihttp] Error: You have exceeded your rate limit. Try again later or contact us at https://api.openei.org:443/contact/ for assistance
2022-10-27 12:59:24.074 ERROR (MainThread) [custom_components.openei] Error fetching openei data: 
2022-10-27 13:00:00.425 ERROR (SyncWorker_0) [openeihttp] Error: You have exceeded your rate limit. Try again later or contact us at https://api.openei.org:443/contact/ for assistance
2022-10-27 13:00:00.427 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/config/custom_components/openei/__init__.py", line 117, in _async_refresh_data
    self._data = await self.hass.async_add_executor_job(
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/openei/__init__.py", line 148, in get_sensors
    rate.update()
  File "/usr/local/lib/python3.10/site-packages/openeihttp/__init__.py", line 113, in update
    self.update_data()
  File "/usr/local/lib/python3.10/site-packages/openeihttp/__init__.py", line 143, in update_data
    raise APIError
openeihttp.APIError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/config/custom_components/openei/__init__.py", line 121, in _async_refresh_data
    raise UpdateFailed() from exception
homeassistant.helpers.update_coordinator.UpdateFailed
2022-10-27 13:00:00.509 ERROR (SyncWorker_5) [openeihttp] Error: You have exceeded your rate limit. Try again later or contact us at https://api.openei.org:443/contact/ for assistance
2022-10-27 13:00:00.511 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/config/custom_components/openei/__init__.py", line 117, in _async_refresh_data
    self._data = await self.hass.async_add_executor_job(
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/openei/__init__.py", line 148, in get_sensors
    rate.update()
  File "/usr/local/lib/python3.10/site-packages/openeihttp/__init__.py", line 113, in update
    self.update_data()
  File "/usr/local/lib/python3.10/site-packages/openeihttp/__init__.py", line 143, in update_data
    raise APIError
openeihttp.APIError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/config/custom_components/openei/__init__.py", line 121, in _async_refresh_data
    raise UpdateFailed() from exception
homeassistant.helpers.update_coordinator.UpdateFailed
2022-10-27 13:00:00.522 ERROR (SyncWorker_4) [openeihttp] Error: You have exceeded your rate limit. Try again later or contact us at https://api.openei.org:443/contact/ for assistance
2022-10-27 13:00:00.524 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/config/custom_components/openei/__init__.py", line 117, in _async_refresh_data
    self._data = await self.hass.async_add_executor_job(
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/openei/__init__.py", line 148, in get_sensors
    rate.update()
  File "/usr/local/lib/python3.10/site-packages/openeihttp/__init__.py", line 113, in update
    self.update_data()
  File "/usr/local/lib/python3.10/site-packages/openeihttp/__init__.py", line 143, in update_data
    raise APIError
openeihttp.APIError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/config/custom_components/openei/__init__.py", line 121, in _async_refresh_data
    raise UpdateFailed() from exception
homeassistant.helpers.update_coordinator.UpdateFailed
2022-10-27 13:00:00.533 ERROR (SyncWorker_6) [openeihttp] Error: You have exceeded your rate limit. Try again later or contact us at https://api.openei.org:443/contact/ for assistance

Now, is the rate limiting the cause of the error ("Error fetching openei data: "), or is the error the cause of the retries and thus the rate limiting? I'll have to try disabling the integration for a full day to see if it recovers.

DanaGoyette commented 2 years ago

With debug logging enabled, nothing changes in the logs (nothing DEBUG between the ERROR messages):

2022-10-27 17:57:36.671 ERROR (MainThread) [custom_components.openei] Error fetching openei data: 
2022-10-27 17:58:17.276 ERROR (MainThread) [custom_components.openei] Error fetching openei data: 
2022-10-27 17:59:37.815 ERROR (MainThread) [custom_components.openei] Error fetching openei data: 
2022-10-27 18:00:00.595 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/config/custom_components/openei/__init__.py", line 117, in _async_refresh_data
    self._data = await self.hass.async_add_executor_job(
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/openei/__init__.py", line 153, in get_sensors
    value = getattr(rate, SENSOR_TYPES[sensor].key)
  File "/usr/local/lib/python3.10/site-packages/openeihttp/__init__.py", line 197, in current_adjustment
    adj = self._data["energyratestructure"][rate_structure][0]["adj"]
KeyError: 'adj'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/config/custom_components/openei/__init__.py", line 121, in _async_refresh_data
    raise UpdateFailed() from exception
homeassistant.helpers.update_coordinator.UpdateFailed
firstof9 commented 2 years ago

The fact that there's no content or explanation after that dangling colon seems like a bug to me

It isn't, that means the data returned is null.

I'll work on handling a rate limit error better.

firstof9 commented 2 years ago

The latest beta version should handle rate limit issues gracefully now. Please give it a try.