API error 429 - Quota exceeded

[codemunkie15]

Version of the custom_component

v0.6.2

Describe the bug

I have the update interval set at 10 seconds and crop mode set as Combine, and this uses up all the request quota of the Photos API in around 15 hours (Photos API allows 10,000 requests per day). Is there any way to reduce the number of API calls that are happening without increasing the update interval, so I can keep it running 24/7 please? Any more caching that could be implemented?

The error handling around this also isn't great. It starts spamming the logs with the below error even after I set the update interval to Never.

Debug log

2024-04-14 00:38:33.201 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 452, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_app.py", line 543, in _handle
    resp = await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_middlewares.py", line 114, in impl
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 92, in security_filter_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 83, in forwarded_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 26, in request_context_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 88, in ban_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 236, in auth_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/headers.py", line 32, in headers_middleware
    response = await handler(request)
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/http.py", line 73, in handle
    result = await handler(request, **request.match_info)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/camera/__init__.py", line 823, in get
    return await self.handle(request, camera)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/camera/__init__.py", line 841, in handle
    image = await _async_get_image(
            ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/camera/__init__.py", line 192, in _async_get_image
    else await camera.async_camera_image(width=width, height=height)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/google_photos/camera.py", line 145, in async_camera_image
    return await self.coordinator.get_media_data(width, height)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/google_photos/coordinator.py", line 262, in get_media_data
    result = await self._get_combined_media_data(width, height)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/google_photos/coordinator.py", line 309, in _get_combined_media_data
    await self._get_media_by_id(secondary_media.get("id")),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/google_photos/coordinator.py", line 349, in _get_media_by_id
    return await self.hass.async_add_executor_job(_get_media_item)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/google_photos/coordinator.py", line 347, in _get_media_item
    return service.mediaItems().get(mediaItemId=media_id).execute()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/googleapiclient/http.py", line 938, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 429 when requesting https://photoslibrary.googleapis.com/v1/mediaItems/AGd-TmPZTJYyIJTBT6e70Ohn1n1qXRf2H_SQstaTN7rxtNfKOrN_-ojM8cF-2W9OXYVW40PpCNbf5GxDPw_mY1NE_9vwY77bnQ?alt=json returned "Quota exceeded for quota metric 'All requests' and limit 'All requests per day' of service 'photoslibrary.googleapis.com' for consumer 'project_number:608688248141'.". Details: "[{'@type': 'type.googleapis.com/google.rpc.ErrorInfo', 'reason': 'RATE_LIMIT_EXCEEDED', 'domain': 'googleapis.com', 'metadata': {'quota_metric': 'photoslibrary.googleapis.com/all_requests', 'quota_limit_value': '10000', 'quota_location': 'global', 'consumer': 'projects/608688248141', 'service': 'photoslibrary.googleapis.com', 'quota_limit': 'ApiCallsPerProjectPerDay'}}, {'@type': 'type.googleapis.com/google.rpc.Help', 'links': [{'description': 'Request a higher quota limit.', 'url': 'https://cloud.google.com/docs/quota#requesting_higher_quota'}]}]">

[Daanoz]

This will be tricky to solve, technically the integration could cache images for a longer time so they would not require to be redownloaded, but this would be against Google's Acceptable Use policy.

Alternatively, thinking outside the box a bit, you could try setting up this integration twice in Home Assistant, with 2 different set of credentials. In your HA dashboard dynamically switch the entity that is used to display the photo's, for the first half of day entity_1, the second half entity_2... No guarantees that it works, but it could...

You could also maybe add some smartness in when the images are refreshed. I doubt you are looking 24/7 to your dashboard. Basically disable the automatic refresh, and then manually trigger the update every 10 seconds when you are at home, not a sleep or whatever.

[codemunkie15]

Thanks for the feedback. I have already implemented changing the refresh interval depending on home presence etc which should help. I have also manually increased the album refresh time in the source code to reduce the API calls as my album doesn't change often.

I am also noticing that album data is lost on a Home Assistant restart, so if I have a large album (20k photos) and I have to restart Home Assistant a few times during a day, it's easily going to max out the 10,000 API request limit, re-populating the album data every time. Are there any plans to store album data permanently, to survive a restart?

Thanks

[janstadt]

Im running into this as well. Would love some level of caching or something as i end up with the broken image icon on 3 devices after a few restarts.