search

gilesknap / gphotos-sync

Google Photos and Albums backup with Google Photos Library API
Apache License 2.0
2.03k stars 168 forks source link

requests.exceptions.HTTPError: 429 Client Error: Too Many Requests #320

Closed qrkourier closed 2 years ago

qrkourier commented 3 years ago

Describe the bug I'm unable to re-sync to download my newer photos despite having successfully done some in prior years. The search API is responding with a backoff error, and the 429-response branch isn't quite addressing this particular issue. I plan to work on the problem in my fork and to track progress and gather feedback here.

To Reproduce

~/Sites/gphotos-sync 429-reponse
# check out branch "429-response"
❯ pip install --editable .                                                               
Looking in indexes: https://pypi.org/simple, https://bitbucket-tools:****@netfoundry.jfrog.io/netfoundry/api/pypi/python/simple
Obtaining file:///home/kbingham/Sites/gphotos-sync
Requirement already satisfied: attrs in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from gphotos-sync==2.14.0+5.g497c1ce) (21.2.0)
Requirement already satisfied: exif in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from gphotos-sync==2.14.0+5.g497c1ce) (1.3.1)
Requirement already satisfied: appdirs in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from gphotos-sync==2.14.0+5.g497c1ce) (1.4.4)
Requirement already satisfied: requests_oauthlib in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from gphotos-sync==2.14.0+5.g497c1ce) (1.3.0)
Requirement already satisfied: pyyaml>=4.2b1 in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from gphotos-sync==2.14.0+5.g497c1ce) (5.4.1)
Requirement already satisfied: psutil in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from gphotos-sync==2.14.0+5.g497c1ce) (5.8.0)
Requirement already satisfied: plum-py>=0.5.0 in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from exif->gphotos-sync==2.14.0+5.g497c1ce) (0.7.0)
Requirement already satisfied: requests>=2.0.0 in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from requests_oauthlib->gphotos-sync==2.14.0+5.g497c1ce) (2.25.1)
Requirement already satisfied: oauthlib>=3.0.0 in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from requests_oauthlib->gphotos-sync==2.14.0+5.g497c1ce) (3.1.1)
Requirement already satisfied: certifi>=2017.4.17 in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from requests>=2.0.0->requests_oauthlib->gphotos-sync==2.14.0+5.g497c1ce) (2021.5.30)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from requests>=2.0.0->requests_oauthlib->gphotos-sync==2.14.0+5.g497c1ce) (1.26.5)
Requirement already satisfied: chardet<5,>=3.0.2 in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from requests>=2.0.0->requests_oauthlib->gphotos-sync==2.14.0+5.g497c1ce) (4.0.0)
Requirement already satisfied: idna<3,>=2.5 in /home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from requests>=2.0.0->requests_oauthlib->gphotos-sync==2.14.0+5.g497c1ce) (2.10)
Installing collected packages: gphotos-sync
  Attempting uninstall: gphotos-sync
    Found existing installation: gphotos-sync 2.14.0+5.g497c1ce
    Uninstalling gphotos-sync-2.14.0+5.g497c1ce:
      Successfully uninstalled gphotos-sync-2.14.0+5.g497c1ce
  Running setup.py develop for gphotos-sync
Successfully installed gphotos-sync-2.14.0+5.g497c1ce

~/Sites/gphotos-sync 429-reponse 
❯ pip show gphotos-sync
Name: gphotos-sync
Version: 2.14.0+5.g497c1ce
Summary: Google Photos and Albums backup tool
Home-page: https://github.com/gilesknap/gphotos-sync
Author: Giles Knap
Author-email: gilesknap@gmail.com
License: MIT
Location: /home/kbingham/Sites/gphotos-sync
Requires: attrs, exif, appdirs, requests-oauthlib, pyyaml, psutil
Required-by: 

❯ gphotos-sync --retry-download --max-threads 3 --log-level warn ~/Downloads/from_gphotos
07-28 22:52:03 WARNING  gphotos-sync 2.14.0+5.g497c1ce 2021-07-28 22:52:03.683065 
07-28 22:52:04 WARNING  Indexing Google Photos Files ... 
07-28 22:52:05 WARNING  indexed 6 items 
07-28 22:52:05 WARNING  Downloading Photos ... 
07-28 22:54:39 ERROR    FAILURE 1 downloading photos/2020/06/VID_20200604_201343.mp4 - 400 Client Error: Bad Request for url: https://video-downloads.googleusercontent.com/AGQNM9LFZ2qrswOXVFUH0ZnxMb51VQiQyZ-CDPk-4Z1eHp9QrT9_rD1pRenCLKpKCx_ulQ9HBGAwQ2zXfC_lccD1H96d2s9gUOk-k9F8sfJ0dY63fIGBmgG39W7VfVmJ3n8IePdbNXwimkLNQdPCk6f3eYozj8JTOTFrVoooy2XuiHyQw9hRM0esFapuQuSQDLNFhR9q-VKzda4RuezpB2YnzyCIaJJpb7_wfK05qJlIidLsAWg9ZoDYZHo15MM7KTA3540sLtfMZOynf7P5gO6tiWJg_Ml4f9BCTWbimAL5FV0wkVtlJyL6ijrpbFHW6g_9KVltrSk7X4FBaIjfYji9U1DuVUBeGvDz9DzqpcXqZdmF3fBfKQf0e9hgIBNAkbDF7ufMFG_23r46hxs5sjx9MAuQxA_UQXjkWmNFV6Bpta9YUC4UhxcGwHYRRb1_1WKH6YWxfAwiUUAc72u_ScJaF58X0GIzGzQpbIDKd_DvpImXY4fgZNwMYpEHJmBQ1mnNFa7u7V-v6UTKsW2BhOkcx1d8gH1L1EoVOfW6eYiykzndb4QhlHkRg5UtOAKx5jXDcl4zYcx2buHc5xb8QysTo09JA_bDFcSH-gcCKkwc1UBFZ5UjxOlKKdropGN2MLOiSEhhkIw_XG906riyRJgdw0x7d82YDkrxH0dd2SHMAFsNfPESFOTN0OtG1zlBCqi0w1RA2q6pM9XcgmUdQ7UckcHY0oe3lTvW9_wKlE1kfEJH6Gt4FcWoKbhoNjWjTUANuUhdQVrj5siqGn9nAHV2xQ4IYHYOi3x_4J28ngNWDvjQQ5RzUDXe-MR83phzPkpKJvxf82ItZtBzYQTo9sTQ80Bmlh55Hw_4fxoQ5wWmCp0UHtgHN-m19Ov-cbd1ZOjjnBeBI9xT_RrL0S7M2PCt0g0wHbf-mOCSrUzmV1BHGheRyPLQtXOxMlXQxGcGiZUjCEw7ANwy 
07-28 22:54:39 ERROR    FAILURE 2 downloading photos/2017/07/zelda-spooked-by-skates.mp4 - 400 Client Error: Bad Request for url: https://video-downloads.googleusercontent.com/AGQNM9JRfQxZa83qGToCMa3gn5CXNyrf4TveOxkNGVyCB7Gmq09kbPXjKB7X7NnxDLyGnZvBfjfK49TifCsPsVJapQZNIOjXm8tJWorpjxELs3Du09iNOqBKo_sDkble8PeAMwRAprAluGN6_uBiyE70zeRnZ58GCmaZe7NiareBLZXxFqHICeFKT9J1ZMc2HdEyyDcSmT8dak1ufCQXN3OYLUPnRHSs6qrFOjdHxMwfcw_BnRFV18Q85p0VE-7QAcTDdyXmFOMNTOb2jX0ATkGmYKwU49RjBKqzjQClWmK_rJlLV5TRLTTd0E_qJ1529rf0BZ15TEM1Q2d5dX4Yg-xMjTOFRDH2uf0_ffi8PC1o0HjBqjMh_yoqFdVuD95kwwomsGRHowTLN4A0JFwFLw2zyjQeanCrnEEg7_s1THwHyJ9y7NsFk-1GKUOcBNTfkVLC9zApYrEcAUnoPTv1qfUNeWwtgPTbCauk3bo27QeduALZDxprTB_4JFv4_RmKtxFI6Q12HNxDEk1LECkbsgK1pQFbjPPcXFzc13e67VPmktEMxnT-wVlREjLjSfSNlU_n22IRSZGfphUTVb1DDNQwMWYX3v_9FS90Gb0Rb91dQm52AatzWRBIKuaZGR1Dp3p7X_pIQDTypjlFrIp5nD5bHdJ8UAlzQWhro69_oSul2bsBl4gFZCct3YiYKmJmXpqWIN7UMDVCYJX8caP97P5KTRyaHvGz54O5oG4Y2ggZ-sicXphuv4EAgEZrtFhXAZRgWjTM0mPoiald2_Zyuayhe2lTRdhrraiY5PyoPq3-XkeFg81E_7qXpin1ot-CYJvrJ-3LboS7WYHcXSk3P6h9i9yr_2n4vc2vyKGlmZXT67PYB1emt9HTrG0Jjmo-7F2KITiaU5fzNGT_bNBaj8VAi0NbOPnZd4HdPipLmSXzz7bO3WVyQtnIeRHvJVNybjQO8lfuAYNJ 
07-28 22:54:39 WARNING  Downloaded 20 Items, Failed 2, Already Downloaded 26973 
07-28 22:54:39 WARNING  Indexing Shared (titled) Albums ... 
07-28 22:56:26 WARNING  Indexed 255 Shared (titled) Albums 
07-28 22:56:26 WARNING  Indexing Albums ... 
07-28 23:56:14 ERROR    
Process failed. 
Traceback (most recent call last):
  File "/home/kbingham/Sites/gphotos-sync/gphotos/Main.py", line 418, in main
    self.start(args)
  File "/home/kbingham/Sites/gphotos-sync/gphotos/Main.py", line 369, in start
    self.do_sync(args)
  File "/home/kbingham/Sites/gphotos-sync/gphotos/Main.py", line 344, in do_sync
    self.google_albums_sync.index_album_media()
  File "/home/kbingham/Sites/gphotos-sync/gphotos/GoogleAlbumsSync.py", line 150, in index_album_media
    self.index_albums_type(
  File "/home/kbingham/Sites/gphotos-sync/gphotos/GoogleAlbumsSync.py", line 203, in index_albums_type
    first_date, last_date = self.fetch_album_contents(
  File "/home/kbingham/Sites/gphotos-sync/gphotos/GoogleAlbumsSync.py", line 75, in fetch_album_contents
    response = self._api.mediaItems.search.execute(body)
  File "/home/kbingham/Sites/gphotos-sync/gphotos/restclient.py", line 100, in execute
    result.raise_for_status()
  File "/home/kbingham/.pyenv/versions/3.9.0/lib/python3.9/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://photoslibrary.googleapis.com/v1/mediaItems:search
07-28 23:56:14 WARNING  Done. 

# command ran for 1h 4m 11s
gilesknap commented 3 years ago

@qrkourier thanks this is very helpful.

I have been unable to test the backoff code in any meaningful way with my connection speed.

synman commented 3 years ago

when I got throttled on my big account, I let it sit idle for a full day then started back up with a max threads of 5. Delete your bad ids file and try again in a day or two with lower thread settings. worked for me.

gilesknap commented 2 years ago

@synman @qrkourier UPDATE: I'm now using github actions for CI and it runs multiple targets simultaneously. This is causing me to see HTTP 429.

So I now have a way of testing this!

Once I get the next release out, I'll spend some time on this issue.

In the meantime one thing that helps is to use the option --threads 1 since it slows down the number of requests.

gilesknap commented 2 years ago

Looks like I fixed it. I'd forgotten to put 429 in the list of errors to do backoff for. Simple fix.

https://github.com/gilesknap/gphotos-sync/runs/6476970287?check_suite_focus=true

@synman @qrkourier Please can you try the HEAD of main to see if it works for your usescase?

Thanks

gilesknap commented 2 years ago

OK closing as fixed.