mandarons / icloud-docker

Dockerized iCloud Client - make a local copy of your iCloud documents and photos, and keep it automatically up-to-date.
BSD 3-Clause "New" or "Revised" License
1.15k stars 49 forks source link

[BUG] Syncing a lot of files leads to an Error 503 with ServerErrorCode "THROTTLED" #202

Closed nicx closed 4 months ago

nicx commented 7 months ago

I am syncing a lot of files and photos. After a while I get the following error:

2024-03-18 15:42:44,280 :: DEBUG :: icloudpy.services.drive.http :: base.py :: 142 :: {'document_id': '4C1F5CA2-53A4-4B0F-9DD0-D4A74570E44E', 'item_id': 'CJC_vaYFEAAiEEwfXKJTpEsPndDUp0Vw5E4', 'owner_dsid': 1422876560, 'data_token': {'url': 'https://cvws.icloud-content.com/B/AQy8TsEsuQ9LiHBK6UCJboLUWz3qAQg12TyNGB38Qq_CUmnwlVuhIl2B/P1020780.JPG?o=AlinjVtQHibtBF8nI1aaCI3UTKiXZDLLOKF8XzEHT4HP&v=1&x=3&a=CAogvHFGzCT3cPnh3H2rVxPWSn4iIr143RtxqWlGbM9pFWISbRDslo-Q5TEY7PPqkeUxIgEAUgTUWz3qWgShIl2BaiYtemaIsuf6T88ptQamYoR3Q1MnXz06pGGVlYuTrFb3hauiCvm8RHImRz7H2YQI33GgLzx4RTlW7MCXxSdmWUWc6SMlm3067s2TQiAYD5M&e=1710776564&fl=&r=7876aff2-9255-4357-a148-7fde241c86b3-1&k=N_UvyH9TWM2LuxOGYxWu7g&ckc=com.apple.clouddocs&ckz=com.apple.CloudDocs&p=101&s=tkrd_hRjOxHR-Jlf7DsrEN8AKeM', 'token': 'CAog7jrRPCakzqia71DZfIWuJq1yFRHs+S7uP0LxS8+/AngSbRDslo+Q5TEY7PPqkeUxIgEAUgTUWz3qWgShIl2BaiYwt6ugFWvic/pULFL7msuYP98UJbgfhJlTU14HgVVBofJ/GLUbTXImQpRoM0HpW3JVK2SDWlYfEK/3SCde1WU6dBfFxozVneO2GA3NCqY=', 'signature': 'AQy8TsEsuQ9LiHBK6UCJboLUWz3q', 'wrapping_key': 'N/UvyH9TWM2LuxOGYxWu7g==', 'reference_signature': 'AQg12TyNGB38Qq/CUmnwlVuhIl2B'}, 'thumbnail_token': {'url': 'https://cvws.icloud-content.com/B/AbJoaGVEpOUMpnZQQpucNiJZltOLAQbTyKUjXGaeiDSp4tuMA3PeitN0/P1020780.jpg?o=Arf6wiG2TBxWzL3fYS7N-CT_ou4sHQXWLZEOHX-6jdjf&v=1&x=3&a=CAogdjXj0EPDWkgxbI6DpozVGuOlV38EDhz9CA-ahGud1bYSbRDrlo-Q5TEY6_PqkeUxIgEAUgRZltOLWgTeitN0aiaFHYiW9cXaq8tjmu-xUIpLtIjwgJGdT4dXBmyQytofArdk4gSx_HImegVWZtHlknwvTIUJOclcxkmhK-Vfdan7JZPK56tEsjCJvJimKdc&e=1710776564&fl=&r=7876aff2-9255-4357-a148-7fde241c86b3-1&k=T6ImFV8ZCP6iUv0WyFA20Q&ckc=com.apple.clouddocs&ckz=com.apple.CloudDocs&p=101&s=pHndd8f9aRrKgzAnZoafv223PmE', 'token': 'CAogIcTNJ3hsrp0Vk2Bbo+rWX/ZWoTMa0Muu2V6qlSq3ZRoSbRDrlo+Q5TEY6/PqkeUxIgEAUgRZltOLWgTeitN0aiYLmwLemSyGbMg+qPVnDHTS26/Zn5f98oymJHg/HdErw4C1fpSemnImOOdY6NEijszVRyYOjcvoZovHnTWj8rgrs1EQs7wOtZRBdponMOg=', 'signature': 'AbJoaGVEpOUMpnZQQpucNiJZltOL', 'wrapping_key': 'T6ImFV8ZCP6iUv0WyFA20Q==', 'reference_signature': 'AQbTyKUjXGaeiDSp4tuMA3PeitN0'}, 'double_etag': '1xtj::1xti'}
2024-03-18 15:42:44,281 :: DEBUG :: icloudpy.services.drive.http :: base.py :: 75 :: GET https://cvws.icloud-content.com/B/AQy8TsEsuQ9LiHBK6UCJboLUWz3qAQg12TyNGB38Qq_CUmnwlVuhIl2B/P1020780.JPG?o=AlinjVtQHibtBF8nI1aaCI3UTKiXZDLLOKF8XzEHT4HP&v=1&x=3&a=CAogvHFGzCT3cPnh3H2rVxPWSn4iIr143RtxqWlGbM9pFWISbRDslo-Q5TEY7PPqkeUxIgEAUgTUWz3qWgShIl2BaiYtemaIsuf6T88ptQamYoR3Q1MnXz06pGGVlYuTrFb3hauiCvm8RHImRz7H2YQI33GgLzx4RTlW7MCXxSdmWUWc6SMlm3067s2TQiAYD5M&e=1710776564&fl=&r=7876aff2-9255-4357-a148-7fde241c86b3-1&k=N_UvyH9TWM2LuxOGYxWu7g&ckc=com.apple.clouddocs&ckz=com.apple.CloudDocs&p=101&s=tkrd_hRjOxHR-Jlf7DsrEN8AKeM 
2024-03-18 15:42:44,282 :: DEBUG :: urllib3.connectionpool :: connectionpool.py :: 291 :: Resetting dropped connection: cvws.icloud-content.com
2024-03-18 15:42:44,671 :: DEBUG :: urllib3.connectionpool :: connectionpool.py :: 474 :: https://cvws.icloud-content.com:443 "GET /B/AQy8TsEsuQ9LiHBK6UCJboLUWz3qAQg12TyNGB38Qq_CUmnwlVuhIl2B/P1020780.JPG?o=AlinjVtQHibtBF8nI1aaCI3UTKiXZDLLOKF8XzEHT4HP&v=1&x=3&a=CAogvHFGzCT3cPnh3H2rVxPWSn4iIr143RtxqWlGbM9pFWISbRDslo-Q5TEY7PPqkeUxIgEAUgTUWz3qWgShIl2BaiYtemaIsuf6T88ptQamYoR3Q1MnXz06pGGVlYuTrFb3hauiCvm8RHImRz7H2YQI33GgLzx4RTlW7MCXxSdmWUWc6SMlm3067s2TQiAYD5M&e=1710776564&fl=&r=7876aff2-9255-4357-a148-7fde241c86b3-1&k=N_UvyH9TWM2LuxOGYxWu7g&ckc=com.apple.clouddocs&ckz=com.apple.CloudDocs&p=101&s=tkrd_hRjOxHR-Jlf7DsrEN8AKeM HTTP/1.1" 200 3217307
2024-03-18 15:42:44,672 :: DEBUG :: icloudpy.base :: base.py :: 94 :: Saved session data to file
2024-03-18 15:42:44,673 :: DEBUG :: icloudpy.base :: base.py :: 98 :: Cookies saved to session_data/****
2024-03-18 15:42:44,673 :: DEBUG :: root :: sync_drive.py :: 144 :: No changes detected. Skipping the file /app/icloud/drive/Medien/Fotos/Familie/Türkei 04.10.10 - 14.10.10/P1020780.JPG ...
2024-03-18 15:42:44,674 :: DEBUG :: icloudpy.services.drive.http :: base.py :: 75 :: GET https://p101-docws.icloud.com:443/ws/com.apple.CloudDocs/download/by_id 
2024-03-18 15:42:44,884 :: DEBUG :: urllib3.connectionpool :: connectionpool.py :: 474 :: https://p101-docws.icloud.com:443 "GET /ws/com.apple.CloudDocs/download/by_id?document_id=7592570E-7E83-41E5-B603-C004DFA31E9F HTTP/1.1" 503 None
2024-03-18 15:42:44,885 :: DEBUG :: icloudpy.base :: base.py :: 94 :: Saved session data to file
2024-03-18 15:42:44,885 :: DEBUG :: icloudpy.base :: base.py :: 98 :: Cookies saved to session_data/****
2024-03-18 15:42:44,885 :: DEBUG :: icloudpy.services.drive.http :: base.py :: 142 :: {'uuid': 'aac5556f-86bc-4d2b-a91c-111a28727fa9', 'error_code': 'THROTTLED', 'retryAfter': 12, 'serverErrorCode': 'THROTTLED'}
2024-03-18 15:42:44,886 :: ERROR :: icloudpy.base :: base.py :: 186 :: Service Unavailable (503)
2024-03-18 15:42:44,886 :: DEBUG :: icloudpy.services.drive.http :: base.py :: 75 :: GET https://p101-docws.icloud.com:443/ws/com.apple.CloudDocs/download/by_id 
2024-03-18 15:42:45,246 :: DEBUG :: urllib3.connectionpool :: connectionpool.py :: 474 :: https://p101-docws.icloud.com:443 "GET /ws/com.apple.CloudDocs/download/by_id?document_id=29998EA6-1CA3-4C0E-8762-06A0973E911F HTTP/1.1" 200 None
2024-03-18 15:42:45,247 :: DEBUG :: icloudpy.base :: base.py :: 94 :: Saved session data to file
2024-03-18 15:42:45,248 :: DEBUG :: icloudpy.base :: base.py :: 98 :: Cookies saved to session_data/****
2024-03-18 15:42:45,248 :: DEBUG :: icloudpy.services.drive.http :: base.py :: 142 :: {'document_id': '29998EA6-1CA3-4C0E-8762-06A0973E911F', 'item_id': 'CJC_vaYFEAAiECmZjqYco0wOh2IGoJc-kR8', 'owner_dsid': 1422876560, 'data_token': {'url': 'https://cvws.icloud-content.com/B/AS-RbT9xJmPYkEMW_qfFpyvUAXhlAQBg3EURkvboewhG3qV45fXmhTP7/P1020782.JPG?o=Aqxr6hav45AhQ64Jdm4a38mAaM9cxUTgaeYB6QdYA1YZ&v=1&x=3&a=CAog2XscEC7Bnzqg9_5b3NMMqUuQkfxWKAyhachv0CX0WzASbRCzno-Q5TEYs_vqkeUxIgEAUgTUAXhlWgTmhTP7aiYMgoR2PcZhVOUG8okfEHomwCqx8irr_OfvcQhWWktvgSgtDOkjkHImRDjGonqea_hv0rDgSoBkgu75IE3g9NlZyvZBRNR8kdRyhXLKx6o&e=1710776565&fl=&r=60be1cc5-1cfd-4250-8e5c-4acec937f630-1&k=xPC-n-cw96BtIK5_CnQMnA&ckc=com.apple.clouddocs&ckz=com.apple.CloudDocs&p=101&s=H-OItn6i-I-7vsFqRFcsoF0xYaQ', 'token': 'CAoglXmmxSFrQmQaHfuHEAe+YJ3xRsyp/yHbDZNesDbgeUUSbRC0no+Q5TEYtPvqkeUxIgEAUgTUAXhlWgTmhTP7aibC0ustBH9vclI5yxH16hx7zreN8mQ8uf4v0ltK3XjTmqTsgPN+oHImGQvgjgL8G59LMU6GIirv6wtOVE8Pl5ShebocoeW4DcUqlQjSNqg=', 'signature': 'AS+RbT9xJmPYkEMW/qfFpyvUAXhl', 'wrapping_key': 'xPC+n+cw96BtIK5/CnQMnA==', 'reference_signature': 'AQBg3EURkvboewhG3qV45fXmhTP7'}, 'thumbnail_token': {'url': 'https://cvws.icloud-content.com/B/AQeRZmg9p3H9KXWA7qpCPVlmzlyIATy2jltVzRlPP11w29zf4-eHHufc/P1020782.jpg?o=AhaIbNuZhRH8Tq4xeOAHiZrzUIK5BAytgkujbRN-QI_u&v=1&x=3&a=CAogS-LisolxAvEPMUABfBwGI_0YL3FxVKpmCzYIm18sYSASbRCzno-Q5TEYs_vqkeUxIgEAUgRmzlyIWgSHHufcaiZZagU_PNtBTYkiBnYWon4gNn4r8e532uwLd9tA4CswsPzjlr6R_XImvjchd9NN--ANENwVaANXY6r9blLe2EUx-2eTqkvNnNA6US39pmI&e=1710776565&fl=&r=60be1cc5-1cfd-4250-8e5c-4acec937f630-1&k=x-G74UJs0VRddNgUiEqWgA&ckc=com.apple.clouddocs&ckz=com.apple.CloudDocs&p=101&s=OrlRPzosWg0c2unH5rUvnZFzTsY', 'token': 'CAogqODQ6bpsctc08zUT7pgfEPc4T8NqgpBk5kAAbyJeYTMSbRCzno+Q5TEYs/vqkeUxIgEAUgRmzlyIWgSHHufcaiYlqDHZjPGX43yQoq+ESnZ/n+GfytGRgVFzF6hOdHRm6VozkZ8FEnImRDjFSc8UtTn++n2cz7C7gO9lIrBGSxDY8hW0g2c4Ku+FNU3/Cr8=', 'signature': 'AQeRZmg9p3H9KXWA7qpCPVlmzlyI', 'wrapping_key': 'x+G74UJs0VRddNgUiEqWgA==', 'reference_signature': 'ATy2jltVzRlPP11w29zf4+eHHufc'}, 'double_etag': '1xrr::1xrq'}
2024-03-18 15:42:45,248 :: DEBUG :: icloudpy.services.drive.http :: base.py :: 75 :: GET https://cvws.icloud-content.com/B/AS-RbT9xJmPYkEMW_qfFpyvUAXhlAQBg3EURkvboewhG3qV45fXmhTP7/P1020782.JPG?o=Aqxr6hav45AhQ64Jdm4a38mAaM9cxUTgaeYB6QdYA1YZ&v=1&x=3&a=CAog2XscEC7Bnzqg9_5b3NMMqUuQkfxWKAyhachv0CX0WzASbRCzno-Q5TEYs_vqkeUxIgEAUgTUAXhlWgTmhTP7aiYMgoR2PcZhVOUG8okfEHomwCqx8irr_OfvcQhWWktvgSgtDOkjkHImRDjGonqea_hv0rDgSoBkgu75IE3g9NlZyvZBRNR8kdRyhXLKx6o&e=1710776565&fl=&r=60be1cc5-1cfd-4250-8e5c-4acec937f630-1&k=xPC-n-cw96BtIK5_CnQMnA&ckc=com.apple.clouddocs&ckz=com.apple.CloudDocs&p=101&s=H-OItn6i-I-7vsFqRFcsoF0xYaQ 
2024-03-18 15:42:45,250 :: DEBUG :: urllib3.connectionpool :: connectionpool.py :: 291 :: Resetting dropped connection: cvws.icloud-content.com
2024-03-18 15:42:45,545 :: DEBUG :: urllib3.connectionpool :: connectionpool.py :: 474 :: https://cvws.icloud-content.com:443 "GET /B/AS-RbT9xJmPYkEMW_qfFpyvUAXhlAQBg3EURkvboewhG3qV45fXmhTP7/P1020782.JPG?o=Aqxr6hav45AhQ64Jdm4a38mAaM9cxUTgaeYB6QdYA1YZ&v=1&x=3&a=CAog2XscEC7Bnzqg9_5b3NMMqUuQkfxWKAyhachv0CX0WzASbRCzno-Q5TEYs_vqkeUxIgEAUgTUAXhlWgTmhTP7aiYMgoR2PcZhVOUG8okfEHomwCqx8irr_OfvcQhWWktvgSgtDOkjkHImRDjGonqea_hv0rDgSoBkgu75IE3g9NlZyvZBRNR8kdRyhXLKx6o&e=1710776565&fl=&r=60be1cc5-1cfd-4250-8e5c-4acec937f630-1&k=xPC-n-cw96BtIK5_CnQMnA&ckc=com.apple.clouddocs&ckz=com.apple.CloudDocs&p=101&s=H-OItn6i-I-7vsFqRFcsoF0xYaQ HTTP/1.1" 200 2234904
2024-03-18 15:42:45,546 :: DEBUG :: icloudpy.base :: base.py :: 94 :: Saved session data to file
2024-03-18 15:42:45,547 :: DEBUG :: icloudpy.base :: base.py :: 98 :: Cookies saved to session_data/****
2024-03-18 15:42:45,547 :: DEBUG :: root :: sync_drive.py :: 144 :: No changes detected. Skipping the file /app/icloud/drive/Medien/Fotos/Familie/Türkei 04.10.10 - 14.10.10/P1020782.JPG ...
2024-03-18 15:42:45,547 :: DEBUG :: icloudpy.services.drive.http :: base.py :: 75 :: GET https://p101-docws.icloud.com:443/ws/com.apple.CloudDocs/download/by_id 
2024-03-18 15:42:45,922 :: DEBUG :: urllib3.connectionpool :: connectionpool.py :: 474 :: https://p101-docws.icloud.com:443 "GET /ws/com.apple.CloudDocs/download/by_id?document_id=EF8FD012-53A4-48A2-B32F-3BF7B8AE6DFC HTTP/1.1" 503 None
2024-03-18 15:42:45,923 :: DEBUG :: icloudpy.base :: base.py :: 94 :: Saved session data to file
2024-03-18 15:42:45,924 :: DEBUG :: icloudpy.base :: base.py :: 98 :: Cookies saved to session_data/****
2024-03-18 15:42:45,924 :: DEBUG :: icloudpy.services.drive.http :: base.py :: 142 :: {'uuid': 'fe7d0961-8d1c-4a41-9c37-c5ba547b67af', 'error_code': 'THROTTLED', 'retryAfter': 9, 'serverErrorCode': 'THROTTLED'}
2024-03-18 15:42:45,924 :: ERROR :: icloudpy.base :: base.py :: 186 :: Service Unavailable (503)

My config looks like this:

app:
  logger:
    # level - debug, info, warning (default) or error
    level: "warning"
    # log filename icloud.log (default)
    filename: "icloud.log"
  credentials:
    # iCloud drive username
    username: "timo.klinger@netbag.de"
    # Retry login interval - default is 10 minutes (600)
    retry_login_interval: -1
  # Drive destination
  root: "icloud"
  smtp:
    # If you want to recieve email notifications about expired/missing 2FA credentials then uncomment
     email: "****"
     #password:
     # Uncomment this if your SMTP username is different than your sender address (for services like AWS SES)
     # username: ""
     # default to is same as email above
     to: "****.de"
     host: mail.****.de
     port: 25
    # If your email provider doesn't handle TLS
     no_tls: true
  region: global # valid values are - global (default - uses .com) or china (uses .com.cn)
drive:
  destination: "drive"
  remove_obsolete: true
  sync_interval: 1800
  #filters:
    # File filters to be included in syncing iCloud drive content
    #folders: 
    #  - "Documents"
    #  - "Software"
    #file_extensions:
      # File extensions to be included
      #- "pdf"
      #- "png"
      #- "jpg"
      #- "jpeg"
photos:
  destination: "photos"
  remove_obsolete: true
  sync_interval: 1800
  filters:
    albums:
      #- "album1"
      #- "album2"
    file_sizes:
      - "original"
      # - "medium"
      # - "thumb"

Is this a problem? Anything I could do to avoid this error?

mandarons commented 7 months ago

Your requests are being throttled by the server. Try setting sync_interval to 86400 (once a day).

nicx commented 4 months ago

with sync interval set to 86400 the error is very rare. so this works, thanks (even if I would like to sync more often :))