search

mandarons / icloud-docker

Dockerized iCloud Client - make a local copy of your iCloud documents and photos, and keep it automatically up-to-date.
BSD 3-Clause "New" or "Revised" License
1.15k stars 49 forks source link

[BUG] Files are downloaded individually #250

Closed Christopher-Tan closed 2 months ago

Christopher-Tan commented 3 months ago

Describe the bug I'm attempting to sync the drive for the first time. Because of the large number of files (over 250,000) from conda packages, etc. the process is 1) extremely slow at a rate of 1 file/second even for files that are a few bytes when it's working and 2) very prone to crashing: once it starts stalling and gets stuck on a file without making any progress for hours (and you restart the container), the future syncs crash after hours of waiting with a 503 or 504 error; I'm now no longer making any progress for the past two entire days. I've restarted 3-4 times now, and every time my logs look like this:

2024/07/29 04:26:01 | stdout | 2024-07-29 09:26:01,498 :: ERROR :: icloudpy.base :: base.py :: 186 :: Service Unavailable (503)
-- | -- | --
2024/07/29 04:15:00 | stdout | crond[134]: USER root pid 198 cmd run-parts /etc/periodic/15min
2024/07/29 04:00:00 | stdout | crond[134]: USER root pid 197 cmd run-parts /etc/periodic/hourly
2024/07/29 04:00:00 | stdout | crond[134]: USER root pid 196 cmd run-parts /etc/periodic/15min
2024/07/29 03:45:00 | stdout | crond[134]: USER root pid 195 cmd run-parts /etc/periodic/15min
2024/07/29 03:30:00 | stdout | crond[134]: USER root pid 194 cmd run-parts /etc/periodic/15min
2024/07/29 03:15:00 | stdout | crond[134]: USER root pid 193 cmd run-parts /etc/periodic/15min
2024/07/29 03:00:00 | stdout | crond[134]: USER root pid 192 cmd run-parts /etc/periodic/hourly
2024/07/29 03:00:00 | stdout | crond[134]: USER root pid 191 cmd run-parts /etc/periodic/15min
2024/07/29 02:45:00 | stdout | crond[134]: USER root pid 190 cmd run-parts /etc/periodic/15min
2024/07/29 02:30:00 | stdout | crond[134]: USER root pid 189 cmd run-parts /etc/periodic/15min
2024/07/29 02:15:00 | stdout | crond[134]: USER root pid 188 cmd run-parts /etc/periodic/15min
2024/07/29 02:00:00 | stdout | crond[134]: USER root pid 187 cmd run-parts /etc/periodic/hourly
2024/07/29 02:00:00 | stdout | crond[134]: USER root pid 186 cmd run-parts /etc/periodic/15min
2024/07/29 01:45:00 | stdout | crond[134]: USER root pid 185 cmd run-parts /etc/periodic/15min
2024/07/29 01:30:00 | stdout | crond[134]: USER root pid 184 cmd run-parts /etc/periodic/15min
2024/07/29 01:15:00 | stdout | crond[134]: USER root pid 183 cmd run-parts /etc/periodic/15min
2024/07/29 01:00:00 | stdout | crond[134]: USER root pid 182 cmd run-parts /etc/periodic/hourly
2024/07/29 01:00:00 | stdout | crond[134]: USER root pid 181 cmd run-parts /etc/periodic/15min
2024/07/29 00:45:00 | stdout | crond[134]: USER root pid 180 cmd run-parts /etc/periodic/15min
2024/07/29 00:30:00 | stdout | crond[134]: USER root pid 179 cmd run-parts /etc/periodic/15min
2024/07/29 00:15:00 | stdout | crond[134]: USER root pid 178 cmd run-parts /etc/periodic/15min
2024/07/29 00:00:00 | stdout | crond[134]: USER root pid 177 cmd run-parts /etc/periodic/hourly
2024/07/29 00:00:00 | stdout | crond[134]: USER root pid 176 cmd run-parts /etc/periodic/15min
2024/07/28 23:45:00 | stdout | crond[134]: USER root pid 175 cmd run-parts /etc/periodic/15min
2024/07/28 23:30:00 | stdout | crond[134]: USER root pid 174 cmd run-parts /etc/periodic/15min
2024/07/28 23:15:00 | stdout | crond[134]: USER root pid 173 cmd run-parts /etc/periodic/15min
2024/07/28 23:00:00 | stdout | crond[134]: USER root pid 172 cmd run-parts /etc/periodic/hourly
2024/07/28 23:00:00 | stdout | crond[134]: USER root pid 171 cmd run-parts /etc/periodic/15min
2024/07/28 22:45:00 | stdout | crond[134]: USER root pid 170 cmd run-parts /etc/periodic/15min
2024/07/28 22:30:00 | stdout | crond[134]: USER root pid 169 cmd run-parts /etc/periodic/15min
2024/07/28 22:15:00 | stdout | crond[134]: USER root pid 168 cmd run-parts /etc/periodic/15min
2024/07/28 22:00:00 | stdout | crond[134]: USER root pid 167 cmd run-parts /etc/periodic/hourly
2024/07/28 22:00:00 | stdout | crond[134]: USER root pid 166 cmd run-parts /etc/periodic/15min
2024/07/28 21:45:00 | stdout | crond[134]: USER root pid 165 cmd run-parts /etc/periodic/15min
2024/07/28 21:39:18 | stdout | 2024-07-29 02:39:18,405 :: INFO :: root :: sync.py :: 82 :: Syncing drive...
2024/07/28 21:39:16 | stdout | crond[134]: crond (busybox 1.36.1) started, log level 5
2024/07/28 21:39:16 | stdout |  
2024/07/28 21:39:16 | stdout | ====================================================
2024/07/28 21:39:16 | stdout | User GID:    911
2024/07/28 21:39:16 | stdout | User UID:    911
2024/07/28 21:39:16 | stdout |  
2024/07/28 21:39:16 | stdout | https://www.buymeacoffee.com/mandarons
2024/07/28 21:39:16 | stdout | https://github.com/sponsors/mandarons
2024/07/28 21:39:16 | stdout | To support this project, please consider sponsoring.
2024/07/28 21:39:16 | stdout | ====================================================

To Reproduce Steps to reproduce the behavior:

  1. Sync an iCloud account with many files

Expected behavior

  1. Instead of traversing folders when they don't exist, you should just download the entire folder at the same time

Screenshots N/A

Configuration

app:
    logger:
        level: "info"
        filename: "/config/icloud.log"
    credentials:
        username: removed
        retry_login_interval: 600
    root: "/icloud"
    region: global
drive:
    destination: "drive"
    remove_obsolete: false
    sync_interval: 86400
photos:
    destination: "photos"
    remove_obsolete: false
    sync_interval: 300
    all_albums: true

Additional context On other containers (for family members) that do work, the sync restarts two times after the first and then doesn't do anything after that (the sync time is a few minutes). I have to restart the container manually to restart syncing. Is this expected behaviour? It will just sit idly for days.

mandarons commented 2 months ago

This is likely due to large number of small files. When you have so many small files to download, it is very likely that the app will hit icloud server's rate limit (I don't know how much is it). Rate limiting is typically done for number of requests and not the amount of data transferred.

I am afraid, there isn't any solution for this as this app isn't an official client. You likely need to exclude the conda folder from this app and download it manually from icloud.com.