Downloads consistently stall when including videos

[dbyler]

Overview

I've had successful photos-only downloads running icloud_photos_downloader via Docker on my Mac, but every time I try to include videos, it stalls within a couple minutes.

This command works for me:

docker run -it --rm --name icloudpd -v /Volumes/MyDrive/Photos/data:/data -e TZ=America/Los_Angeles icloudpd/icloudpd:latest icloudpd --directory /data --username xxxxx@xxxxx.com --watch-with-interval 3600 --skip-videos --auto-delete

However, when I remove the --skip-videos flag, the download invariably stalls after a couple minutes.

Steps to Reproduce

1. Open Docker Desktop for Mac (I'm running Docker 4.27.2 on macOS 14.1 on a M1 machine)
2. Run the shell command docker run -it --rm --name icloudpd -v /Volumes/MyDrive/Photos/data:/data -e TZ=America/Los_Angeles icloudpd/icloudpd:latest icloudpd --directory /data --username xxxxx@xxxxx.com --watch-with-interval 3600 --auto-delete
3. Watch and wait

Expected Behavior

The full download executes successfully

Actual Behavior

icloudpd usually manages to fetch a couple videos before stalling.

You can see three separate attempts in this screenshot, with Little Snitch showing the dropouts. The downloads don't restart.

[tsetsko]

I have the same issue - seems to cannot debug it and there is no pattern other than it stops at videos but it is at random intervals.

[AndreyNikiforov]

Does the size of the video matter? I wonder if large videos are more prone to stalling... Just guessing at this point as I never experienced such behavior and cannot repro.

[tsetsko]

@AndreyNikiforov I will be able to provide more information later today

[dbyler]

FWIW I'm able to use the compiled binary successfully with videos. No idea why Docker wasn't working but that was an effective workaround for me.

[tsetsko]

@AndreyNikiforov I guess this might help: Stuck at: "2024-04-10 23:02:37 DEBUG Downloading /data/2023/11/30/IMG_0703.MOV..." I went on iCloud and checked the video is 7.2 GB (18 min video at 4k)

@dbyler what do you do with the compiled binary - how do you run that?

[dbyler]

Download the binary, make it executable (e.g. chmod +x icloudpd-1.17.3-macos-amd64), and run (e.g. ./icloudpd-1.17.3-macos-amd64)

[tripleee]

I transferred some 30,000 files recently and it had a couple of hiccups but simply restarting the app (installed on my Mac in a virtual environment using pip, using Homebrew Python managed with pyenv) allowed it to continue. It's a nuisance but at least there is a workaround.

In one instance it crashed with a traceback (nothing interesting; connection aborted. Remote end closed connection without response); in another, it just didn't do anything for several hours and I had to interrupt it with ctrl-C.

[AndreyNikiforov]

I have a theory that some token may be expiring in the middle of long transfer. Slowing down downloading, one may be able to repro it. If repro, then we can brainstorm on fixes, one of which may be resuming partially downloaded content after re-authentication.

[nlgtuankiet]

cc @AndreyNikiforov I also have this issue, download stop after 1-2 hours, have to restart the icloudpd command. Like the author, I also have videos, some > 4gb, my library is 16k files Also tried to workaround with:, but no luck yet

while true; do date; icloudpd --directory . --username example@hi.com --auth-only; sleep 300; done

edit: And my download speed is ~  24Mbps, so a 4gb would take 22 min to download

[nlgtuankiet]

Update 2

I don't think it releated to video because my last run stuck at a photo

[wattsofsnacks]

Having a similar issue, with a similarly large library to the original report (~55,000 photos/videos in my case). I'm running the precompiled binary for linux-amd64. I've noticed a few different behaviors, all of which feel like the same issue. --skip-videos does not affect.

• Freezes while downloading photos
• Freezes while downloading videos
• Freezing while checking existing content (e.g. <filename> already exists) - this one is interesting, because it occurs before actually downloading any photos or videos

I've also antidotally noticed long pauses while downloading files, sometimes up to a minute. I have a 1gig Google Fiber connection, so it feels like it may be rate limiting on the iCloud side? Haven't done any network tracing to debug further.

Like other comments, if I restart the job after it hangs, it will usually re-run and download a few more items before hanging again. I let it run overnight in a script that killed the process and restarted it every 5 minutes. It only make it through downloading 10% of my library. According to iCloud, my library is 317GB.

At this point, it takes so long to check through the already downloaded content (only about 10%), things usually hang before downloading any new content.

I can open a new issue if that would be helpful, since it doesn't appear to be related to video downloads. Also willing to help debug further and work on an improvement.

[AndreyNikiforov]

Having a similar issue, with a similarly large library to the original report (~55,000 photos/videos in my case). I'm running the precompiled binary for linux-amd64. I've noticed a few different behaviors, all of which feel like the same issue. --skip-videos does not affect.

• Freezes while downloading photos
• Freezes while downloading videos
• Freezing while checking existing content (e.g. <filename> already exists) - this one is interesting, because it occurs before actually downloading any photos or videos

I've also antidotally noticed long pauses while downloading files, sometimes up to a minute. I have a 1gig Google Fiber connection, so it feels like it may be rate limiting on the iCloud side? Haven't done any network tracing to debug further.

Like other comments, if I restart the job after it hangs, it will usually re-run and download a few more items before hanging again. I let it run overnight in a script that killed the process and restarted it every 5 minutes. It only make it through downloading 10% of my library. According to iCloud, my library is 317GB.

At this point, it takes so long to check through the already downloaded content (only about 10%), things usually hang before downloading any new content.

I can open a new issue if that would be helpful, since it doesn't appear to be related to video downloads. Also willing to help debug further and work on an improvement.

Thanks for details. I don't think opening new issue worth it, because symptoms and root cause are most likely the same.