remove download directory from progress-bar

[menkej]

Currently the full download path is shown in the progress bar line. If the download path given by "-d" is pretty long, this leads to a "flickering" for longer filenames. To reduce this I suggest removing everything given by "-d" from the path and only show the directory sub-structure+filename and omit the base dir as it is given out on INFO the line before anyways.

I'd like some opinions before implementing

[andrewawni]

Would like to give this a shot. Another suggestion I saw implemented on some shell prompts was to to use prefixes from directory names, so something like /path/to/my/directory/filename would turn into /p/t/m/d/filename

[boredazfcuk]

I have found that iCloudPD will always detect live photos as requiring download. This triggers an unnecessary download run, as all files will then be attempted to be downloaded, but obviously the log shows "file already exists"

In my docker container, I use the --only-print-filenames option to create a list of file destinations. I then use that list to compare against the files that exist on the HDD. If all the files exist already, I don't kick off a download run.

I would like the current functionality to remain as it is. That way, I can trigger downloads only if new files actually exist and therefore reduce the load I put on the Apple servers as this helps with the Apple throttling issue.

[menkej]

I would like the current functionality to remain as it is. That way, I can trigger downloads only if new files actually exist and therefore reduce the load I put on the Apple servers as this helps with the Apple throttling issue

I'm just talking about the progress bar which should not interfere with the way you use the program, or do I miss your point here?

[boredazfcuk]

I just have a vague recollection that using --no-progress-bar suppressed displaying filenames... Maybe that's not the case?

[boredazfcuk]

Here's a log snippet showing what I mean

2020-10-19 13:31:44 INFO     ***** boredazfcuk/icloudpd container for icloud_photo_downloader started *****
2020-10-19 13:31:44 INFO     ***** /usr/local/bin/sync-icloud.sh script version: 2020/10/19_01:02 *****
2020-10-19 13:31:44 INFO     Alpine Linux v3.12
2020-10-19 13:31:44 INFO     Python version: 3.8.5
2020-10-19 13:31:44 INFO     icloudpd version: 1.6.0
2020-10-19 13:31:44 INFO     pyicloud-ipd version: 0.10.1
2020-10-19 13:31:45 INFO     Interactive session: False
2020-10-19 13:31:45 INFO     Local user: user:1000
2020-10-19 13:31:45 INFO     Local group: group:1000
2020-10-19 13:31:45 INFO     Force GID: False
2020-10-19 13:31:45 INFO     LAN IP Address:
2020-10-19 13:31:45 INFO     Apple ID:
2020-10-19 13:31:45 INFO     Apple ID password: usekeyring
2020-10-19 13:31:45 INFO     Authentication Type: 2FA
2020-10-19 13:31:45 INFO     Cookie path: /config/
2020-10-19 13:31:45 INFO     Cookie expiry notification period: 7
2020-10-19 13:31:45 INFO     Download destination directory: /home/user/iCloud
2020-10-19 13:31:45 INFO     Folder structure: {:%Y}
2020-10-19 13:31:45 INFO     Directory permissions: 750
2020-10-19 13:31:45 INFO     File permissions: 640
2020-10-19 13:31:45 INFO     Syncronisation interval: 43200
2020-10-19 13:31:45 INFO     Time zone: Europe/London
2020-10-19 13:31:45 INFO     Additional command line options: --auto-delete --set-exif-datetime
2020-10-19 13:31:45 INFO     Telegram notifications enabled
2020-10-19 13:31:45 INFO     Telegram token 123
2020-10-19 13:31:45 INFO     Telegram chat id 123
2020-10-19 13:31:45 INFO     Telegram notification URL: https://api.telegram.org/123123/sendMessage
2020-10-19 13:31:45 INFO     Sending Telegram startup notification
2020-10-19 13:31:45 INFO     Telegram startup notification sent successfully
2020-10-19 13:31:45 INFO     Download notifications: Enabled
2020-10-19 13:31:45 INFO     Delete notifications: Enabled
2020-10-19 13:31:45 INFO     Group, group:1000, already created
2020-10-19 13:31:45 INFO     User, user:1000, already created
2020-10-19 13:31:45 INFO     Correct owner on config directory, if required
2020-10-19 13:31:45 INFO     Correct group on config directory, if required
2020-10-19 13:31:45 INFO     Using password stored in keyring
2020-10-19 13:31:45 INFO     Set owner, user, on iCloud directory, if required
2020-10-19 13:31:45 INFO     Set group, group, on iCloud directory, if required
2020-10-19 13:31:45 INFO     Correct owner on icloudpd temp directory, if required
2020-10-19 13:31:45 INFO     Correct group on icloudpd temp directory, if required
2020-10-19 13:31:45 INFO     Correct owner on config directory, if required
2020-10-19 13:31:45 INFO     Correct group on config directory, if required
2020-10-19 13:31:45 INFO     Correct owner on keyring directory, if required
2020-10-19 13:31:45 INFO     Correct group on keyring directory, if required
2020-10-19 13:31:45 INFO     Set 750 permissions on iCloud directories, if required
2020-10-19 13:31:45 INFO     Set 640 permissions on iCloud files, if required
2020-10-19 13:31:45 INFO     Sync user
2020-10-19 13:31:45 INFO     Check 2FA Cookie
2020-10-19 13:31:45 INFO     Cookie exists, check expiry date
2020-10-19 13:36:45 INFO     Valid two factor authentication cookie found. Days until expiration: 89
2020-10-19 13:36:45 INFO     Check download directory mounted correctly
2020-10-19 13:36:45 INFO     Check for new files using password stored in keyring...
**/home/user/iCloud/2020/IMG_2111.HEIC**
2020-10-19 13:37:15 INFO     Check successful
2020-10-19 13:37:15 INFO     Check detected 1 files requiring download. Verifying list accuracy
2020-10-19 13:37:15 INFO     Ignoring 0 files which have already been downloaded
2020-10-19 13:37:15 INFO     New files detected: 1
2020-10-19 13:37:15 INFO     Starting download of 1 new files for user:    
2020-10-19 13:37:15 INFO     Downloading new files using password stored in keyring...
2020-10-19 13:37:15 DEBUG    Authenticating...
2020-10-19 13:37:20 DEBUG    Looking up all photos and videos from album All Photos...
2020-10-19 13:37:20 INFO     Downloading 839 original photos and videos to /home/user/iCloud/ ...
2020-10-19 13:37:23 INFO     Downloading /home/user/iCloud/2020/IMG_2111.HEIC
2020-10-19 13:37:23 INFO     /home/user/iCloud/2020/IMG_2109.HEIC already exists.
...
2020-10-19 13:37:45 INFO     /home/user/iCloud/2019/IMG_7806.JPG already exists.
2020-10-19 13:37:45 INFO     /home/user/iCloud/2018/IMG_6824-68203.JPG deduplicated.
2020-10-19 13:37:45 INFO     /home/user/iCloud/2018/IMG_6029-106914.JPG deduplicated.
2020-10-19 13:37:45 INFO     /home/user/iCloud/2018/IMG_6824-68203.JPG already exists.
2020-10-19 13:37:45 INFO     /home/user/iCloud/2018/IMG_6029-106914.JPG already exists.
2020-10-19 13:37:45 INFO     All photos have been downloaded!
2020-10-19 13:37:45 INFO     Deleting any files found in 'Recently Deleted'...
2020-10-19 13:37:50 INFO     New files downloaded: 1
2020-10-19 13:37:50 INFO     Sending Telegram downloaded files notification
2020-10-19 13:37:51 INFO     Telegram downloaded files notification sent successfully
2020-10-19 13:37:51 INFO     Convert HEIC files to JPEG
2020-10-19 13:37:51 INFO     Convert HEIC to JPEG...
2020-10-19 13:37:51 INFO     Converting /home/user/iCloud/2020/IMG_2111.HEIC to /home/user/iCloud/2020/IMG_2111.JPG
File contains 1 images
Written to /home/user/iCloud/2020/IMG_2111.JPG
2020-10-19 13:37:51 INFO     Syncronisation complete for user
2020-10-19 13:37:51 INFO     Web cookie expires: 2020-12-18 @ 12:32:43
2020-10-19 13:37:51 INFO     Two factor authentication cookie expires: 2021-01-17 @ 12:32:53
2020-10-19 13:37:51 INFO     Next syncronisation at 01:37

The line I've highlighted in bold is the output of the progress bar that I tee to a log file, and stdout. If the filenames are no longer displayed as part of the progress bar, or if the prefixed directory names that andrewawni suggested are implemented, then I'll not be able to grab the full file names for my check.

[AndreyNikiforov]

@boredazfcuk your log output resembles results of --no-progress-bar option or some other mechanism of disabling progress bar. This suggestion is about progress bar output only and is not expected to affect output without progress bar. Btw progress bar output today is shortened in the middle, as progress bar is a visual progress, not an exact representation of what happened (as the case with logs).

[boredazfcuk]

I don't disable the progress bar, all I do is redirect stderr to stdout so that I can capture errors too... But that should still show the progress?

The exact command is:

su "${user}" -c '(/usr/bin/icloudpd --directory "/home/${0}/iCloud" --cookie-directory "${1}" --username "${2}" --folder-structure "${3}" --only-print-filenames 2>&1; echo $? >/tmp/icloudpd/icloud_check_exit_code) | tee /tmp/icloudpd/icloudpd_check.log' -- "${user}" "${config_dir}" "${apple_id}" "${folder_structure}" 

[menkej]

Actially I noticed that the "flickering" I'm seeing is probably not because of the long path itself but because of some longer filenames:

/tmp/flat/2020/08/02/IMG_9227.JPG /tmp/flat/2020/08/02/61804231335__BB8380FF-96E4-4C79-A78F-8886C7C88945.JPG /tmp/flat/2020/08/01/IMG_9225.HEIC

Somehow the truncate_middle does not capture it. Maybe the "96" is just to long for my terminal. I'll check that later. How was this 96 decided? I guess that change is from you, @AndreyNikiforov ?

[menkej]

The exact command is: ...

If you only print the filenames this disables the progress bar. So changing the progress bar behaviour does not affect your scripts.

[boredazfcuk]

If you only print the filenames this disables the progress bar. So changing the progress bar behaviour does not affect your scripts.

Ah OK. I didn't know that was happening when specifying that option.

[AndreyNikiforov]

Somehow the truncate_middle does not capture it. Maybe the "96" is just to long for my terminal. I'll check that later. How was this 96 decided? I guess that change is from you, @AndreyNikiforov ?

@menkej I don't think I changed anything in file name truncation logic.