search

boredazfcuk / docker-icloudpd

An Alpine Linux container for the iCloud Photos Downloader command line utility
1.85k stars 167 forks source link

Photos App Fix Synology not working #477

Closed wanglewis closed 10 months ago

wanglewis commented 10 months ago

I just met this situation, that the Synology Photos App not indexing the downloaded photos, even when I set the env. to true and the Debug shows the Information correct.

2024/01/07 00:17:06 | stdout | 2024-01-07 00:17:06 INFO     Next synchronisation at 00:16:54 2024/01/07 00:17:06 | stdout | 2024-01-07 00:17:06 INFO     Total time taken: 00:00:12 2024/01/07 00:17:06 | stdout | 2024-01-07 00:17:06 INFO     Synchronisation ended at 00:17:06 2024/01/07 00:17:06 | stdout | 2024-01-07 00:17:06 INFO     Days remaining until expiration: 87 2024/01/07 00:17:06 | stdout | 2024-01-07 00:17:06 INFO     Multifactor authentication cookie expires: 2024-04-03 @ 23:01:42 2024/01/07 00:17:06 | stdout | 2024-01-07 00:17:06 INFO     Web cookie expires: 2024-03-04 @ 23:01:33 2024/01/07 00:17:06 | stdout | 2024-01-07 00:17:06 INFO     Synchronisation complete for wanglewis 2024/01/07 00:17:02 | stdout | 2024-01-07 00:17:02 INFO     Fixing Synology Photos App import issue... 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 INFO     All photos have been downloaded 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 INFO     Found 20 consecutive previously downloaded photos. Exiting 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2023/2023_12/IMG_1943.PNG already exists 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2023/2023_12/IMG_1944.PNG already exists 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2023/2023_12/IMG_1945.PNG already exists 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2023/2023_12/IMG_1946.PNG already exists 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2023/2023_12/IMG_1947.PNG already exists 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2023/2023_12/IMG_1948.JPG already exists 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2023/2023_12/IMG_1950.PNG already exists 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2023/2023_12/IMG_1951.JPG already exists 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2023/2023_12/IMG_1953.JPG already exists 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2023/2023_12/IMG_1954_HEVC.MOV already exists 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2023/2023_12/IMG_1954.HEIC already exists 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2023/2023_12/IMG_1955_HEVC.MOV already exists 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2023/2023_12/IMG_1955.HEIC already exists 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2023/2023_12/IMG_1956_HEVC.MOV already exists 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2023/2023_12/IMG_1956.HEIC already exists 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    Skipping IMG_1957.MOV, only downloading photos.(Item type was: movie) 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    Skipping dji_fly_20231231_204048_0018_1704059938588_video_trim.mov, only downloading photos.(Item type was: movie) 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    Skipping dji_fly_20240101_000500_0019_1704067070570_video_trim.mov, only downloading photos.(Item type was: movie) 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    Skipping dji_fly_20240101_000500_0019_1704067472843_video_trim.mov, only downloading photos.(Item type was: movie) 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    Skipping dji_fly_20240101_000500_0019_1704067563324_video_trim.mov, only downloading photos.(Item type was: movie) 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    Skipping dji_fly_20240101_000500_0019_1704067799940_video_trim.mov, only downloading photos.(Item type was: movie) 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    Skipping dji_fly_20240101_000530_0020_1704067863344_video_trim.mov, only downloading photos.(Item type was: movie) 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    Skipping dji_fly_20240101_000902_0022_1704067958564_video_trim.mov, only downloading photos.(Item type was: movie) 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    Skipping dji_fly_20240101_000902_0022_1704068067488_video_trim.mov, only downloading photos.(Item type was: movie) 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    Skipping dji_fly_20240101_001312_0023_1704068216753_video_trim.mov, only downloading photos.(Item type was: movie) 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    Skipping dji_fly_20240101_001714_0025_1704068593049_video_trim.mov, only downloading photos.(Item type was: movie) 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    Skipping dji_fly_20240101_001824_0026_1704068690224_video_trim.mov, only downloading photos.(Item type was: movie) 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    Skipping dji_fly_20240101_002602_0027_1704068868232_video_trim.mov, only downloading photos.(Item type was: movie) 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    Skipping 1704072340898.mov, only downloading photos.(Item type was: movie) 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2024/2024_01/IMG_1960.JPG already exists 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2024/2024_01/IMG_1961.JPG already exists 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2024/2024_01/IMG_1962.JPG already exists 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2024/2024_01/IMG_1963.JPG already exists 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2024/2024_01/IMG_1965.JPG already exists 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2024/2024_01/IMG_1966_HEVC.MOV already exists 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2024/2024_01/IMG_1966.HEIC already exists 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2024/2024_01/IMG_1967_HEVC.MOV already exists 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2024/2024_01/IMG_1967.HEIC already exists 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2024/2024_01/IMG_1968_HEVC.MOV already exists 2024/01/07 00:17:01 | stdout | 2024-01-07 00:17:01 DEBUG    /Foto/2024/2024_01/IMG_1968.HEIC already exists 2024/01/07 00:16:58 | stdout | 2024-01-07 00:16:58 INFO     Downloading ??? original photos to /Foto ... 2024/01/07 00:16:58 | stdout | 2024-01-07 00:16:58 DEBUG    Looking up all photos from album All Photos... 2024/01/07 00:16:55 | stdout | 2024-01-07 00:16:55 DEBUG    Authenticating... 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Failsafe file /Foto/.mounted exists, continuing 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Check download directory mounted correctly... 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Keyring file exists, continuing 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Synchronisation starting at 00:16:54 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Sync user: wanglewis 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Keyring file exists, continuing 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Directory is writable: /config/python_keyring/ 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Failsafe file /Foto/.mounted exists, continuing 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Check download directory mounted correctly... 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Directory is writable: /config/pythonkeyring/ 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Ignore Synology extended attribute directories: Enabled 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Authentication domain: com 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Downloading from: icloud.com 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Convert HEIC to JPEG: false 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Skip videos: true 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Live photo size: original 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Stop downloading when prexisiting files count is: 20 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Downloading photos from: Download All Photos 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Number of most recently added photos to download: Download All Photos 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Skip live photos: false 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Skip download check: true 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Single pass mode: false 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Photo size: original 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Delete after download: false 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Auto delete: false 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Set EXIF date/time: false 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Synchronisation delay (minutes): 0 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Synchronisation interval: 86400 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Folder structure: {:%Y/%Y%m} 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Download destination directory: /Foto 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Cookie expiry notification period: 7 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Cookie path: /config/lewis861217googlemailcom 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Authentication Type: MFA 2024/01/07 00:16:54 | stdout | 2024-01-07 00:16:54 INFO     Apple ID: lewis861217@googlemail.com 2024/01/07 00:16:53 | stdout | hostname: icloud_foto_wy: Host not found 2024/01/07 00:16:53 | stdout | 2024-01-07 00:16:53 INFO     Loading configuration from: /config/icloudpd.conf 2024/01/07 00:16:53 | stdout | 2024-01-07 00:16:53 INFO     Python version: 3.11.6 2024/01/07 00:16:53 | stdout | 2024-01-07 00:16:53 INFO     Alpine Linux 3.19.0 2024/01/07 00:16:53 | stdout | 2024-01-07 00:16:53 INFO      For support, please go here: https://github.com/boredazfcuk/docker-icloudpd  2024/01/07 00:16:53 | stdout | 2024-01-07 00:16:53 INFO      boredazfcuk/icloudpd container for icloud_photo_downloader v1.0.705 started 

wanglewis commented 10 months ago

Im using the same user name, id, group and gid as the Synology user

boredazfcuk commented 10 months ago

Please see this for further information: https://github.com/boredazfcuk/docker-icloudpd/issues/281

The Synology fix was working for a single user, but nobody identified the reason why. I have not removed the feature because of this. It seems everyone else worked around the issue by scheduling a re-index.

wanglewis commented 10 months ago

Thanks for the quick answer. The script worked

lonevvolf commented 9 months ago

@boredazfcuk I'm currently looking into this on my end - is there an interest to integrate a real fix for Synology if we can find out why it doesn't work for everyone?

lonevvolf commented 9 months ago

After some research, I've found the following: https://forum.syncthing.net/t/share-my-solution-to-get-synology-photos-auto-index-working-with-syncthing/21410

Basically, since Docker works directly with the filesystem, these events are not monitored by the Synology indexer service. If we could change the Docker image to use SMB/CIFS as described here: https://docs.docker.com/storage/volumes/#create-cifssamba-volumes the events would trigger indexing. Not yet sure how much is involved in making that happen, or if it would have a significant performance impact.

boredazfcuk commented 9 months ago

Previously there have been issues raised because the container could not read the files in the download path. The root cause for this was due to people mounting a directory on the host device to a CIFS/SMB share, then mounting this location as a volume inside the Docker container and pointing the container's download directory to that location. Changing the image to point at a CIFS/SMB share would likely result in the same issue.

I'm not aware of anyone taking the approach that you have suggested here: https://docs.docker.com/storage/volumes/#create-cifssamba-volumes. It may not suffer from the same issue previously reported, as Docker is not creating a bind mount over the top of a CIFS/SMB mounted share. However, this approach would not likely need any changes to the container. Please report back any findings you get. It may just be that we expand the CONFIGURATION.MD file with an installation guide for Synology and presumably other NAS devices.

lonevvolf commented 9 months ago

I've started testing this solution, and initial testing looks promising. It actually works, and the Synology refreshes as expected. Funny enough, though, the container throws an error on new files: IOError while writing file to /home/user/iCloud/PrimarySync/2024/01/27/IMG_6973.HEIC. You might have run out of disk space, or the file might be too large for your OS. Skipping this file...

It does actually create the folder and files, though. Not sure if @boredazfcuk has any insight here.
In order to get the volumes working, I had to copy the existing container into a new one with the volume mounted:

1) Remove the mapping from the original container for the data directory (but leave the config) 2) Run the following command to create the volume:

sudo docker volume create \
    --driver local \
    --opt type=cifs \
    --opt device=//Synology/photo \
    --opt o=addr=Synology,username=<yourusername>,password='<yourpassword>',file_mode=0777,dir_mode=0777,vers=3.0 \
    --name cif-volume

3) Create the new container as a copy of the old:

sudo docker run -d --name icloudpd-exp \
--volumes-from icloudpd \
-v cif-volume:/home/user/iCloud \
boredazfcuk/icloudpd

where "icloudpd" is the old container name, and "icloudpd-exp" is the new container name.

boredazfcuk commented 9 months ago

This is a result of writing to CIFS share.

I suggested a potential fix here, but never received a reply: https://github.com/boredazfcuk/docker-icloudpd/issues/436