Closed tokyoqiang closed 2 years ago
Same issue here on a Synology DS918+ and Synology Photos. I am also using my own UID/GUID to upload these images.
I have been looking into this for the past 2 days now and I believe I have narrowed down the issue. It seems that Synology does not kickstart the indexing services (synofoto-bin-index-tool) and that specifically relates to docker volumes mapped to Synology volumes.. There are plenty of scripts/tools to actually tackle this problem, but I would ideally like to avoid running a separate indexing service on my NAS.
I did manage to find a clue on how the Synology indexing service can be initiated and that is via "touch" on the uploaded file itself, which worked for me and the image immediately appeared on the Synology Photos app - see Reddit post here
@tokyoqiang - can you give the following command a try? Do note that you need to run "touch" on the file itself and not on the folder:
sudo -i
touch /volume1/PATH/TO/IMAGE/FILE.HEIC
@boredazfcuk - I took a quick look at the code and it does not seem that a touch of the file is taking place for successful uploads. There are a few references to "_touch --reference="${heic_file}" "${heicfile%.HEIC}.JPG", but these are only taking place for conversions it seems. Is this something you could introduce?
Thanks for your help!
@tokyoqiang - I have setup a temporary solution via the task scheduler.
I enforce a file "touch" for files not older than a day in my specified volume. This triggers the indexing for newly added files.
Task scheduler -> Run command -> 'user defined scripts':
find /var/services/homes/USERNAME/Photos/ -mindepth 1 -type d -mtime -1 -exec /usr/syno/bin/synoindex -R photo -A {} \;
Replace USERNAME with your own. Make sure you to schedule the script to run daily or change the mtime value to something different, ie. -2. which in theory would search for files that have been added within the past two days.
@boredazfcuk - would still appreciate if you could consider my message above, because that would help our images appear instantly in our photos app.
Cheers
Hi,
When the files are downloaded, their date/time stamp is set to be the same date/time as the photo was taken. By touching the files after download, the date/time stamp is set to the current date/time. This will then cause a mismatch between the local file and the file stored in iCloud.
I've implemented a new variable synology_photos_app_fix
which should touch the file twice, to trigger the re-index, but also set the date/time stamp back to the correct time.
I've not tested it, as I don't have a Synology device. Can you give it a try and let me know?
Thanks, mate. I wasn't aware that touch would change the date. Looks like I ruined my entire library :) I am starting from scratch and will keep you up to date!
Cheers
Hi @boredazfcuk, quick update. I downloaded my entire library again (appx 200gb) with the synology_photos_app_fix
flag as active and it seems that the touch is taking place, however not reverting the date back to the original time/date. See below sample set of images taken in February:
I am using the following compose:
version: '3.3'
services:
icloudpd:
container_name: photobackup
restart: always
environment:
- user=x
- user_id=x
- group=x
- group_id=x
- apple_id=x@x.com
- authentication_type=2FA
- 'folder_structure={:%Y/%m}'
- notification_days=3
- synchronisation_interval=21600
- TZ=Europe/Amsterdam
- 'download_path=/icloud'
- notification_type=Webhook
- notification_title="iCloud Photos"
- skip_check=false
- auto_delete=true
- synology_photos_app_fix=true
network_mode: host
volumes:
- '/volume1/docker/photobackup:/config'
- '/volume1/homes/x/Photos/MobileBackup/x:/icloud'
image: boredazfcuk/icloudpd:latest
FYI- I did double check on the UID and GUID and these seem in order. The creator/owner of the file is reflected correctly.
Hi,
I've just tested it and confirmed it is does work as intended. It creates a reference file for each HEIC, changes the time on the HEIC file to current, changes the time back by using the reference file, and finally removes the reference file.
How are you viewing the files? From within the container itself? I'm just wondering if there is a mismatch between what you are seeing and the date/time stamps of the actual files... Is this the photos app which has indexed the files on the first touch but not the second?
I've set the synology_photos_app_fix=true, and in the log I can see it's playing with the newly downloaded files, like creating tmp files and such, but the Synology Photos app is not indexing anything still.
Can you force a manual index? As described here:
There's a request here, for a modification which would allow synchronisations to occur at specific times of day. If I implemented this, it would allow you to schedule one of the multimedia re-indexing runs, shortly afterwards. Keeping your photos app fairly up to date.
yeah I can do manual re-index via synology photos app.
I guess I'll just do that for now.
Hi,
I've just tested it and confirmed it is does work as intended. It creates a reference file for each HEIC, changes the time on the HEIC file to current, changes the time back by using the reference file, and finally removes the reference file.
How are you viewing the files? From within the container itself? I'm just wondering if there is a mismatch between what you are seeing and the date/time stamps of the actual files... Is this the photos app which has indexed the files on the first touch but not the second?
Hi @boredazfcuk, sorry for my delayed response. The screenshot I have shared is from DSM (DiskStation manager). That's Synology's linux gui. I have tried running stat
which also outputs create/modify date of when the file has been downloaded instead of when it was taken. The below is an image from my January folder (2022/01) however reflecting download date of April 13:
stat IMG_2663.HEIC
File: IMG_2663.HEIC
Size: 1984341 Blocks: 3880 IO Block: 4096 regular file
Device: a6h/166d Inode: 378942 Links: 1
Access: (0640/-rw-r-----) Uid: ( 1034/ myuser) Gid: ( 100/ users)
Access: 2022-04-13 10:43:21.642180650 +0200
Modify: 2022-04-13 10:43:21.642180650 +0200
Change: 2022-04-13 21:10:12.251096053 +0200
Birth: -
What I am now a bit concerned about is the timeline of images that will be shown in the Photos App given the fact that the images are only reflecting the "upload" date/time. I am running a re-index as we speak and it may take a bit more time for me to be able to share the results. Will keep you up to date.
Also, what also confuses me is the fact that when running the container, it does not attempt to download the images again although there is a mismatch in time/date and file name. The reason I am wondering is because from what I understood, the container would attempt to match existing (on disk) images by comparing date/time and name vs the same attributes on the cloud and attempt to download if no match is made.
Finally, I also verified if the fix is running and that does not seem to be the issue:
2022-04-29 09:54:28 INFO Synchronisation starting at 09:54:28
2022-04-29 09:54:28 INFO Check 2FA Cookie
2022-04-29 09:54:28 INFO Cookie exists, check expiry date
2022-04-29 09:54:28 INFO Valid two factor authentication cookie found. Days until expiration: 44
2022-04-29 09:54:28 INFO Check download directory mounted correctly
2022-04-29 09:54:28 INFO Failsafe file /icloud/.mounted exists, continuing
2022-04-29 09:54:28 INFO Starting download of new files for user: x
2022-04-29 09:54:28 INFO Downloading new files using password stored in keyring file...
2022-04-29 09:54:28 INFO iCloudPD launch command: /usr/bin/icloudpd --directory /icloud --cookie-directory /config --folder-structure {:%Y/%m} --username x@x.com --auto-delete --until-found 10 2>/tmp/icloudpd/icloudpd_download_error
2022-04-29 09:54:29 DEBUG Authenticating...
2022-04-29 09:54:32 DEBUG Looking up all photos and videos from album All Photos...
2022-04-29 09:54:33 INFO Downloading ??? original photos and videos to /icloud ...
2022-04-29 09:54:38 INFO /icloud/2022/04/DBDEAA43-4A3D-A3FB.jpg already exists.
2022-04-29 09:54:38 INFO /icloud/2022/04/725D0CB8-2BB1-8940.jpg already exists.
2022-04-29 09:54:38 INFO /icloud/2022/04/153C49B1-B4C4-A310.jpg already exists.
2022-04-29 09:54:38 INFO /icloud/2022/04/IMG_7509.HEIC already exists.
2022-04-29 09:54:38 INFO /icloud/2022/04/IMG_7508.HEIC already exists.
2022-04-29 09:54:38 INFO /icloud/2022/04/7BE848B4-7F88-B323.JPG already exists.
2022-04-29 09:54:38 INFO /icloud/2022/04/IMG_7506.HEIC already exists.
2022-04-29 09:54:38 INFO /icloud/2022/04/IMG_7505.HEIC already exists.
2022-04-29 09:54:38 INFO /icloud/2022/04/IMG_7487.HEIC already exists.
2022-04-29 09:54:38 INFO /icloud/2022/04/IMG_7486.HEIC already exists.
2022-04-29 09:54:38 INFO Found 10 consecutive previously downloaded photos. Exiting
2022-04-29 09:54:38 INFO All photos have been downloaded!
2022-04-29 09:54:38 INFO Deleting any files found in 'Recently Deleted'...
2022-04-29 09:54:39 INFO Fixing Synology Photos App import issue...
2022-04-29 09:54:39 INFO Synchronisation complete for userx
2022-04-29 09:54:39 INFO Web cookie expires: 2022-05-13 @ 15:16:12
2022-04-29 09:54:39 INFO Two factor authentication cookie expires: 2022-06-12 @ 12:16:32
2022-04-29 09:54:39 INFO Days remaining until expiration: 44
2022-04-29 09:54:39 INFO iCloud login counter = 63
2022-04-29 09:54:39 INFO Synchronisation ended at 09:54:39
2022-04-29 09:54:39 INFO Total time taken: 00:00:11
2022-04-29 09:54:39 INFO Next synchronisation at 15:54:28
Cheers, onestix
I've set the synology_photos_app_fix=true, and in the log I can see it's playing with the newly downloaded files, like creating tmp files and such, but the Synology Photos app is not indexing anything still.
Same on me. I see people start the nfs service and mount the directory into the docker container to solve this issue, if anyone wants to try. I have too many photos and have already download and manually re-indexed so just use it... :-)
I've set the synology_photos_app_fix=true, and in the log I can see it's playing with the newly downloaded files, like creating tmp files and such, but the Synology Photos app is not indexing anything still.
Same on me. I see people start the nfs service and mount the directory into the docker container to solve this issue, if anyone wants to try. I have too many photos and have already download and manually re-indexed so just use it... :-)
How do you mount it?
also, do you have a script set to re-index? Or you just click yourself
Is this fix actually needed then guys? It sounds like it does what was requested, but that doesn't actually fix the issue.
If it doesn't actually fix the problem, then I'd like to remove it.
I've now created a single_pass
variable which will exit after a single synchronisation. This should mean you can schedule synchronisations at specific times of day. You will also be able to schedule re-indexes of the photo app 15mins (or whatever) later.
It does solve my problem and pix are being indexed with the touch, but I would appreciate it if you could come back to me on my previous comment?
Your issue suggests that the "fix" is not working. The underlying application should set the file's date/time stamp to be the same as the EXIF data's DateTimeOriginal field (if it exists).
The fix changes the date/time stamp to be the current date/time, then changes it back to to what it was. It seems it has not changed it back for your files.
Can you check the EXIF data on the file and post the details?
Connect to your container's command line like this:
docker exec -it icloudpd /bin/ash
then run the identify on the HEIC file (you'll need to make sure the user/path is correct for your system:
identify -verbose /home/user/iCloud/2022/04/IMG_2663.HEIC | grep exif:Date
I think I'm onto something. I just SSH'ed into the Synology and ran this command [sudo /var/packages/SynologyPhotos/target/usr/bin/synofoto-bin-index-tool -t reindex -i /volume1/homes/xxx/Photos] and the photos app started to re-index.
so is there any way to incorporate this command into the docker? so that every time after it downloads some photo, it just run this command?
HI @boredazfcuk, thanks for your response.
I did not manage to find my images under path /home/user/iCloud/2022/04/xx
when using docker exec.
I have the following environment and volumes mapped:
environment:
- 'download_path=/icloud'
volumes:
- '/volume1/docker/photobackup:/config'
- '/volume1/homes/myuser/Photos/MobileBackup/myuser:/icloud'
Given the above, I have my pictures located here: cd /icloud/2022/01
. Having found the images under this directory, I ran identify -verbose IMG_2989.HEIC | grep exif:Date
on an image taken in January and the following is the response:
/icloud/2022/01 # identify -verbose IMG_2989.HEIC | grep exif:Date
exif:DateTime: 2022:01:14 13:19:25
exif:DateTimeDigitized: 2022:01:14 13:19:25
exif:DateTimeOriginal: 2022:01:14 13:19:25
The exif data seems to be correctly reflecting the date/time the image was taken. I have tested a few more images and they all seem fine.
Could it be that my folder structure within docker is the reason why the modification of the files is not taking place?
I did validate my UID/GID as well and the files are correctly written with my username, so I dont think this is causing the issue here:
- user=myuser
- user_id=1028
- group=users
- group_id=100
Thanks mate, onestix
@flyingwasabi - if you are looking at scheduling this, then you can do so via Syno's task scheduler, ie. run command every day at 2am. You can of course run this more often but it is an "intensive" task that may slow down your Syno while it is being executed. This would obviously leave a gap as to what you see in Syno Photos vs actually downloaded/available images on your Cloud. I am trying to avoid this option because I would need my images to show immediately in Syno photos.
Also, you could try running the "basic" index, which is less intensive/quicker. It should do the job as well.
/var/packages/SynologyPhotos/target/usr/bin/synofoto-bin-index-tool -t basic_reindex -i /var/services/homes/USERNAME/Photos
@flyingwasabi - if you are looking at scheduling this, then you can do so via Syno's task scheduler, ie. run command every day at 2am. You can of course run this more often but it is an "intensive" task that may slow down your Syno while it is being executed. This would obviously leave a gap as to what you see in Syno Photos vs actually downloaded/available images on your Cloud. I am trying to avoid this option because I would need my images to show immediately in Syno photos.
Also, you could try running the "basic" index, which is less intensive/quicker. It should do the job as well.
/var/packages/SynologyPhotos/target/usr/bin/synofoto-bin-index-tool -t basic_reindex -i /var/services/homes/USERNAME/Photos
Right I’m currently running it in the synology task scheduler every 12 hours. However i was asking because I thought if we could build this script into the docker script, that way we wont be having any ‘empty’ scans trigger by scheduler, which would be more efficient.
But it’s not a big deal anyway.
@flyingwasabi - the issue with this command is that it reindexes the entire folder and not a single image- hence resource intensive. The "touch" proposal triggers the indexing of a single image- hence my proposal above.
Can you double check if the "basic_reindex" (as opposed to "reindex") also solves your issue? This one is less resource intensive and should do the job as well.
If we are not getting anywhere with the "touch" fix, then we might as well consider a "basic_reindex" once the job is completed. I agree that I would prefer having this executed by the container as opposed to a scheduled job in task scheduler which creates a gap between uploaded images vs running of scheduled job.
Cheers, onestix
I think I'm onto something. I just SSH'ed into the Synology and ran this command [sudo /var/packages/SynologyPhotos/target/usr/bin/synofoto-bin-index-tool -t reindex -i /volume1/homes/xxx/Photos] and the photos app started to re-index.
so is there any way to incorporate this command into the docker? so that every time after it downloads some photo, it just run this command?
Not easily. The Docker container is its own little Linux OS which has no access to the rest of your Synology device, only to the volumes you mount into it. You could mount the /var/packages/SynologyPhotos directory to the same location in your container and then kicking off the command, but at a guess the database lives somewhere else, so that would need mapping in too. There may also be permissions issues as you're using sudo to invoke it (so using the hosts root account), and your Docker containers may be running as a restricted user.
The exif data seems to be correctly reflecting the date/time the image was taken. I have tested a few more images and they all seem fine.
Could it be that my folder structure within docker is the reason why the modification of the files is not taking place?
It shouldn't be the folder structure, but I'm wondering if it could be a permissions issue creating the tmp file. It seems that the first touch command is working, as it's setting the creation time to be the time the file was downloaded. It just seems that it's not changing it back. That must be something to do with the time of the temp file being incorrect, or the tmp files not being created at all so the script has no reference time for the file.
I've pushed a new version of the container which has a persist_temp_files
variable. Can you set it to True
in your container, delete a few of your photos inside the container, then restart it so it downloads the files again. This time it should leave the temp files behind. If they don't exist, then there's a problem creating them. If they do exist, then can you check the time stamps with stat
?
Thanks @boredazfcuk, appreciate your help. Will give it a try right away and keep you posted.
Is this fix actually needed then guys? It sounds like it does what was requested, but that doesn't actually fix the issue.
If it doesn't actually fix the problem, then I'd like to remove it.
I've now created a
single_pass
variable which will exit after a single synchronisation. This should mean you can schedule synchronisations at specific times of day. You will also be able to schedule re-indexes of the photo app 15mins (or whatever) later.
@boredazfcuk It doesn't fix but I think it's really a useful function for synology users. I found the reindex command for Synology (https://post.smzdm.com/p/757676/):
Usage:
synoindex [OPTIONS]
Index Options:
-h, --help
this help text
-A dirpath
add a dir
-a filepath
add a file
-D dirpath
delete a dir
-d filepath
delete a file
-N new_dirpath old_dirpath
rename a dir
-n new_filepath old_filepath
rename a file
-R [all|media|photo|music|video|thumb|dirpath]
all: reindex all dirpath that registered in each package
media: reindex dirpath that registered in MediaIndex package
photo: reindex photo dirpath
music: reindex music dirpath
video: reindex video dirpath
thumb: check converted video of each video file
dirpath: reindex this specific dirpath
-R user:{user_name}
reindex personal photo dirpath
-R share:{share_name}
reindex share dirpath
-R [type_music|type_video|type_photo]
reindex dirpath that registered with specific type in MediaIndex
Package Index Options:
-P [MediaIndex|{package_name}] {index_option}
index operation only apply on this package
-p [MediaIndex|{package_name}] {index_option}
index operation apply all packages except for this package
File Index Options:
-f {index_option}
index operation apply on file index
-U photo
update photo images
we can mount the synoindex
into the docker container and reindex after sync complete using synoindex -R /path/to/photos/
.
Hi @boredazfcuk , quick update here. The TMP files are being created:
Interesting observation here is that they are being created with owner "root", whereas my actual image files are created under my own username that I have defined as part of the compose:
- user=xxx
- user_id=1027
- group=users
- group_id=100
Given your hint that this may relate to permissions, I thought I'd change both "user" and "user_id" to that of my "docker" user- and voila, the modify date now seems to be reflecting the correct date the image was taken:
Can I just confirm with you that this is now as expected? So the 'modify' date reflects the date that the image was taken and 'create' will always reflect the date that the image was uploaded?
Also, Synology Photos is picking up these new images with the Modify date (actual date image taken), which in theory is exactly what we wanted to achieve:
So, this is looking really good and I'll attempt to download my entire library with this fix (again)! :)
Thanks mate!
Is this fix actually needed then guys? It sounds like it does what was requested, but that doesn't actually fix the issue. If it doesn't actually fix the problem, then I'd like to remove it. I've now created a
single_pass
variable which will exit after a single synchronisation. This should mean you can schedule synchronisations at specific times of day. You will also be able to schedule re-indexes of the photo app 15mins (or whatever) later.@boredazfcuk It doesn't fix but I think it's really a useful function for synology users. I found the reindex command for Synology (https://post.smzdm.com/p/757676/):
Usage: synoindex [OPTIONS] Index Options: -h, --help this help text -A dirpath add a dir -a filepath add a file -D dirpath delete a dir -d filepath delete a file -N new_dirpath old_dirpath rename a dir -n new_filepath old_filepath rename a file -R [all|media|photo|music|video|thumb|dirpath] all: reindex all dirpath that registered in each package media: reindex dirpath that registered in MediaIndex package photo: reindex photo dirpath music: reindex music dirpath video: reindex video dirpath thumb: check converted video of each video file dirpath: reindex this specific dirpath -R user:{user_name} reindex personal photo dirpath -R share:{share_name} reindex share dirpath -R [type_music|type_video|type_photo] reindex dirpath that registered with specific type in MediaIndex Package Index Options: -P [MediaIndex|{package_name}] {index_option} index operation only apply on this package -p [MediaIndex|{package_name}] {index_option} index operation apply all packages except for this package File Index Options: -f {index_option} index operation apply on file index -U photo update photo images
we can mount the
synoindex
into the docker container and reindex after sync complete usingsynoindex -R /path/to/photos/
.
You mention that it is not "fixing" your problem. Do you mind providing some more details here? You can try running below command and check if the image shows up in the Synology Photos app. Do note that you need to run "touch" on the file itself and not on the folder:
sudo -i
touch /volume1/PATH/TO/IMAGE/FILE.HEIC
Also, regarding your proposal to run synoindex - please see comments above. Synoindex forces an index of the full folder and not a single file. It is resource intensive and not in the interest of everyone to be performing a full folder index after a single image upload. Take my example, I am currently performing my uploads on a very regular basis and have approximately 200,000 pictures in the folder. A reindex here would take hours. If you do wish to do so, then run a scheduled task?
well if synoindex force the full index, it's not a good idea for me too, I didn't realize that.
I delete these photos, recreate the container with user/user_id/group/group_id set, and synology_photos_app_fix=True. I see photos downloaded and modify date is same as the picture was taken. Heres the log and files, still not trigger re-index in synology photos.
And then I tried two ways of manually touch file:
outside the docker container (works, time changed and re-index triggered)
root@DS220plus:~# sudo -i
root@DS220plus:~# id
uid=0(root) gid=0(root) groups=0(root),2(daemon),19(log)
root@DS220plus:~# touch /volume2/homes/xxx/Photos/iCloud/2022/05/IMG_7462.HEIC
root@DS220plus:~# stat /volume2/homes/xxx/Photos/iCloud/2022/05/IMG_7462.HEIC
File: /volume2/homes/xxx/Photos/iCloud/2022/05/IMG_7462.HEIC
Size: 3919298 Blocks: 7656 IO Block: 4096 regular file
Device: 69h/105d Inode: 1283879 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1026/ xxx) Gid: ( 100/ users)
Access: 2022-05-06 01:40:16.283155202 +0800
Modify: 2022-05-06 01:40:16.231155198 +0800
Change: 2022-05-06 01:40:16.231155198 +0800
Birth: -
inside the docker container (not working, time changed but re-index not triggered)
root@DS220plus:/# docker exec -it icloudpd sh
/ # id
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)
/ # touch /home/xxx/iCloud/2022/05/IMG_7461.HEIC
/ # stat /home/xxx/iCloud/2022/05/IMG_7461.HEIC
File: /home/xxx/iCloud/2022/05/IMG_7461.HEIC
Size: 3709901 Blocks: 7248 IO Block: 4096 regular file
Device: 69h/105d Inode: 1283881 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1026/ xxx) Gid: ( 100/ users)
Access: 2022-05-06 01:42:37.002163746 +0800
Modify: 2022-05-06 01:42:37.002163746 +0800
Change: 2022-05-06 01:42:37.002163746 +0800
Birth: -
werid..
I delete these photos, recreate the container with user/user_id/group/group_id set, and synology_photos_app_fix=True. I see photos downloaded and modify date is same as the picture was taken. Heres the log and files, still not trigger re-index in synology photos.
And then I tried two ways of manually touch file:
- outside the docker container (works, time changed and re-index triggered)
root@DS220plus:~# sudo -i root@DS220plus:~# id uid=0(root) gid=0(root) groups=0(root),2(daemon),19(log) root@DS220plus:~# touch /volume2/homes/xxx/Photos/iCloud/2022/05/IMG_7462.HEIC root@DS220plus:~# stat /volume2/homes/xxx/Photos/iCloud/2022/05/IMG_7462.HEIC File: /volume2/homes/xxx/Photos/iCloud/2022/05/IMG_7462.HEIC Size: 3919298 Blocks: 7656 IO Block: 4096 regular file Device: 69h/105d Inode: 1283879 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 1026/ xxx) Gid: ( 100/ users) Access: 2022-05-06 01:40:16.283155202 +0800 Modify: 2022-05-06 01:40:16.231155198 +0800 Change: 2022-05-06 01:40:16.231155198 +0800 Birth: -
- inside the docker container (not working, time changed but re-index not triggered)
root@DS220plus:/# docker exec -it icloudpd sh / # id uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video) / # touch /home/xxx/iCloud/2022/05/IMG_7461.HEIC / # stat /home/xxx/iCloud/2022/05/IMG_7461.HEIC File: /home/xxx/iCloud/2022/05/IMG_7461.HEIC Size: 3709901 Blocks: 7248 IO Block: 4096 regular file Device: 69h/105d Inode: 1283881 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 1026/ xxx) Gid: ( 100/ users) Access: 2022-05-06 01:42:37.002163746 +0800 Modify: 2022-05-06 01:42:37.002163746 +0800 Change: 2022-05-06 01:42:37.002163746 +0800 Birth: -
werid..
Interesting. Option two is irrelevant and not expected to work, because you are running the command from within the container and this is the exact issue we trying to resolve in first place (file changes on mounted docker drives are not indexed by Syno). Can you try running the container with your docker user? To find out the UID and GUID of your docker user, ssh into your Syno and type in id docker
. You may also try to run it as root (def not recommended) but for troubleshooting purposes it should help.
I don't have docker user...
root@DS220plus:~# id docker
id: ‘docker’: no such user
I tried running as root, it always stuck in 'Generating list of files in iCloud...'
@0x5e, that's really strange. Do you mind looking up your users? There must be one dedicated to docker:
There's only 4 user: admin, 2 users greate by me, and guest. admin and guest are disabled. I'm DSM 7.1 and docker installed from package center. really strange...
Hm, you mentioned that root doesn't work for you because it gets stuck on the file check. Can you try running as root while skipping the file check? Env skip_check=true
Currently using icloudpd installed by TruenNAS Scale
, Synology OS
connects truenas through smb sharing.
DSM 7.1-42661 Update 1
. 1.2.0-0263
synology_photos_app_fix = true
to trigger the indexing of newly downloaded photos immediately?basic_reindex
command can Re-Index files but keep indexed data (only clear unexisted files)
sudo /var/packages/SynologyPhotos/target/usr/bin/synofoto-bin-index-tool -t basic_reindex -i /volume1/photo/
Hi @Alano-i, I am not familiar with that setup, but I believe it should work. Give it a try and let us know.
The challenge that we have is that Synology does not index media files on mounted/external folders, hence the touch, to trigger the indexing service.
I've pushed a new version to Docker hub which now performs the touch commands as the ${user} (instead of root). Can someone see if this triggers the re-index? I'm wondering if the Synology doesn't pick up on it because it's the root account which is manipulating the files.
@boredazfcuk - works for me! Running the container as my "docker" user. Thanks!
Does it work with the container being run as root though? There may be a bunch of other problems if the main script is not launched with user id 0 as it handles user switching within the script via su
. It also has to fix up permissions/ownership, and may not be able to do that if the script doesn't have the correct permissions.
Hi, I've read the whole thread as I have the same issue. But can someone explain why the touch inside the container would work and makes the file to be indexed, while just placing the file doesn't? Above with the 'option 1 / 2' scenario, manually touching did not kick off the indexing. How does the script itself would manage this? For me it is still not working, running docker container as root.
I would have expected the download to trigger a rebuild too, but I don't have one of these devices to actually test it though, so can only go by the feedback I receive:
It does solve my problem and pix are being indexed with the touch
I took it to mean that the touch
solution in the container was giving the wanted behaviour... Triggering an index for each file that is touched, and not having to perform a full/partial re-index of all files, as that would be processor intensive, especially on large libraries.
I was thinking that it may just be ignoring operations performed on the filesystem performed by root, so that system/background operations don't trigger the re-index all the time.
Hi, I've read the whole thread as I have the same issue. But can someone explain why the touch inside the container would work and makes the file to be indexed, while just placing the file doesn't? Above with the 'option 1 / 2' scenario, manually touching did not kick off the indexing. How does the script itself would manage this? For me it is still not working, running docker container as root.
Synology does not index media files added by certain protocols. See here https://kb.synology.com/en-in/DSM/help/DSM/AdminCenter/application_mediaindexservice_general?version=6
Multimedia files added to indexed folders via the NFS or SCP protocol will not be scanned automatically. To index such files, click Re-index.
The same also applies for Docker mapped folders.
Have you tried sshing into your NAS and running touch on a single image? Also, I assume you that have added the folder in question to your indexed folders (Control Panel > Indexing Service > Media Indexing > Indexed Folder)?
Yes I get that media is not scanned. When I touch it or copy a file into the folder it gets indexed and shows up. So back to my question, how does a touch from within the container should fix this issues? Of docker volumes are not indexed anyway.
On 12 May 2022, at 22:45, boredazfcuk @.***> wrote:
I would have expected the download to trigger a rebuild too, but I don't have one of these devices to actually test it though, so can only go by the feedback I receive:
It does solve my problem and pix are being indexed with the touch
I took it to mean that the touch solution in the container was giving the wanted behaviour... Triggering an index for each file that is touched, and not having to perform a full/partial re-index of all files, as that would be processor intensive, especially on large libraries.
I was thinking that it may just be ignoring operations performed on the filesystem performed by root, so that system/background operations don't trigger the re-index all the time.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.
The index is triggered by ‘Inotify’ and is ignored for docker containers (or doesn’t work for whatever reason for docker) this is also the case for any docker solution for file syncing like nextcloud and similar solutions. So you can’t use tools like nextcloud to sync Synology folders.
On 12 May 2022, at 22:52, Ron Bruins @.***> wrote:
Yes I get that media is not scanned. When I touch it or copy a file into the folder it gets indexed and shows up. So back to my question, how does a touch from within the container should fix this issues? Of docker volumes are not indexed anyway.
On 12 May 2022, at 22:45, boredazfcuk @.***> wrote:
I would have expected the download to trigger a rebuild too, but I don't have one of these devices to actually test it though, so can only go by the feedback I receive:
It does solve my problem and pix are being indexed with the touch
I took it to mean that the touch solution in the container was giving the wanted behaviour... Triggering an index for each file that is touched, and not having to perform a full/partial re-index of all files, as that would be processor intensive, especially on large libraries.
I was thinking that it may just be ignoring operations performed on the filesystem performed by root, so that system/background operations don't trigger the re-index all the time.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.
I am using my 'docker' user to run the container and the touch does in fact trigger the indexing. Have you tried a different UID/GUID besides root?
That kind of surprised me too. I installed the docker package and I have no docker user… just runs as root (by default?)
On 12 May 2022, at 23:11, onestix @.***> wrote:
I am using my 'docker' user to run the container and the touch does in fact trigger the indexing. Have you tried a different UID/GUID besides root?
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.
Hm strange, try your regular user that you use on your Syno?
I mapped the user for icloudpd to the user for photos. However I’m on dsm 7, how about you?
When I touch the file from ssh it gets indexed immediately… not from within docker.
Maybe I should create a script that pulls exif and does the same at syno level… run it in task scheduler
On 12 May 2022, at 23:24, onestix @.***> wrote:
Hm strange, try your regular user that you use on your Syno?
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.
Running on latest DSM as well.
You could alternatively run a basic reindex on your folder. You can schedule it via task scheduler:
/var/packages/SynologyPhotos/target/usr/bin/synofoto-bin-index-tool -t basic_reindex -i /var/services/homes/USERNAME/Photos
That would save you from tampering around with the touch and changing exif data back.
There is one final solution I can think of and that is if you want to fix this once and for all, including other containers/services you may be running. For that you'd need to install the inotify-tools package from the SynoCommunity and create a little script. You'll find more details here:
https://github.com/letorbi/synoindexwatcher/issues/51#issuecomment-1061583847
This is really a great code that helps me easily download photos from Icloud. My issue is synology moments can't show photos after it completed 'rebuild index'. If I move the photos to another location then move back, it can show. I tried add User/UID/Group/Group ID predicate, it changed user from 1000 to my name, but still fail to show phto.
Can you help me to check the issue? Thanks!