search

letorbi / synoindexwatcher

An automated media index updater for Synology DiskStations.
GNU General Public License v3.0
69 stars 7 forks source link

Recognize hard links as new files #55

Open codecivet opened 1 year ago

codecivet commented 1 year ago

I can see that synoindexwatcher is running:

$ ps aux | grep synoindexwat
root      7003  0.0  0.4  14296  2232 ?        S    Dec15   0:21 /bin/python -m synoindexwatcher --config=/usr/local/etc/synoindexwatcher.conf

And the logfile says it is waiting for updates:

2022-12-15 13:28:44,286 INFO Adding watch for path: /volume1/media/Videos
2022-12-15 13:28:52,937 INFO Waiting for media file changes...

But I can confirm that there is a new file in that hierarchy that has not been added to my media index. I'm assuming it searches recursively, right? Any other info that I can use to debug?

letorbi commented 1 year ago

Hej codecivet,

Synoindex Watcher does not automatically add missing files to the media index. Only new files, which have been moved/copied into one of the watched directories (including sub-directories), will be automatically added to the media index.

However, if you want to re-add all files in the watched folders, you could call

$ python -m synoindexwatcher --rebuild-index --config=/usr/local/etc/synoindexwatcher.conf

from the command-line once. This should re-add all existing files to the media index.

If you want more output for debuuging, you can either set the log-level in your config-file to loglevel=DEBUG or append --loglevel=DEBUG to the command line.

Cheers, Torben

codecivet commented 1 year ago

Thanks for the reply and I should have been more specific that this was a new file that wasn't being detected. I have enabled debugging and added a bunch more debugging statements. My current suspicion is that NFS appears to be triggering (hundreds of?) thousands of MODIFY and CLOSE_WRITE events for large files and those might be overwhelming synoindexwatcher. For now I've tried disabling detection of the MODIFY event, but I'm not sure what repercussions that will have.

letorbi commented 1 year ago

Thanks for the clarification. I will definitely try to look into the issue, but I am quite busy right now, so it might take some time. However, I am always happy about pull request, so feel free to dig deeper into the issue, if you need a fast solution. Sorry that I cannot offer you more right now.

codecivet commented 1 year ago

I think I figured out the issue, although I am still in the process of confirming:

I was using a program that automatically renames and moves files on the NFS share. By default, it does that by using hard links. I believe something about using hard links caused synoindexwatcher (or perhaps inotify) to miss the files getting moved to their final location, probably since it wasn't really a move operation due to using a hard link.

Fortunately, I was able to disable the hard linking feature of my app and that seems to have resolved my issue. I have now reverted all changes I made to the source to check that everything is still working properly.

I haven't done any research to see if there would be an inotify-compatible method of handling hard links, but I'll close this issue once I have done some more verification.

letorbi commented 1 year ago

Awesome, thanks for your work. I really appreciate it. Please keep the issue open. I'd like to keep it as a reminder that the handling of hard links needs to be fixed.