search

LibrePhotos / librephotos

A self-hosted open source photo management service. This is the repository of the backend.
MIT License
7.01k stars 309 forks source link

Directory watch: Monitor /data for updated/deleted/created paths #380

Open cardboarddude opened 3 years ago

cardboarddude commented 3 years ago

What Utilize inotify to watch for file and folder changes within the /data directory and update api_photo.image_path accordingly.

Why On large libraries, re-scans can take a lot of resources and time. By monitoring file and folder changes within /data, re-scans are not necessary for simple image_path updates. This can also be expanded to automatically scan new images added. This feature should be disabled by default.

More I have a library of 85,000 photos which takes over 24hours to scan. When I upload new photos from my phone and want to see them in "recently added," I need to rescan all 85,000 files just to view a couple new images.

I'd like your feedback.

guystreeter commented 3 years ago

inotify requires a file descriptor held open for each file and directory being watched. You can hit per-process limits easily on large collections. The process and kernel resources required to hold open file descriptors can add up quickly. The inotify implementation has some timing-related gaps that can cause files and dirs to be ignored if they are created quickly (like a recursive copy command might do).

cardboarddude commented 3 years ago

@guystreeter Any ideas on how performance could be improved for larger collections

cardboarddude commented 3 years ago

I was thinking, what if the user could specify a specific folder or set of folders within the /data directory to do a quick scan (no hashing of images)?

pivis commented 3 years ago

If we record size/modified time for each file considered during the full scan then we can skip files that were not touched later during the rescans. Still need to stat each of 85,000 files or whatever the library size is, but don't have to read the data/compute the hashes. This must take seconds.

ippocratis commented 2 years ago

For now you can change the scan directory to the current month or year It will make the scan faster Not ideal and not answering to the idea behind this issue which is great . Librephotos needs a way to autoscan for new files only

atlas-shrugged08 commented 1 year ago

+1 to this. Its a major hassle when you have 80k photos and want librephotos to scan new ones. Is there a workaround as of now? i.e. if i change the data directory to scan new folder/pics, will the photos from the previous scan go away?