search

Cloudbox / autoscan

Autoscan replaces the default Plex and Emby behaviour for picking up changes on the file system.
MIT License
611 stars 53 forks source link

feat: file existence checking #50

Closed talisto closed 3 years ago

talisto commented 3 years ago

I'm wondering if it's possible to implement a file check in the processor before it initiates the scan? My media files are downloaded on a separate machine than my Plex server, and due to the nature of the way the scan is triggered, it can sometimes take quite a while before the file is available to my Plex server. I realize that is what the minimum age setting is for, but it's annoying to have to set a really high minimum age just to handle the occasional slow transfer, when most files are available relatively quickly otherwise.

PAS has a retry/delay system to check for the existence of the file every x number of seconds, y number times (after an initial delay), before starting the scan, which is immensely useful. It would be great if this were available in Autoscan as well.

Thanks!!

m-rots commented 3 years ago

Originally Autoscan had this exact behaviour implemented. Files were checked for their existence before sending any requests to the targets. Each time an existence check was unsuccessful, the min-age would be applied again so the existence could be retried later (up to a configurable limit). However, this behaviour was removed in Autoscan 0.4.0 due to our Google Drive (through Bernard) integration.

Essentially, the problem we began facing was the following: Autoscan was file-based, each item in the datastore was required to have both a folder path and a file name. So when we send scan requests to targets, we first checked whether the folder path + file name existed on the file system. When successful, we grabbed the folder path and sent it off to the targets (Plex requires a folder to be scanned, not a file).

Now, here comes the problem: Plex only scans a folder when its parent folder exists. Of course this is totally fine when files are added or have changed, but it becomes problematic when entire folders are being deleted. The following case made us remove everything file-related:

  1. Original files lived in /Shows/The Boys/Season 1/episodes.mkv
  2. /Shows/The Boys is renamed to /Shows/The Boys (2019)

Now, at this point we must scan both the new path to add the correct path for the new file entries and the old path to clean up and remove the previous entries. However, we must scan the path /Shows/The Boys as its parent /Shows must still exist for Plex to do anything. Yet, the architecture we were using (the folder path + file name) caused us to forget the original event (that /Shows/The Boys is deleted) and instead only told us that a few specific files were removed (with no insight of the total number of files). Therefore, Autoscan tried to scan the path /Shows/The Boys/Season 1 to remove the items. But as you can imagine, it did not work as the parent /Shows/The Boys did no longer exist.

With this issue in mind and the only up-side of saving file paths being file existence, we removed the file-based architecture completely and instead moved on to folders exclusively. This has fixed our issue and this will continue to be the way the architecture is designed for Autoscan moving forward.