Adding large share to watched folders breaks samba shares

[spryroX]

Describe the bug Adding a large (+-1500 items) shared folder (samba share mapped to drive) seems to break access to the samba servers shares completely. All smb shares pointing to this server will throw the error "Insufficient system resources exist to complete the requested service" when trying to access them through windows explorer. If I go to the server and restart the smbd service the shares will become accessible again until I try and add it again/refresh the mapped folders.

Adding a folder with less content works, so it almost seems like it's overloading the server trying to do too many requests at once? (I haven't yet looked at the code to see the inner workings)

Just adding the same folder via "Add-videos" works fine, so it seems to be specific to the watched folder functionality EDIT: add-videos stopped processing at around 40%, no errors and refresh bar was still looking active, but no more videos were processed for over 30 mins. Samba share remained available.

To Reproduce Steps to reproduce the behavior:

1. Settings --> Add mapped folder --> Enable watch flag
2. The icon shows up in the left hand corner, when clicking it nothing happens (it also doesn't show the refresh icon inside, it does do this for the smaller share which works)
3. Share becomes unavailable with the aforementioned error message.

Expected behavior Watched folder gets added and content is imported/refreshed

Desktop (please complete the following information):

• OS: Windows
• App Version 0.9.0

[fupdec]

Thank you for describing the bug in detail. The folder watch feature is resource intensive. It goes through all the files and folders within the selected folder and makes a list. Probably a watcher is added to every child folder. I'm not sure if I can improve the performance of this function, as its main functionality is taken from the chokidar plugin that monitors folders. I can only update it to the latest version. If that doesn't work, then all that remains is to add a hint that using large folders can stop the server from working. I've tried monitoring on a 10tb hard drive and it works pretty well. But it was connected to the computer via SATA 6Gbps, and not through the server.

[fupdec]

Also, reducing the number of folders within the monitored folder should improve performance. Verified by personal experience. Can you tell how many folders are inside (approximate number)?

[spryroX]

This was just a small test with only 22 folders :) I have my data ordered by performer / site which is used for automatically tagging/metadata.

Adding all these directories manually is going to be a pain in the butt. Would it be possible to add a layer on top of the watched functionally to serve the directories to chokidar individually? So in other words: User enters directory --> Fetch all subdirectories and add them as seperate watched folders? Maybe with some manual trigger to recheck if any subdirectories were added / removed?

[fupdec]

If I understand you correctly, then you want to add all your folders for tracking and you have a lot of folders. Do you think that adding all the folders one by one to the watched folders will solve the performance problem? If you want to be able to add several folders at a time, then it is better to open a new issue for this.

[spryroX]

If I understand you correctly, then you want to add all your folders for tracking and you have a lot of folders.

I have one big folder with a lot of subfolders for performers or sites. Ideally I'd like to add the main folder as a watched folder and have new files be automatically added to the library when changes occur (I wouldn't mind having to manually trigger updates btw).

Do you think that adding all the folders one by one to the watched folders will solve the performance problem?

I have no idea, but since you mentioned there was a performance hit for nesting I thought it could be worked around by adding the subdirectories as seperate watched folders. I have no experience wit chokidar so I have no idea if this would even make a difference.

What's interesting btw is that there's only a small burst of network activity below 10MB/S when the watching starts. Within 30 seconds the samba share becomes unavailable. CPU usage is very low as well. Is there any logging I can enable to see what it's doing? If not I'll see if I can play around with chokidar outside of the app, found some interesting results online but don't have time to check right now.

[fupdec]

I'm pretty sure adding child folders individually won't solve the performance issue. Plus you will have a lot of folder icons in the menu and this will be annoying. Perhaps the problem is that the folder data is being sent in too large a batch. Sometimes when I work on an application, I get errors about the violence of the JS, which manifests itself in the application freezing for a second. You can try debugging chokidar and see what data it sends / receives in file App.vue:267

[spryroX]

Turns out the huge amounts of images and nfo files probably aren't helping. 🤦 I've found out that chokidar can take an ignore flag to exclude certain files from being watched. Preferrably it would work the other way around, so only watch for the filetypes the app supports. I see you already have a variable for this (extensions), might be interesting to see if you could implement something like was done here

Btw, if I may ask, what do you use for development / debugging?

[spryroX]

Changed the code to the following:

const ignoredExtensions = [/(^|[\/\\])\../, '**/*.jpg', '**/*.nfo', '**/*.png']
      this.watcher = chokidar.watch(dir, {
        ignored: ignoredExtensions,
        persistent: true,
        ignoreInitial: true,
      })

After this it almost instantly showed me the files that could be added (around 1400) 👍

[spryroX]

Larger directory test also succeeded (9000 files) but took a good few minutes to complete. Will it take this long on every start of the app, or will it be quick once the video's have imported into the library?

[fupdec]

I tried many of solutions to filter files with specific extension. I spent much time for get it work. That should be easy to filter, but chokidar works with very specific filtering (or I can't done it in right way). I think problem maybe not in number of files then in number of folders. Can you try to test scanning in different number of folders? Better with complex structure with many child folders.

Btw, if I may ask, what do you use for development / debugging?

Only standart console of chrmomium browser. It opened when app starts in development mode. You can try to use vue-devtools but for me it was useless. Perhaps this will be useful for you at the initial stage, when you get to know View. DOM is clearly displayed there.

Will it take this long on every start of the app, or will it be quick once the video's have imported into the library?

It will be even longer. Chokidar will first scan all folders and files, make a filtered list, and then the application will compare the list with those files that have already been added to the database. The good thing is that file comparison is pretty fast. I have over 15,000 files in my database and the comparison is done in less than a second. I am sure that the weak point in scanning is the speed of reading from the hard disk.