Extremely large number of filesystem calls for single-file watches

Axosoft / nsfw

A super fast and scaleable file watcher that provides a consistent interface on Linux, OSX and Windows

MIT License

902 stars 112 forks source link

Extremely large number of filesystem calls for single-file watches #134

Open bendavis78 opened 3 years ago

bendavis78 commented 3 years ago

When a watching a single file, I'm seeing an extremely large number of filesystem calls. This is a big issue when it comes to network mounted storage, especially in cases of where a cloud provider might charge per transaction.

I'm hosting my app (Theia) on Azure, which uses Azure Files for persistent storage, and I'm seeing thousands of transactions per minute, even when idle:

It seems that nsfw uses polling to watch individual files. Is there an option to disable polling and use inotify instead? What is the reason that nsfw has to use polling and not inotify?

implausible commented 3 years ago

It's because tracking the delete and creation of a file can't be done without file polling. Single file watching through NSFW is deliberately not as performant as it could be due to the requirements for its interface. I would not recommend using nsfw in single file mode if you're manually setting up a ton of individual watches. Instead opt to watch the directory that will contain the files you care about, and then trigger your behavior when the file you care about shows up in the events callback.

bendavis78 commented 3 years ago

delete and creation of a file can't be done without file polling

Couldn't this be done by watching the parent directory and filtering events based on the file name? Are there use cases in which polling would be preferable to watching the parent directory?