The current mechanism we're using to monitor for file system changes requires us monitor each and every directory in the tree, causing us to open a large number of file handles if users add large directories. This change mitigates the impact of that by introducing a shared monitoring mechanism to ensure that we never watch a directory more than once, even if there are multiple observers for that directory.
The unfortunate side effect of this approach is that we're no longer able to be discrimitate when notifying observers about changes specific to them; we have to inform them of all changes, even if they're outside of their directory tree. That's because if there's ever a symlink in the path (which there often is), we loose the ability to determine if a changed URL is below a given root.
Thankfully the performance trade-offs seem to be OK for the time being, but it's definitely something we should keep an eye on. If these do become an issue, one option would be to rate limit change notifications and, longer-term, hopefully we can find a lower-level API that can monitor a directory structure recursively without the same resource demands.
The current mechanism we're using to monitor for file system changes requires us monitor each and every directory in the tree, causing us to open a large number of file handles if users add large directories. This change mitigates the impact of that by introducing a shared monitoring mechanism to ensure that we never watch a directory more than once, even if there are multiple observers for that directory.
The unfortunate side effect of this approach is that we're no longer able to be discrimitate when notifying observers about changes specific to them; we have to inform them of all changes, even if they're outside of their directory tree. That's because if there's ever a symlink in the path (which there often is), we loose the ability to determine if a changed URL is below a given root.
Thankfully the performance trade-offs seem to be OK for the time being, but it's definitely something we should keep an eye on. If these do become an issue, one option would be to rate limit change notifications and, longer-term, hopefully we can find a lower-level API that can monitor a directory structure recursively without the same resource demands.