Inotify queue overflow in file acquisition

[blotus]

We are using fsnotify/fsnotify to watch for new files if the user specifies a glob pattern.

Unfortunately, fsnotify hardcodes the events for which we are notified (see here)

Because crowdsec always adds an inotify watch on the parent directory if the user provides a glob pattern, if this directory contains a lot of files that are written constantly to, we are at risk of overflowing the kernel event queue, even though we are not interested in the events.

fsnotify has a longstanding issue about event filtering, but there does not seem to be a consensus on how it should be handled.

I see a few options available to us:

• Allow increasing the number of goroutines to handle inotify events. This feels like a hack but shouldn´t take much time to implement and likely won´t add a lot of overhead as most events will be discarded immediately (but we will still consume CPU for "nothing")
• Drop fsnotify and do our own minimalist implementation (the main complexity will probably be the cross-platform support)
• Contribute this to fsnotify (the better option long term , but it will probably take some back-and-forth and some time)

[github-actions[bot]]

@blotus: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/crowdsec/blob/master/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.

[github-actions[bot]]

@blotus: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.

• /kind feature
• /kind enhancement
• /kind bug
• /kind packaging

Details

I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/crowdsec/blob/master/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.

[blotus]

/kind bug

[buixor]

A viable option seems to be:

• switch to https://github.com/kubernetes/utils/blob/master/inotify/inotify.go on linux, as it will allow us to filter the event types.
• keep fsnotify for other platforms, and hope we have no windows users facing the same issue :smiley: