search

ansible / receptor

Project Receptor is a flexible multi-service relayer with remote execution and orchestration capabilities linking controllers with executors across a mesh of nodes.
Other
161 stars 81 forks source link

fsnotify watcher: too many open files #789

Open fosterseth opened 1 year ago

fosterseth commented 1 year ago

receptor is not bubbling up this error message in some situations.

https://github.com/ansible/receptor/blob/4addde85f132cc555331041e9a6f7963519c542c/pkg/workceptor/kubernetes.go#L242

On a system where fsnotify cannot create a watcher, ReadString returns EOF, and line has this content

failed to create fsnotify watcher: too many open files

however, we are leveraging EOF to also mean "we are done reading from the pod, all is good"

I think if err is EOF and line is not empty, then we may need to write bubble up line as an error.

We should make sure line is expected to be empty in successfully cases before doing that, however.

fosterseth commented 1 year ago

setting these values (or lower) on your system/VM might help replicate the issue

sysctl fs.inotify.max_user_instances=128
sysctl fs.inotify.max_user_watches=65536