search

luis5tb / bgp-agent

BGP Agent to expose VMs through BGP on OVN environments
Apache License 2.0
20 stars 2 forks source link

BGP Agent is leaking inotify watcher fds #2

Closed mbaldessari closed 3 years ago

mbaldessari commented 3 years ago

So after leaving the agent running a few hours we had podman dying on us with the following error: Error: error creating libpod runtime: error configuring CNI network plugin: could not create new watcher too many open files

The problem is that bgp-agent used up all the inotify watchers on the system: find /proc/*/fd -lname anon_inode:inotify |wc -l

Showed that ~1020 something watchers were being used by the bgp agent container: root 163246 43.2 21.1 38132540 3425504 ? Ssl Mar16 463:00 _ /usr/bin/python3 /usr/local/bin/bgp-agent

Default numbers: [root@ctrl-2-0 ~]# cat /proc/sys/fs/inotify/max_* 16384 1024 8192

luis5tb commented 3 years ago

This is the leaking culprit: https://github.com/luis5tb/bgp-agent/blob/main/bgp_agent/platform/osp/ovn_driver.py#L110-L111