inotify instance limit reached

[PabloLec]

Hi all,

I've just started using watchdog which works perfectly fine, except I quickly encountered the following error:

  File "/usr/local/lib/python3.9/dist-packages/watchdog/observers/inotify_c.py", line 399, in _raise_error                                                                                                                                   
    raise OSError(errno.EMFILE, "inotify instance limit reached")                                                                                                                                                                            
OSError: [Errno 24] inotify instance limit reached     

So this happens when I launch my script with only a single Observer, obviously I've reached my system limit. And by the way, I do not run any other watchdog instances, only "regular" system usage.

My current inotify instances:

INOTIFY WATCHES COUNT / PID USER / COMMAND 91 2120 pablo /usr/libexec/xdg-desktop-portal-gtk 57 2342 pablo /usr/bin/plasmashell 53 70442 pablo /usr/share/code/code /usr/share/code/resources/app/out/bootstrap-fork --type=watcherService 52 2250 pablo /usr/bin/kded5 44 45060 pablo /usr/libexec/gvfsd-trash --spawner :1.6 /org/gtk/gvfs/exec_spaw/0 44 1829 pablo /lib/systemd/systemd --user 15 89484 pablo /opt/google/chrome/chrome 15 2683 pablo /usr/bin/akonadi_maildir_resource --identifier akonadi_maildir_resource_0 13 5339 pablo /usr/lib/firefox/firefox 12 70189 pablo /usr/share/code/code --no-sandbox --unity-launch 11 63565 pablo /usr/bin/nemo 10 13548 pablo /usr/bin/qgis.bin --noversioncheck 9 12408 pablo /usr/share/discord/Discord 8 90721 pablo /usr/share/spotify/spotify 8 1850 pablo /usr/libexec/tracker-miner-fs 6 1847 pablo /usr/bin/python3 /usr/bin/powerline-daemon --foreground 5 70371 pablo /usr/share/code/code --type=renderer --disable-color-correct-rendering --no-sandbox --field 5 2294 pablo /usr/lib/x86_64-linux-gnu/libexec/kactivitymanagerd 4 70276 pablo /usr/share/code/code --type=renderer --disable-color-correct-rendering --no-sandbox --field 4 2259 pablo /usr/bin/kwin_x11 4 1860 pablo /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation 3 90760 pablo /usr/share/spotify/spotify --type=utility --utility-sub-type=network.mojom.NetworkService - 3 89522 pablo /opt/google/chrome/chrome --type=utility --utility-sub-type=network.mojom.NetworkService -- 3 70539 pablo /home/pablo/.config/Code/User/globalStorage/tabnine.tabnine-vscode/binaries/3.6.8/x86_64-un 3 70250 pablo /usr/share/code/code --type=utility --utility-sub-type=network.mojom.NetworkService --field 3 6030 pablo /usr/bin/python3 /usr/bin/terminator 3 12449 pablo /usr/share/discord/Discord --type=utility --utility-sub-type=network.mojom.NetworkService - 2 71849 pablo /home/pablo/.config/Code/User/globalStorage/tabnine.tabnine-vscode/binaries/3.6.8/x86_64-un 2 7079 pablo /usr/lib/x86_64-linux-gnu/libexec/kf5/kiod5 2 70476 pablo /home/pablo/.config/Code/User/globalStorage/tabnine.tabnine-vscode/binaries/3.6.8/x86_64-un 2 70335 pablo /usr/share/code/code --inspect-port=0 /usr/share/code/resources/app/out/bootstrap-fork --ty 2 6266 pablo /home/pablo/.config/Code/User/globalStorage/tabnine.tabnine-vscode/binaries/3.6.8/x86_64-un 2 2679 pablo /usr/bin/akonadi_ical_resource --identifier akonadi_ical_resource_0 2 2671 pablo /usr/bin/akonadi_akonotes_resource --identifier akonadi_akonotes_resource_0 2 2553 pablo /usr/libexec/evolution-source-registry 2 2388 pablo /usr/bin/gmenudbusmenuproxy 2 2017 pablo /usr/libexec/ibus-x11 --kill-daemon 2 2014 pablo /usr/libexec/ibus-extension-gtk3 2 2013 pablo /usr/libexec/ibus-ui-gtk3 2 2006 pablo /usr/bin/ibus-daemon --daemonize --xim 2 1848 pablo /usr/bin/pulseaudio --daemonize=no --log-target=journal 1 89490 pablo cat 1 89489 pablo cat 1 2841 pablo /usr/bin/ksysguardd 1 2713 pablo /usr/bin/akonadi_unifiedmailbox_agent --identifier akonadi_unifiedmailbox_agent 1 2711 pablo /usr/bin/akonadi_sendlater_agent --identifier akonadi_sendlater_agent 1 2708 pablo /usr/bin/akonadi_notes_agent --identifier akonadi_notes_agent 1 2706 pablo /usr/bin/akonadi_newmailnotifier_agent --identifier akonadi_newmailnotifier_agent 1 2705 pablo /usr/bin/akonadi_migration_agent --identifier akonadi_migration_agent 1 2694 pablo /usr/bin/akonadi_mailfilter_agent --identifier akonadi_mailfilter_agent 1 2684 pablo /usr/bin/akonadi_maildispatcher_agent --identifier akonadi_maildispatcher_agent 1 2681 pablo /usr/bin/akonadi_indexing_agent --identifier akonadi_indexing_agent 1 2677 pablo /usr/bin/akonadi_followupreminder_agent --identifier akonadi_followupreminder_agent 1 2676 pablo /usr/bin/akonadi_contacts_resource --identifier akonadi_contacts_resource_0 1 2673 pablo /usr/bin/akonadi_birthdays_resource --identifier akonadi_birthdays_resource 1 2672 pablo /usr/bin/akonadi_archivemail_agent --identifier akonadi_archivemail_agent 1 2641 pablo /usr/libexec/evolution-addressbook-factory 1 2630 pablo /usr/libexec/evolution-calendar-factory 1 2594 pablo /usr/bin/akonadi_control 1 2588 pablo /usr/lib/x86_64-linux-gnu/libexec/kf5/kscreen_backend_launcher 1 2476 pablo /usr/libexec/evolution-data-server/evolution-alarm-notify 1 2421 pablo /usr/bin/korgac 1 2379 pablo /usr/lib/x86_64-linux-gnu/libexec/DiscoverNotifier 1 2369 pablo /usr/lib/x86_64-linux-gnu/libexec/kdeconnectd 1 2356 pablo /usr/bin/kaccess 1 2347 pablo /usr/bin/xembedsniproxy 1 2311 pablo /usr/bin/ksmserver 1 2268 pablo /usr/bin/kglobalaccel5 1 2210 pablo /usr/libexec/ibus-engine-simple 1 2198 pablo /usr/lib/x86_64-linux-gnu/libexec/xdg-desktop-portal-kde 1 2171 pablo /usr/libexec/goa-identity-service 1 2152 pablo /usr/lib/x86_64-linux-gnu/libexec/kf5/klauncher --fd=9 1 2119 pablo /usr/libexec/goa-daemon 1 2092 pablo /usr/libexec/gvfs-afc-volume-monitor 1 2091 pablo /usr/libexec/xdg-desktop-portal 1 2082 pablo /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork 1 2020 pablo /usr/libexec/ibus-portal 1 2012 pablo /usr/libexec/ibus-dconf 1 1906 pablo /usr/libexec/gvfs-udisks2-volume-monitor 1 1846 pablo /usr/bin/pipewire-me

I'm running Ubuntu 21.04.

I understand that this is an OS related problem. But do you have any idea how to counter this ? And I mean counter this within Python code without admin rights. I'm just discovering inotify and I guess other softwares/tools might have something to handle this situation.

[HolgerHees]

I'm running in the same situation. I have to monitor ~200 files

/proc/sys/fs/inotify/max_user_watches = 262144 and /proc/sys/fs/inotify/max_user_instances = 128 (default on most linux systems)

now I realized that each file watch will create a new inotify instance which is way to much. Normally you should create one instance and add all watched files to this single instance.

[BoboTiG]

@HolgerHees I'm always open to improvements, if you are free to propose a PR I will be glad to review it.

[HolgerHees]

@BoboTiG I was already thinking about it, but I saw that it needs a bigger refactoring of inotify_c.py and inotify_buffer.py, because currently they can't really share a inotify instance, as long as each buffer has its own read event loop.

for now I increased /proc/sys/fs/inotify/max_user_instances to 1024.

Maybe if the pain is getting to much for me, I will look deeper into it ;-)

But everything else works perfectly for me

[tovrstra]

More info on this issue can be found in #275.