InotifyEvents seem to not be closed / garbage collected

[yacn]

Platform we're running on : Ubuntu 18.04, Python 3.6.8

Context: We have a long running service that uses watchdog to watch a log file and parse it when it's moved.

Anyways, we've been tracing a memory leak in it with tracemalloc and it appears like watchdog is the culprit, it's the top memory consumer compared to the snapshot taken when starting the app:

/opt/app/env/lib/python3.6/site-packages/watchdog/observers/inotify_c.py:0: size=2470 KiB (+2388 KiB), count=47172 (+47157), average=54 B

some more showing it's one of the top mem incrementors:

/opt/app/env/lib/python3.6/site-packages/watchdog/observers/inotify_c.py:243: size=1280 KiB (+704 KiB), count=2 (+0), average=640 KiB
/opt/app/env/lib/python3.6/site-packages/watchdog/observers/inotify_c.py:462: size=1634 KiB (+184 KiB), count=33095 (+3379), average=51 B"
/opt/app/env/lib/python3.6/site-packages/watchdog/utils/unicode_paths.py:59: size=151 KiB (+151 KiB), count=2016 (+2016), average=77 B
/opt/app/env/lib/python3.6/site-packages/watchdog/events.py:118: size=114 KiB (+112 KiB), count=2558 (+2531), average=46 B

When looking at the tracebacks for this, these are the top memory consuming lines:

81837 memory blocks: 3994.4 KiB
  File "/opt/app/env/lib/python3.6/site-packages/watchdog/observers/inotify_c.py", line 462
    self._wd = wd
56312 memory blocks: 3342.1 KiB
  File "/opt/app/env/lib/python3.6/posixpath.py", line 90
    path += b
56313 memory blocks: 3080.0 KiB
  File "/opt/app/env/lib/python3.6/site-packages/watchdog/observers/inotify_c.py", line 323
    inotify_event = InotifyEvent(wd, mask, cookie, name, src_path)
217 memory blocks: 2573.5 KiB
  File "/usr/lib/python3.6/ssl.py", line 651
    return self._sslobj.peer_certificate(binary_form)
2 memory blocks: 2560.4 KiB
  File "/opt/app/env/lib/python3.6/site-packages/watchdog/observers/inotify_c.py", line 243
    self._moved_from_events[event.cookie] = event

So it seems to me that the issue is the InotifyEvent objects that get created are never closed / garbage collected until the Observer object stops / joins.

I've tried issuing a del event after we are finished using it but this has no perceptible effect, something else is holding on to a reference to the event.

How could we "forcibly" close or delete the InotifyEvent or FileSystemEvent when we're done with it?

[ghost]

Can confirm. In the project i'm currently working on we use watchdog as well. We also used the tracemalloc package to look for the top memory consumers, and using the InotifyObserver of watchdog, it was leading the charts with increasing memory usage over time. The scripts were running on a Ubuntu 19.10 machine, Python 3.7.4. The only workaround for us was to explictly use the PollingObserver instead of the Observer class.

[BoboTiG]

Definitely, something may need to be improved. Do you have a clue or even a patch almost ready for that? :crossed_fingers: :)

[Gandouge78]

Hey I have the problem on my embedded system. Each Observer I created on a folder with a lot of activity will increase his memory and provoke my system to kill process.

It's a big issue, does anyone has a solution ?

Polling is not an option for us

[RoySRC]

Does this problem still persists? Or has it been fixed in one of the future releases.

[ddsulli1]

This issue seems to exist in v3.0.0

[Tantael]

Confirming, this issue still exists in 3.0.0

[zhuoyaozzy]

looks like this issue still exits in the latest version (v6.0)

[neerava2023]

Having similar issues, my service has a really fast rate of file changes and the memory usage builds up really quickly. I have noticed the same areas of concern.

Also I wanted to add that the event_queue is never really cleaned up leading to further build up.