gorakhargosh / watchdog

Python library and shell utilities to monitor filesystem events.
http://packages.python.org/watchdog/
Apache License 2.0
6.46k stars 690 forks source link

Lots of missing events when using not PollingObserver #391

Open Laberbear opened 7 years ago

Laberbear commented 7 years ago

I've had huge issues with this module when copying and deleting multiple folders at once in the watched folder. Lots of events go missing indefinitely even when falling back to the example LoggingEventHandler. However the issue only seems to be affected when the default Observer for Windows is used. As soon as I force the PollingObserver, all events are catched easily.

System: Windows 10 64 Bit (Build 14393) Python 3.5.2 watchdog (0.8.3)

Is anyone else also having issues with this?

brunohadlich commented 6 years ago

Same problem using:

Windows 10 64 Bit (Build 16299.125) Python 3.6.4 watchdog (0.8.3)

As described by @Laberbear this happens when a bunch of files is copied with folders in the middle, I tested copying 6660 files with no folder and the modules captured all events but when I added these files to a folder and did the copy it captured 6656 files, all the files were empty.

The situation is much worse when copying a folder with multiple subfolders and files that range from KB to MB. When copying NVIDIA GPU Computing Toolkit folder with 3916 files from System Programs to the observed folder the maximum number of events that were captured was 2756.

Does anyone know what can be causing this?

moooV252 commented 3 years ago

Windows 10 64bit (19042.928) Python 3.8.6 Watchdog 2.0.3

I'm having the same issue - if I'm using Observer() it misses 90% of events, but catches everything if I change it to PollingObserver().

I've tried changing observer.event_queue.maxsize and a timeout, no effect at all.

UPDATE: I've narrowed it down to watchdog.observers.winapi.PATH_BUFFER_SIZE If I increase it to 256000 it starts working and catches most of events (I'm adding around 500 files in less than 1 second), but I haven't tested it over the network (as mentioned in the source file)

UPDATE2: Yep, this tweak doesn't work over the network.