search

seb-m / pyinotify

Monitoring filesystems events with inotify on Linux.
http://github.com/seb-m/pyinotify/wiki
MIT License
2.29k stars 379 forks source link

Watching IN_CLOSE_WRITE results in IN_CREATE to also be fired #141

Open fthiery opened 7 years ago

fthiery commented 7 years ago

Hi there; not 100% sure this is pyinotify-related, but i noticed that sometimes when i receive the IN_CLOSE_WRITE for a relatively large file, the file itself isn't finalized (i checked this by printing os.path.getsize on the event processing callback).

I added a sleep(1) there, but i'd be happier if i could get the confirmation that the file is really finished writing and available for processing. Any idea ?

fthiery commented 7 years ago

Edit: i was listening to IN_CLOSE_WRITE, and noticed that IN_CREATE was also triggering the callback self.handle_event:

    self.wm = pyinotify.WatchManager()
    pyinotify.Notifier(self.wm, default_proc_fun=self.handle_event)
    self.wm.add_watch(f, pyinotify.IN_CLOSE_WRITE, auto_add=True)

Is this expected ?

leelouch commented 7 years ago

Hello, I noticed the same behavior. any workaround ? Thx

fthiery commented 7 years ago

Well, i just drop the call:

def on_event(self, event):
        if event.maskname == 'IN_CREATE':
            pass
        elif event.maskname == 'IN_CLOSE_WRITE':
            self.do_action(event.pathname)
leelouch commented 7 years ago

Thanks for your quick answer. in process_IN_CLOSE_WRITE method I droped the event IN_CREATE as you mentioned. but still receiving IN_CLOSE_WRITE multiple time during the action process (move in my example). I am catching only IN_CLOSE_WRITE event

wm = pyinotify.WatchManager()
mask = (
    pyinotify.IN_CLOSE_WRITE
)

and if you have a look to the log, the file ...175645.JPG is called twice and lead to an empty file! this is weird ! I am not sure what is happening. any idea ? A sleep, will help me solving the problem for small files ! but for big files this will not work unfortunately.

2017-04-10T15:56:59.657821 - classify IN_CLOSE_WRITE /volume1/photo/Iphone/2017/IMG_20170410_175642.JPG
2017-04-10T15:56:59.774213 -    mv /volume1/photo/Iphone/2017/IMG_20170410_175642.JPG /media/photo/./2017/04/IMG_20170410_175642.JPG
2017-04-10T15:57:01.765019 - classify IN_CLOSE_WRITE /volume1/photo/Iphone/2017/IMG_20170410_175645.JPG
2017-04-10T15:57:01.780264 -    mv /volume1/photo/Iphone/2017/IMG_20170410_175645.JPG /media/photo/./2017/04/IMG_20170410_175645.JPG
2017-04-10T15:57:01.789337 - classify IN_CLOSE_WRITE /volume1/photo/Iphone/2017/IMG_20170410_175645.JPG
2017-04-10T15:57:01.820743 -    mv /volume1/photo/Iphone/2017/IMG_20170410_175645.JPG /media/photo/./2017/04/IMG_20170410_175645.JPG
2017-04-10T15:57:03.551751 - classify IN_CLOSE_WRITE /volume1/photo/Iphone/2017/IMG_20170410_175650.JPG
2017-04-10T15:57:03.568519 -    mv /volume1/photo/Iphone/2017/IMG_20170410_175650.JPG /media/photo/./2017/04/IMG_20170410_175650.JPG
2017-04-10T15:57:03.573922 - classify IN_CLOSE_WRITE /volume1/photo/Iphone/2017/IMG_20170410_175650.JPG
2017-04-10T15:57:03.592129 -    mv /volume1/photo/Iphone/2017/IMG_20170410_175650.JPG /media/photo/./2017/04/IMG_20170410_175650.JPG
fthiery commented 7 years ago

Well, it's possible that the software that is actually writing the files is opening and closing them multiple times in a row. What happens if you print os.path.getsize or os.system('md5sum %s' % file) ? Is it changing the contents of the file between the 2 calls ?

leelouch commented 7 years ago

it seems that the problem is related to DS_PHOTO (synology photo tool) somehow! I added a sleep(1) before processing the file to allow DS_PHOTO to see that the file has been uploaded !

fthiery commented 7 years ago

If the double call always happens, you can also ignore the first one by adding the items to a list and popping it the second time it appears...