search

madhuneal / ppss

Automatically exported from code.google.com/p/ppss
0 stars 0 forks source link

daemon mode lockup when using inotify #38

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?
1. ppss --daemon -d /data/convert -c '/data/convert.sh "$ITEM"'
2.
3.

What is the expected output? What do you see instead?

  After a minute or two ppss never processes any new files.

  If the --disable-inotify option is used, the locked never happens.

What version of the product are you using? On what operating system?
  2.83

Please provide any additional information below.

 Here is an extra from the logfile. It seems to be a loop processing the same file, even though it has been processed and no longer exists in the directory.

Nov 14 18:01:15: INFO  Now processing /data/convert/file-34226659.wav
Nov 14 18:01:15: INFO  Current active workers is 4
Nov 14 18:01:16: INFO  Now processing /data/convert/file-34226659.wav
Nov 14 18:01:16: INFO  Current active workers is 4
Nov 14 18:01:17: INFO  Now processing /data/convert/file-34226659.wav
Nov 14 18:01:17: INFO  Current active workers is 4
Nov 14 18:01:18: INFO  Now processing /data/convert/file-34226659.wav
Nov 14 18:01:18: INFO  Current active workers is 4
Nov 14 18:01:19: INFO  Now processing /data/convert/file-34226659.wav
Nov 14 18:01:19: INFO  Current active workers is 4
Nov 14 18:01:19: INFO  Now processing /data/convert/file-34226659.wav
Nov 14 18:01:19: INFO  Current active workers is 4
Nov 14 18:01:20: INFO  Now processing /data/convert/file-34226659.wav
Nov 14 18:01:20: INFO  Current active workers is 4
Nov 14 18:01:21: INFO  Now processing /data/convert/file-34226659.wav
Nov 14 18:01:21: INFO  Current active workers is 4
Nov 14 18:01:22: INFO  Now processing /data/convert/file-34226659.wav
Nov 14 18:01:22: INFO  Current active workers is 4
Nov 14 18:01:23: INFO  Now processing /data/convert/file-34226659.wav
Nov 14 18:01:23: INFO  Current active workers is 4
Nov 14 18:01:24: INFO  Now processing /data/convert/file-34226659.wav
Nov 14 18:01:24: INFO  Current active workers is 4

Original issue reported on code.google.com by vince.sw...@gtempaccount.com on 14 Nov 2010 at 6:19

GoogleCodeExporter commented 9 years ago 
Thanks, I look into it.

Original comment by Louwrentius on 16 Nov 2010 at 11:19

GoogleCodeExporter commented 9 years ago 
Hi.

I think I found the problem.

Some files were being copied into the watched directory via NFS. It
seems the NFS daemon writes the files in chunks instead of a single
write-then-close event. Hence inotify was triggered multiple times for
the same file which broke ppss.

Vince.

Original comment by vince.sw...@gtempaccount.com on 18 Nov 2010 at 2:28

GoogleCodeExporter commented 9 years ago 
Thanks, that is very interesting. it means that PPSS should take this into 
account.

Original comment by Louwrentius on 19 Nov 2010 at 9:36

GoogleCodeExporter commented 9 years ago 
Ha, you where quite right:

server:/storage/ppss# inotifywait /storage/ppss/ -m -e close -q --format '%w%f'
/storage/ppss/test1.bin
/storage/ppss/test1.bin
/storage/ppss/test1.bin
/storage/ppss/test1.bin
/storage/ppss/test1.bin
/storage/ppss/test1.bin
/storage/ppss/test1.bin
/storage/ppss/test1.bin
/storage/ppss/test1.bin
/storage/ppss/test1.bin
/storage/ppss/test1.bin
/storage/ppss/test1.bin
/storage/ppss/test1.bin
/storage/ppss/test1.bin
/storage/ppss/test1.bin
/storage/ppss/test1.bin
/storage/ppss/test1.bin
/storage/ppss/test1.bin
/storage/ppss/test1.bin
/storage/ppss/test1.bin
/storage/ppss/test1.bin
/storage/ppss/test1.bin
/storage/ppss/test1.bin
/storage/ppss/test1.bin
/storage/ppss/test1.bin
/storage/ppss/test1.bin
/storage/ppss/test1.bin

I will see what I can do about this.

Original comment by Louwrentius on 21 Nov 2010 at 6:49

GoogleCodeExporter commented 9 years ago 
It seems that adding | uniq sorta resolved this issue, but there is a cacht. 
When an event is caught, the file must be closed. But due to the multiple close 
events, the file is reported to be closed before the file transfer is finished. 

So inotify must be disabled for directories that are exported through NFS. 
Regular polling should be used.

Original comment by Louwrentius on 21 Nov 2010 at 6:59

GoogleCodeExporter commented 9 years ago 
PPSS now disabled inotify if the source directory is exported with NFS. If this 
can be confirmed, it is appreciated.

Original comment by Louwrentius on 21 Nov 2010 at 7:52

GoogleCodeExporter commented 9 years ago

Original comment by Louwrentius on 26 Nov 2010 at 9:01