search

bpaquet / node-logstash

Simple logstash implmentation in nodejs : file log collection, sent with zeromq
Other
517 stars 141 forks source link

input_file does not work with the files on NFS (nfsv4) #134

Open ngrigoriev opened 8 years ago

ngrigoriev commented 8 years ago

I have attempted to store the logs from multiple small servers on one NFS share and have them processed on one small dedicated host running NLS instead of having the agent on each host. It did not work - input_file could never detect any changes in the files being written.

I did a bit of research (I am not a JS developer) and turned out that "inotify" mechanism used by Node.JS (offered via fs.watch() ) is not working with the files on NFS. Probably it is NFSv4 vs NFSv3, hard to say - but it does not. And I could not find a definitive answer why and what can be done to get it working on Linux with node.js.

I have noticed that input_file offered an alternative based on "tail" command. It actually worked for me except one thing: it runs "tail -f" and since that tail_file.js still uses Node's "fs" module to watch the directory it never detects the files being rotated. Thus, to fix the problem I had to switch from "tail -f -n..." to "tail -F -n..." in tail_file.js. This seems to work.

ngrigoriev commented 8 years ago

Actually, it seems that fs.watchFile() is cross-platform and, while less efficient, should probably work for all cases. Maybe another improvement for input_file would be to add a flag that forces the use of watchFile() instead of watch().