Has anyone tried running this in a cluster mode with pm2

[astutesoftware]

I was wondering what would happen if file watching is clustered using pm2. Will all the cluster nodes get the same lines or each one of them would get a different line. I am working on a watcher where a few thousand lines are written every second and thought clustering it could let my node app use a different CORE for processing.

Any thoughts?

[soldair]

Each process that runs tailfd will get all the lines. But reading the lines is often the small part of the job. It might be fine to do that in each process.

your work on the other hand (parsing etc.) you want to load balance equally to workers. there are 2 ways as i see them:

• run a master process, and the others receive lines over a socket from the "master"
• if you have each worker handle only every nth line and ignore the others.

i made http://npmjs.org/forkfriend a few years ago for this problem. i have no idea if it still works but im adding it here just in case it helps =)

[soldair]

sorry miss-clicked close and comment. the benefit of this module is that if the file is unlinked moved etc it will keep reading till the descriptor becomes inactive. and if any new descriptors appear at the path it will start to read from them as well.

if you dont need these features i have another called keep-reading. It doesn't use any watchers just polling read and also doesn't parse lines for performance. You would have to use split2 or your own for that.

if you need all the watching features but not lines the "parent" module for tailfd is watchfd which just emits changes and lets you choose how to read them.