Use cluster.

[thomasvnoort]

To improve performance I've added support for cluster. I thought about using a more advanced library than cluster, but I think this will suffice. Also, logging could include the worker's pid but I'm not sure if that's useful.

[vbauer]

@thomasvnoort It's awesome, thank you!

[vbauer]

Hmm... Maybe it isn't so great as I thought. I've just done some benchmarks with siege and it shows the same performance for 1 worker and for 30 workers. Actually one worker works a little bit faster than 30 workers. Even one nodejs process can execute a lot of phantomjs workers.

@thomasvnoort Could you please provide your benchmarks?

[thomasvnoort]

What version of node are you using? Because there's this.

[vbauer]

I've checked on the latest Node.js version (4.1.1).

Benchmark 1: siege -c 30 -t 20s "http://localhost:8891/?url=google.com&force=true"

Results for ./bin/manet --workers 1:

Transactions:                     50 hits
Availability:                 100.00 %
Elapsed time:                  21.64 secs
Data transferred:               1.61 MB
Response time:                  9.82 secs
Transaction rate:               2.31 trans/sec
Throughput:                     0.07 MB/sec
Concurrency:                   22.69
Successful transactions:          50
Failed transactions:               0
Longest transaction:           20.02
Shortest transaction:           3.73

Results for ./bin/manet --workers 30:

Transactions:                     34 hits
Availability:                 100.00 %
Elapsed time:                  19.75 secs
Data transferred:               1.22 MB
Response time:                  8.77 secs
Transaction rate:               1.72 trans/sec
Throughput:                     0.06 MB/sec
Concurrency:                   15.10
Successful transactions:          37
Failed transactions:               0
Longest transaction:           18.13
Shortest transaction:           2.79

Benchmark 2: siege -c 30 -t 20s "http://localhost:8891/?url=google.com"

Results for ./bin/manet --workers 1:

Transactions:                   1147 hits
Availability:                 100.00 %
Elapsed time:                  19.74 secs
Data transferred:              37.76 MB
Response time:                  0.01 secs
Transaction rate:              58.11 trans/sec
Throughput:                     1.91 MB/sec
Concurrency:                    0.78
Successful transactions:        1147
Failed transactions:               0
Longest transaction:            0.11
Shortest transaction:           0.00

Results for ./bin/manet --workers 30:

Transactions:                   1136 hits
Availability:                 100.00 %
Elapsed time:                  19.11 secs
Data transferred:              37.40 MB
Response time:                  0.01 secs
Transaction rate:              59.46 trans/sec
Throughput:                     1.96 MB/sec
Concurrency:                    0.73
Successful transactions:        1136
Failed transactions:               0
Longest transaction:            0.18
Shortest transaction:           0.00

[thomasvnoort]

You're absolutely right! Using `cluster gives little to no performance improvement, sorry about that. I've investigated a bit more and I guess this is mostly because the shell commands to phantomjs are async, which is where most of the time is spent. I'm ambivalent about reverting this PR, it doesn't help much in performance but it might improve reliability since the workers are respawned automatically when one crashes. I'll leave that decision up to you.

[vbauer]

I think I'll revert this changes, because we can use something like Forever to restart application "on crash", but anyway, thank you for contribution.

[thomasvnoort]

Sure, and again, sorry for failing to test this properly!

[vbauer]

No problem. :)