Benchmarking against other proxies

[thibaultcha]

A benchmark against other proxies (Umbrella, Tyk...) is required before launch along with a blogpost probably.

[subnetmarco]

In regard to benchmarking, I think it would make sense to have a distributed multi-client benchmarking tool that we can start also for benchmarking the performance between critical code changes, something like http://www.process-one.net/en/tsung/

Distributed: the load can be distributed on a cluster of client machines (http://www.process-one.net/docs/tsung/user_manual.pdf)

[SGrondin]

There's a SAAS for that.

[SGrondin]

https://www.blitz.io/ http://blazemeter.com/ https://loader.io/ http://loadimpact.com/

[subnetmarco]

SAAS services could be an option, but they are very expensive with many concurrent requests so we need to evaluate that.

[mschurenko]

I'm not sure why we need a "distributed multi-client benchmarking tool". Why can't we just run something like siege on a large ec2 instance and use a lot of threads? We only have to benchmark one instance of kong. As long as the instance used to run siege has more threads than the one running kong I don't see why that wouldn't be fine.

[subnetmarco]

Benchmarking the interaction between Kong and Cassandra requires lots of incoming requests.

siege will work for benchmarking a single nginx server on one machine, but it wouldn't work when we want to simulate a bigger load on Cassandra that requires multiple nginx servers.

We can start with siege (and an easy way to launch it and retrieve benchmarking information), but keep in mind that for deeper benchmarking we would need more than one client (and more than one nginx server to simulate lots of incoming connections on Cassandra).

[montanaflynn]

Besides Tsung there is also http://locust.io

[subnetmarco]

Sounds good, can we set something up @mschurenko @ataner ?

[montanaflynn]

@mschurenko @ataner Can we get Kong running on AWS? This would serve for both benchmarking and learning more about how different settings (OS, nginx, kong, cassandra) and architectures (cassandra on the same box vs external vs clusters) effect performance.

When we have that we need to benchmark the following scenarios:

1) Kong vs Nginx vs Tyk proxies 2) Kong Proxy vs Direct API

So that means we also need to set up an AWS instance for raw nginx and tyk using the same instance type and applicable configurations.

[thibaultcha]

+1 we need to know where we're at now. Also Umbrella

We should talk about setups (docker and cassandra)

[thibaultcha]

So are those finished? Published? Please update the issue @SGrondin

[SGrondin]

I'm still waiting on your Tyk instructions to remove all rate limiting. I couldn't figure it out whatsoever. By default it has some rate limiting and the docs are unclear about wheter it's even possible.

[thibaultcha]

My Tyk instructions? For 2 weeks? I don't have such thing lol. i never removed the rate limiting from Tyk.

[hbakhtiyor]

@thibaultcha any updates? what's the results of kong vs tyk?

[hadifarnoud]

updates on this? very crucial for decision making