Open bkstorm opened 3 years ago
@bkstorm Thanks for the details and the link to the repo. We'll triage this asap.
Hello @bkstorm and thank you again for the detailed report! I followed the instructions on your reproduction repository.
One thing we noticed is that the autocannon
command for testing the API results in non-2xx responses.
I switched the benchmark-api
line in package.json
to:
"benchmark-api": "autocannon -m POST -i body.json -H 'Content-Type'='application/json' http://localhost:4000/graphql"
And that seems to have fixed the problem. Using this new command, here are the results:
➜ hasura-engine-benchmark git:(master) ✗ npm run benchmark-api
> hasura-engine-benchmark@1.0.0 benchmark-api
> autocannon -m POST -i body.json http://localhost:4000/graphql --headers Content-Type='application/json'
Running 10s test @ http://localhost:4000/graphql
10 connections
┌─────────┬──────┬──────┬───────┬──────┬─────────┬─────────┬───────┐
│ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │
├─────────┼──────┼──────┼───────┼──────┼─────────┼─────────┼───────┤
│ Latency │ 2 ms │ 2 ms │ 6 ms │ 8 ms │ 2.77 ms │ 1.39 ms │ 15 ms │
└─────────┴──────┴──────┴───────┴──────┴─────────┴─────────┴───────┘
┌───────────┬────────┬────────┬─────────┬────────┬─────────┬─────────┬────────┐
│ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │
├───────────┼────────┼────────┼─────────┼────────┼─────────┼─────────┼────────┤
│ Req/Sec │ 2455 │ 2455 │ 2925 │ 3035 │ 2909 │ 153.23 │ 2454 │
├───────────┼────────┼────────┼─────────┼────────┼─────────┼─────────┼────────┤
│ Bytes/Sec │ 893 kB │ 893 kB │ 1.06 MB │ 1.1 MB │ 1.06 MB │ 55.9 kB │ 893 kB │
└───────────┴────────┴────────┴─────────┴────────┴─────────┴─────────┴────────┘
Req/Bytes counts sampled once per second.
32k requests in 11.01s, 11.6 MB read
The benchmark for hasura
is:
➜ hasura-engine-benchmark git:(master) ✗ npm run benchmark-hasura
> hasura-engine-benchmark@1.0.0 benchmark-hasura
> autocannon -m POST -i body.json http://localhost:8082/v1/graphql --headers Content-Type='application/json'
Running 10s test @ http://localhost:8082/v1/graphql
10 connections
┌─────────┬──────┬──────┬───────┬───────┬─────────┬─────────┬───────┐
│ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │
├─────────┼──────┼──────┼───────┼───────┼─────────┼─────────┼───────┤
│ Latency │ 2 ms │ 6 ms │ 15 ms │ 19 ms │ 6.47 ms │ 3.59 ms │ 48 ms │
└─────────┴──────┴──────┴───────┴───────┴─────────┴─────────┴───────┘
┌───────────┬────────┬────────┬────────┬────────┬────────┬─────────┬────────┐
│ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │
├───────────┼────────┼────────┼────────┼────────┼────────┼─────────┼────────┤
│ Req/Sec │ 1191 │ 1191 │ 1409 │ 1648 │ 1432.5 │ 128.14 │ 1191 │
├───────────┼────────┼────────┼────────┼────────┼────────┼─────────┼────────┤
│ Bytes/Sec │ 342 kB │ 342 kB │ 404 kB │ 473 kB │ 411 kB │ 36.7 kB │ 342 kB │
└───────────┴────────┴────────┴────────┴────────┴────────┴─────────┴────────┘
Req/Bytes counts sampled once per second.
14k requests in 10.02s, 4.11 MB read
So while there is some difference, it's nowhere near the difference in the OP.
I will not be closing this yet because we are still investigating some ways to improve the performance still, but hopefully the above should shed some light into the big difference you're currently seeing.
Bug description
I try to benchmark our server in production environment. I use autocannon for that purpose. Case 1: I will benchmark hasura engine (requests will be sent to hasura engine which will forward theme to remote schema: our API server). Case 2: I will send directly requests to the API server. The result is very surprised. The request rate in case 1 is hundred times lower than case 2. I test again in local environment, the result does not change. As you can see, the latency in case 1 is higher than case 2.
Case 1:
Case 2:
How to reproduce
I just create a repository to reproduce this issue: https://github.com/bkstorm/hasura-engine-benchmark
Expected behavior
I hope the request rate in two cases are similar.
Hasura information
Environment & setup