Request rate to remote schema through hasura engine is very slow

[bkstorm]

Bug description

I try to benchmark our server in production environment. I use autocannon for that purpose. Case 1: I will benchmark hasura engine (requests will be sent to hasura engine which will forward theme to remote schema: our API server). Case 2: I will send directly requests to the API server. The result is very surprised. The request rate in case 1 is hundred times lower than case 2. I test again in local environment, the result does not change. As you can see, the latency in case 1 is higher than case 2.

Case 1:

Case 2:

How to reproduce

I just create a repository to reproduce this issue: https://github.com/bkstorm/hasura-engine-benchmark

Expected behavior

I hope the request rate in two cases are similar.

Hasura information

Environment & setup

• OS: Mac OS
• Database: PostgreSQL
• Node.js version: v12.22.1
• Hasura version: v1.3.3

[coco98]

@bkstorm Thanks for the details and the link to the repo. We'll triage this asap.

[eviefp]

Hello @bkstorm and thank you again for the detailed report! I followed the instructions on your reproduction repository.

One thing we noticed is that the autocannon command for testing the API results in non-2xx responses.

I switched the benchmark-api line in package.json to:

"benchmark-api": "autocannon -m POST -i body.json -H 'Content-Type'='application/json' http://localhost:4000/graphql"

And that seems to have fixed the problem. Using this new command, here are the results:

➜  hasura-engine-benchmark git:(master) ✗ npm run benchmark-api
> hasura-engine-benchmark@1.0.0 benchmark-api
> autocannon -m POST -i body.json http://localhost:4000/graphql --headers Content-Type='application/json'
Running 10s test @ http://localhost:4000/graphql
10 connections
┌─────────┬──────┬──────┬───────┬──────┬─────────┬─────────┬───────┐
│ Stat    │ 2.5% │ 50%  │ 97.5% │ 99%  │ Avg     │ Stdev   │ Max   │
├─────────┼──────┼──────┼───────┼──────┼─────────┼─────────┼───────┤
│ Latency │ 2 ms │ 2 ms │ 6 ms  │ 8 ms │ 2.77 ms │ 1.39 ms │ 15 ms │
└─────────┴──────┴──────┴───────┴──────┴─────────┴─────────┴───────┘
┌───────────┬────────┬────────┬─────────┬────────┬─────────┬─────────┬────────┐
│ Stat      │ 1%     │ 2.5%   │ 50%     │ 97.5%  │ Avg     │ Stdev   │ Min    │
├───────────┼────────┼────────┼─────────┼────────┼─────────┼─────────┼────────┤
│ Req/Sec   │ 2455   │ 2455   │ 2925    │ 3035   │ 2909    │ 153.23  │ 2454   │
├───────────┼────────┼────────┼─────────┼────────┼─────────┼─────────┼────────┤
│ Bytes/Sec │ 893 kB │ 893 kB │ 1.06 MB │ 1.1 MB │ 1.06 MB │ 55.9 kB │ 893 kB │
└───────────┴────────┴────────┴─────────┴────────┴─────────┴─────────┴────────┘
Req/Bytes counts sampled once per second.
32k requests in 11.01s, 11.6 MB read

The benchmark for hasura is:

➜  hasura-engine-benchmark git:(master) ✗ npm run benchmark-hasura
> hasura-engine-benchmark@1.0.0 benchmark-hasura
> autocannon -m POST -i body.json http://localhost:8082/v1/graphql --headers Content-Type='application/json'
Running 10s test @ http://localhost:8082/v1/graphql
10 connections
┌─────────┬──────┬──────┬───────┬───────┬─────────┬─────────┬───────┐
│ Stat    │ 2.5% │ 50%  │ 97.5% │ 99%   │ Avg     │ Stdev   │ Max   │
├─────────┼──────┼──────┼───────┼───────┼─────────┼─────────┼───────┤
│ Latency │ 2 ms │ 6 ms │ 15 ms │ 19 ms │ 6.47 ms │ 3.59 ms │ 48 ms │
└─────────┴──────┴──────┴───────┴───────┴─────────┴─────────┴───────┘
┌───────────┬────────┬────────┬────────┬────────┬────────┬─────────┬────────┐
│ Stat      │ 1%     │ 2.5%   │ 50%    │ 97.5%  │ Avg    │ Stdev   │ Min    │
├───────────┼────────┼────────┼────────┼────────┼────────┼─────────┼────────┤
│ Req/Sec   │ 1191   │ 1191   │ 1409   │ 1648   │ 1432.5 │ 128.14  │ 1191   │
├───────────┼────────┼────────┼────────┼────────┼────────┼─────────┼────────┤
│ Bytes/Sec │ 342 kB │ 342 kB │ 404 kB │ 473 kB │ 411 kB │ 36.7 kB │ 342 kB │
└───────────┴────────┴────────┴────────┴────────┴────────┴─────────┴────────┘
Req/Bytes counts sampled once per second.
14k requests in 10.02s, 4.11 MB read

So while there is some difference, it's nowhere near the difference in the OP.

I will not be closing this yet because we are still investigating some ways to improve the performance still, but hopefully the above should shed some light into the big difference you're currently seeing.