[QUESTION] Why "Hello World" response is faster in Node/Express than Crow webserver?

lucpattyn commented 5 years ago

I wrote a simple hello world app with Express and another one with Crow.

Here are the codes for Crow inside main:

CROW_ROUTE(app, "/") ([]{

    return crow::response("<html><body>Hello There!</body></html>");

});

app.port(34000) .multithreaded() .run();

And here is Node Express code :

const express = require('express') const app = express() const port = 35000

app.get('/', (req, res) => {res.send('Hello World!')})

app.listen(port, () => {console.log(Example app listening on port ${port})})

If I send 8000 requests in 10 seconds through Gatlin, node only has 28 failed requests where as crow drops almost 350 requests.

What am I doing wrong ?

arthurafarias commented 5 years ago

First of all. You should compare same stuff. One you send an html document on another you just send a hello world string. The length is different. You should use apache benchmark for that purpose.

lucpattyn commented 5 years ago

The performance remains the same even if I do this: return crow::response("Hello World!");

I am using Gatling to test. The performance of Crow apparently looks disappointing !

arthurafarias commented 5 years ago

Hey, I've tested by myself and what you do is not what I get.

My example app in node.js

const express = require('express')
const app = express()
const port = 35000

app.get('/', (req, res) => { res.send('Hello World!') })

app.listen(port, () => { console.log(`Example app listening on port ${port}`) })

My example app using this library

#include "crow.h"

int main()
{
    crow::SimpleApp app;

    CROW_ROUTE(app, "/")([](){
        return "Hello world";
    });

    app.port(18080).multithreaded().run();
}

Test results using apache benchmark of express app

➜  ~ ab -e output.csv -c 300 -n 100000 -k http://127.0.0.1:35000/
This is ApacheBench, Version 2.3 <$Revision: 1807734 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests

Server Software:        
Server Hostname:        127.0.0.1
Server Port:            35000

Document Path:          /
Document Length:        12 bytes

Concurrency Level:      300
Time taken for tests:   6.774 seconds
Complete requests:      100000
Failed requests:        0
Keep-Alive requests:    100000
Total transferred:      21600000 bytes
HTML transferred:       1200000 bytes
Requests per second:    14762.41 [#/sec] (mean)
Time per request:       20.322 [ms] (mean)
Time per request:       0.068 [ms] (mean, across all concurrent requests)
Transfer rate:          3113.95 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1  35.0      0    1014
Processing:    10   19   6.3     18     124
Waiting:       10   19   6.3     18     124
Total:         10   20  36.9     18    1073

Percentage of the requests served within a certain time (ms)
  50%     18
  66%     18
  75%     19
  80%     20
  90%     23
  95%     30
  98%     37
  99%     42
 100%   1073 (longest request)

Test results of crow app

➜  ~ ab -e output.csv -c 300 -n 100000 -k http://127.0.0.1:18080/
This is ApacheBench, Version 2.3 <$Revision: 1807734 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests

Server Software:        Crow/0.1
Server Hostname:        127.0.0.1
Server Port:            18080

Document Path:          /
Document Length:        11 bytes

Concurrency Level:      300
Time taken for tests:   4.271 seconds
Complete requests:      100000
Failed requests:        0
Keep-Alive requests:    100000
Total transferred:      12900000 bytes
HTML transferred:       1100000 bytes
Requests per second:    23411.25 [#/sec] (mean)
Time per request:       12.814 [ms] (mean)
Time per request:       0.043 [ms] (mean, across all concurrent requests)
Transfer rate:          2949.27 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1  36.5      0    1026
Processing:     0   11   5.6     10      48
Waiting:        0   11   5.6     10      48
Total:          0   13  37.5     10    1070

Percentage of the requests served within a certain time (ms)
  50%     10
  66%     13
  75%     15
  80%     16
  90%     19
  95%     22
  98%     24
  99%     26
 100%   1070 (longest request)

Memory footprint of both applications

1 - Crow: 172 KiB 2 - Node: 15.6 MiB

How are you doing your tests? Maybe your benchmark tool is a bottleneck.

lucpattyn commented 5 years ago

I am using Gatling for bench marking. I think some configuration for crow needs to be set in the server. Maybe increase timeout value or compile with some specific compiler switches ?

Will this project be maintained. I think beast is a good alternative for Crow.

arthurafarias commented 5 years ago

No, just clone, clone dependencies, checkout latest release and compile the example given.

arthurafarias commented 5 years ago

Could you share your entire setup that can I test by myself with apache benchmark?

lucpattyn commented 5 years ago

I git cloned and compiled for Mac OSX according to the instructions. Then used gcc (g++) to compile the cpp file. I can see you are making a huge number of requests successfully to the server. I am trying 15000 hits in 20 seconds using Gatling from another laptop from inside the same network (internal lan) and both nodejs and crow drops a lot of requests (node drops around 1000, and crow around 2000) and the error says premature socket close. I am wondering how you make that many requests (100000) without a single drop. It seems you are doing it from the same pc. I am using one Mac Desktop as my server and using a laptop from the same network. Maybe you can try to hit your server from a different pc (not 127.0.0.1).

arthurafarias commented 5 years ago

The problem is:

What are you testing? Your network or the library itself? If you want to test the library you should eliminate all spurious effects that could lead your test to be unconclusive. That is, you should guarantee that there are no elements that can influence your test besides the element under test itself.

In the first experiment you tried to use a tool that is a bottleneck itself. It seems that Gatling cannot put both servers through the limit. Its a tool based on JVM is not something that you should use to profile high performance elements.

In the second test you continue to apply the same tool introducing the same methodological error, and now you introduce another source of error: a physical network. Certainly your network cannot handle a sufficient amount of requests necessary to reach the limits of both technologies resulting in another bottleneck that leads to an unconclusive experiment, again. To justify using a physical network in the middle, the bandwidth should be greater than the loopback adapter in your network. This never gonna happend.

arthurafarias commented 5 years ago

I could explain you this principle with thermodinamics, but it would be really tedious. There is a master thesis[1] from a portuguese engineer that can explain profiling methodologies and spurious variables that introduces errors in your experiment. It could help you to profile this library and others, compare them and make correct assumptions based on a solid methodology.

lucpattyn commented 5 years ago

Thanks for the tip. On this note about network, wondering if you could give me another tip : if I host my server in amazon or digital ocean (or similar), how do I ensure the (Linux) server is able to accept 100000 connections in a short span of time (like a few seconds)? Is there an option in Linux or the crow/node server to set max number of connections allowed per second?

arthurafarias commented 5 years ago

I don't think so, but I can't answer your. I don't know even if there is any http server that can do it. What I know is the fact that most web servers can limit the number of concurrent connections. I don't know if there is a property in this library that can control this aspect. I think you should use a solution like Nginx as a proxy to your service as recommended to any microservice including the ones that are built with Node. Nginx, in fact, can control the number of concurrent connections[1].

arthurafarias commented 5 years ago

Checkout this minimal example on how to use opencv + crow that I've built today. I think this library has a lot of potential.

https://github.com/arthurafarias/microservice-opencv-filter-gausian

lucpattyn commented 5 years ago

checking it out, looks exciting !

ipkn / crow

[QUESTION] Why "Hello World" response is faster in Node/Express than Crow webserver? #335