Nginx vs non-nginx results

[vtt]

I don't see the config files for nginx; but from the look of the results, it seems that it was configured as a reversed-proxy; so you'll only see the extra overhead here.

In order to see the benefits of nginx as reversed proxy, the application server must be running multiple workers/threads and you'll see the results much better with nginx vs non-nginx.

Pure nginx as a web/application server can only be benchmarked with the "hello_world" output; ie: use a static index.html to return the text "hello world", and its result will blow away all others.

If added Lua to nginx to output a number from database (eg. using OpenResty), then it should also blow away all other benchmarks.

[greg-nagy]

It would be interesting to see the numbers for Lua for sure.

Would you want to do the setup and run the tests? I am happy to give you access to the aws account and the machines.

[vtt]

Install OpenResty and postgres libs:

# wget https://openresty.org/download/openresty-1.21.4.3.tar.gz
# tar zxvf openresty-1.21.4.3.tar.gz
# cd openresty-1.21.4.3
# ./configure
# make
# make install  # this will install to /usr/local/openresty
# /usr/local/openresty/bin/opm get fffonion/lua-resty-openssl
# /usr/local/openresty/bin/opm get leafo/pgmoon

Edit nginx configuration file: /usr/local/openresty/nginx/conf/nginx.conf

worker_processes  1;
error_log logs/error.log;
events {
  worker_connections 1024;
}
http {
  server {
    listen 8080;
    location / {
      default_type text/html;
      content_by_lua_block {
        ngx.say("Hello World")
      }
    }

    location /count {
      default_type text/html;
      content_by_lua_block {
        local pgmoon = require("pgmoon")
        local pg = pgmoon.new({
          host     = "database.cdgerttxp3su.eu-central-1.rds.amazonaws.com",
          port     = "5432",
          user     = "postgres",
          password = "postgres",
          database = "portal_dev"
        })
        assert(pg:connect())
        local res = assert(pg:query("select count from presence_counters where name = 'group_sittings' limit 1"))
        pg:keepalive()
        pg = nil
        ngx.say(res[1]["count"])
      }
    }
  }
}

Start OpenResty

# /usr/local/openresty/bin/openresty

Run benchmarks with http://localhost:8080/ and http://localhost:8080/count

[vtt]

May need libssl and libpcre and perl for depency:

# apt-get install libpcre3-dev libssl-dev perl make build-essential

[greg-nagy]

it also needed zlib-dev and a configuration for dns in the config, but it's working now \o/

the results are in https://docs.google.com/spreadsheets/d/1qxV0QYc6v51SDrhph7MY6PEOdoGL_IYRNgvRJmV1GKk/edit#gid=609788077

[vtt]

Hmm, something is wrong there; it is showing nginx (using static html - row #9) faster than Lua plain text (row #10)? On my local machine, Lua plain text is more than 2x faster.

Lua plain text

tinvo@development:/usr/local/openresty/nginx# rewrk -c 50 -t 2 -d 30s -h http://localhost:8080/
Beginning round 1...
Benchmarking 50 connections @ http://localhost:8080/ for 30 second(s)
  Latencies:
    Avg      Stdev    Min      Max      
    2.42ms   1.51ms   0.41ms   104.56ms  
  Requests:
    Total: 620376  Req/Sec: 20680.54
  Transfer:
    Total: 108.27 MB Transfer Rate: 3.61 MB/Sec

Nginx index.html

tinvo@development:/usr/local/openresty/nginx# rewrk -c 50 -t 2 -d 30s -h http://localhost:8080/index
Beginning round 1...
Benchmarking 50 connections @ http://localhost:8080/index for 30 second(s)
  Latencies:
    Avg      Stdev    Min      Max      
    6.37ms   2.37ms   0.18ms   61.58ms  
  Requests:
    Total: 235464  Req/Sec: 7849.07
  Transfer:
    Total: 58.61 MB Transfer Rate: 1.95 MB/Sec

200 Errors: connection closed

Lua count with postgres

tinvo@development:/usr/local/openresty/nginx# rewrk -c 50 -t 2 -d 30s -h http://localhost:8080/count
Beginning round 1...
Benchmarking 50 connections @ http://localhost:8080/count for 30 second(s)
  Latencies:
    Avg      Stdev    Min      Max      
    7.02ms   9.34ms   1.16ms   517.32ms  
  Requests:
    Total: 213540  Req/Sec: 7118.53
  Transfer:
    Total: 35.23 MB Transfer Rate: 1.17 MB/Sec

Nginx with Lua Postgres is almost same as static index.html on my local machine

[vtt]

Make sure you are comparing the same number of nginx workers; the config posted here is using only 1 worker; if you port 80 nginx config is using more than 1, then the numbers are skewed

[greg-nagy]

rerun the tests and they roughly show the same thing

Note 1: I use a client in the same zone and a db server on a different host

Note 2: Something seems to be off with your index.html test. The pure nginx static file serve is likely won't be on :8080 as that's for openresty — but I don't know your exact setup

[vtt]

Also, to be fair with other tests; nginx should only be using 1 worker; if you set it to multiple workers for nginx, then other tests will be at disadvantage. Same applies for other tests - one worker and multiple threads

[greg-nagy]

I was using multiple worker setup, generally left it to the framework to figure out how many workers it wants to use you can see the threadcounts on the w/ nginx tab

I have rerun the openresty tests with 5 workers and 10 workers as well. 10 workers was worse than with 5. I added the 5 worker numbers to the chart

as for the static hello world numbers: I wouldn't be that worried about it, as that is not the important usecase now

[vtt]

Don't overset the number of nginx workers; you'll get worse results. Do 'cat /proc/cpuinfo' to see how many CPU it has and set that number of workers (ie: the output + 1).

Here's the output running with 8 nginx workers thread compared to Rust on my local machine

Nginx index.html

tinvo@development:/usr/local/openresty/nginx# rewrk -c 50 -t 2 -d 30s -h http://localhost:8080/index
Beginning round 1...
Benchmarking 50 connections @ http://localhost:8080/index for 30 second(s)
  Latencies:
    Avg      Stdev    Min      Max      
    0.97ms   0.97ms   0.06ms   11.67ms  
  Requests:
    Total: 1539081 Req/Sec: 51305.39
  Transfer:
    Total: 383.08 MB Transfer Rate: 12.77 MB/Sec

Nginx Lua plain text

tinvo@development:/usr/local/openresty/nginx# rewrk -c 50 -t 2 -d 30s -h http://localhost:8080
Beginning round 1...
Benchmarking 50 connections @ http://localhost:8080 for 30 second(s)
  Latencies:
    Avg      Stdev    Min      Max      
    0.85ms   0.68ms   0.05ms   21.04ms  
  Requests:
    Total: 1764972 Req/Sec: 58834.78
  Transfer:
    Total: 308.02 MB Transfer Rate: 10.27 MB/Sec

Nginx Lua Postgres Count

tinvo@development:/usr/local/openresty/nginx# rewrk -c 50 -t 2 -d 30s -h http://localhost:8080/count
Beginning round 1...
Benchmarking 50 connections @ http://localhost:8080/count for 30 second(s)
  Latencies:
    Avg      Stdev    Min      Max      
    1.85ms   3.76ms   0.18ms   480.55ms  
  Requests:
    Total: 810953  Req/Sec: 27033.38
  Transfer:
    Total: 133.79 MB Transfer Rate: 4.46 MB/Sec

Rust Tokio-Postgres-BB8 plain text

tinvo@development:~/framework-test-at-home/rust_axum_tokio-postgres-bb8# cargo run # on port 4000
Compiling rust_axum_tokio-postgres-bb8 v0.1.0 (/root/framework-test-at-home/rust_axum_tokio-postgres-bb8)
    Finished dev [unoptimized + debuginfo] target(s) in 5.02s
     Running `target/debug/rust_axum_tokio-postgres-bb8`

# different terminal
tinvo@development:/usr/local/openresty/nginx# rewrk -c 50 -t 2 -d 30s -h http://localhost:4000
Beginning round 1...
Benchmarking 50 connections @ http://localhost:4000 for 30 second(s)
  Latencies:
    Avg      Stdev    Min      Max      
    1.27ms   0.69ms   0.13ms   24.57ms  
  Requests:
    Total: 1177620 Req/Sec: 39256.61
  Transfer:
    Total: 144.88 MB Transfer Rate: 4.83 MB/Sec

Rust Tokio-Postgres-BB8 Count (using localhost DB)

tinvo@development:/usr/local/openresty/nginx# rewrk -c 50 -t 2 -d 30s -h http://localhost:4000/count
Beginning round 1...
Benchmarking 50 connections @ http://localhost:4000/count for 30 second(s)
  Latencies:
    Avg      Stdev    Min      Max      
    4.87ms   19.18ms  1.25ms   1536.37ms  
  Requests:
    Total: 308021  Req/Sec: 10268.03
  Transfer:
    Total: 34.37 MB Transfer Rate: 1.15 MB/Sec

[vtt]

If your nginx test is worse than Rust, then your nginx configuration is flawed; I have yet to find any framework/language can beat nginx serving http1/2

[greg-nagy]

I have rerun the openresty and rust tests, and the results didn't change significantly.

At this point, any benchmarked technologies except Ruby would be a good fit for us.

A few thoughts about rust vs openresty

• you are running the rust app in dev/debug mode, as it is shown in the examples you have pasted
  • for release build you need to use cargo build --release then start the output ./target/release/<package_name>
• nginx and openresty with LuaJIT are 2 different things
• Rust seems to outperform LuaJIT consistently
  • https://medium.com/@oviyum/why-we-chose-rust-over-lua-for-our-project-955556eb45b6
  • https://programming-language-benchmarks.vercel.app/lua-vs-rust
• Comparing rust to pure nginx feels pointless. I would not want to implement nginx's features in rust unless I have a good reason to do so or work on a rust webserver.
• Running the tests on localhost probably skews the results
• you are welcome to use the aws setup, or you could quickly spin up your own and continue benchmarking if you want to

Thank you for making this test! I hadn't thought about this solution and was pleasantly surprised at how well openresty has performed.

I am closing this issue as the numbers are in for openresty.

[vtt]

I took your advice and run OpenResty on dev.server.dhamma.org port 2222 (Helsinki), and run Rust --release version on port 3000. Both were connected to 'portal_edge' database locally.

/usr/local/openresty/nginx/conf/nginx.conf

worker_processes  auto;
error_log logs/error.log;
events {
  worker_connections 1024;
}
http {
  server {
    listen 2222;
    access_log off;
    location / {
      default_type text/html;
      content_by_lua_block {
        ngx.say("Hello world!")
      }
    }

    location /count {
      default_type text/html;
      content_by_lua_block {
        local pgmoon = require("pgmoon")
        local pg = pgmoon.new({
          host     = "127.0.0.1",
          port     = "5432",
          user     = "counter",
          password = "counter",
          database = "portal_edge"
        })
        assert(pg:connect())
        local res = assert(pg:query("SELECT count FROM presence_counters WHERE name = 'group_sittings' ORDER BY updated_at DESC LIMIT 1"))
        pg:keepalive()
        pg = nil
        ngx.say(res[1]["count"])
      }
    }
  }
}

/home/tinvo/framework-test-at-home/rust_axum_tokio-postgres-bb8/src/main.rs

use axum::{
    extract::State,
    http::StatusCode,
    routing::get,
    Router,
};
use bb8::Pool;
use bb8_postgres::{PostgresConnectionManager, tokio_postgres::NoTls};

#[tokio::main]
async fn main() {
    let db_connection_string = "postgresql://counter:counter@localhost:5432/portal_edge";
    let manager =
        PostgresConnectionManager::new_from_stringlike(db_connection_string, NoTls)
            .unwrap();
    let pool = Pool::builder()
        .max_size(30)
        .build(manager)
        .await
        .unwrap();

    // build our application with some routes
    let app = Router::new()
        .route( "/", get(|| async { "Hello world!" }))
        .route( "/count", get(fetch_count))
        .with_state(pool);

    // run it
    let listener = tokio::net::TcpListener::bind("0.0.0.0:3000")
        .await
        .unwrap();
    axum::serve(listener, app).await.unwrap();
}

type ConnectionPool = Pool<PostgresConnectionManager<NoTls>>;

async fn fetch_count(
    State(pool): State<ConnectionPool>,
) -> Result<String, (StatusCode, String)> {
    let conn = pool.get().await.map_err(internal_error)?;

    let sql_query = "SELECT count FROM presence_counters WHERE name = 'group_sittings' ORDER BY updated_at DESC LIMIT 1";
    let row = conn
        .query_one(sql_query, &[])
        .await
        .map_err(internal_error)?;
    let count: i32 = row.try_get(0).map_err(internal_error)?;

    Ok(count.to_string())
}

/// Utility function for mapping any error into a `500 Internal Server Error`
/// response.
fn internal_error<E>(err: E) -> (StatusCode, String)
where
    E: std::error::Error,
{
    (StatusCode::INTERNAL_SERVER_ERROR, err.to_string())
}
root@dev:/usr/local/openresty/nginx# nano conf/nginx.conf
root@dev:/usr/local/openresty/nginx# cat ~tinvo/framework-test-at-home/rust_axum_tokio-postgres-bb8/src/main.rs 
use axum::{
    extract::State,
    http::StatusCode,
    routing::get,
    Router,
};
use bb8::Pool;
use bb8_postgres::{PostgresConnectionManager, tokio_postgres::NoTls};

#[tokio::main]
async fn main() {
    let db_connection_string = "postgresql://counter:counter@localhost:5432/portal_edge";
    let manager =
        PostgresConnectionManager::new_from_stringlike(db_connection_string, NoTls)
            .unwrap();
    let pool = Pool::builder()
        .max_size(30)
        .build(manager)
        .await
        .unwrap();

    // build our application with some routes
    let app = Router::new()
        .route( "/", get(|| async { "Hello world!" }))
        .route( "/count", get(fetch_count))
        .with_state(pool);

    // run it
    let listener = tokio::net::TcpListener::bind("0.0.0.0:3000")
        .await
        .unwrap();
    axum::serve(listener, app).await.unwrap();
}

type ConnectionPool = Pool<PostgresConnectionManager<NoTls>>;

async fn fetch_count(
    State(pool): State<ConnectionPool>,
) -> Result<String, (StatusCode, String)> {
    let conn = pool.get().await.map_err(internal_error)?;

    let sql_query = "SELECT count FROM presence_counters WHERE name = 'group_sittings' ORDER BY updated_at DESC LIMIT 1";
    let row = conn
        .query_one(sql_query, &[])
        .await
        .map_err(internal_error)?;
    let count: i32 = row.try_get(0).map_err(internal_error)?;

    Ok(count.to_string())
}

/// Utility function for mapping any error into a `500 Internal Server Error`
/// response.
fn internal_error<E>(err: E) -> (StatusCode, String)
where
    E: std::error::Error,
{
    (StatusCode::INTERNAL_SERVER_ERROR, err.to_string())
}

The benchmarks were ran on a VM Falkenstein region.

tinvo@mangala:~$ rewrk -c 50 -t 2 -d 30s -h http://135.181.179.195:2222/
Beginning round 1...
Benchmarking 50 connections @ http://135.181.179.195:2222/ for 30 second(s)
  Latencies:
    Avg      Stdev    Min      Max      
    25.01ms  0.72ms   24.89ms  50.20ms  
  Requests:
    Total:  59910  Req/Sec: 1996.90
  Transfer:
    Total: 10.51 MB Transfer Rate: 358.81 KB/Sec

tinvo@mangala:~$ rewrk -c 50 -t 2 -d 30s -h http://135.181.179.195:3000/
Beginning round 1...
Benchmarking 50 connections @ http://135.181.179.195:3000/ for 30 second(s)
  Latencies:
    Avg      Stdev    Min      Max      
    25.01ms  0.04ms   24.90ms  26.58ms  
  Requests:
    Total:  59908  Req/Sec: 1996.87
  Transfer:
    Total: 7.37 MB Transfer Rate: 251.56 KB/Sec

tinvo@mangala:~$ rewrk -c 50 -t 2 -d 30s -h http://135.181.179.195:2222/count
Beginning round 1...
Benchmarking 50 connections @ http://135.181.179.195:2222/count for 30 second(s)
  Latencies:
    Avg      Stdev    Min      Max      
    25.15ms  0.92ms   24.99ms  52.30ms  
  Requests:
    Total:  59569  Req/Sec: 1985.57
  Transfer:
    Total: 9.83 MB Transfer Rate: 335.48 KB/Sec

tinvo@mangala:~$ rewrk -c 50 -t 2 -d 30s -h http://135.181.179.195:3000/count
Beginning round 1...
Benchmarking 50 connections @ http://135.181.179.195:3000/count for 30 second(s)
  Latencies:
    Avg      Stdev    Min      Max      
    25.25ms  0.08ms   25.06ms  26.49ms  
  Requests:
    Total:  59350  Req/Sec: 1978.26
  Transfer:
    Total: 6.62 MB Transfer Rate: 226.03 KB/Sec

It's not much; but you can still see that nginx + lua is still faster than Rust. You have root on dev.server.dhamma.org; you are welcome to replicate it and test the benchmark on your client.

[vtt]

Forgot to add how the rust was ran:

tinvo@dev:~/framework-test-at-home/rust_axum_tokio-postgres-bb8$ cargo run --release
    Finished release [optimized] target(s) in 0.05s
     Running `target/release/rust_axum_tokio-postgres-bb8`

[vtt]

OpenResty is actually nginx + lua module; it is not an nginx fork; it is using latest stable nginx and add its lua module. There is another postgres module for nginx; I'm going to benchmark that for fun as well and see how it fares with lua and rust.

[vtt]

Here is the new nginx configuration to include postgres module at /count2 location:

worker_processes  auto;
error_log logs/error.log;
events {
  worker_connections 1024;
}
http {
  upstream database {
    postgres_server  127.0.0.1 dbname=portal_edge user=counter password=counter;
  }

  server {
    listen 2222;
    access_log off;
    location / {
      default_type text/html;
      content_by_lua_block {
        ngx.say("Hello world!")
      }
    }

    location /count2 {
      postgres_pass   database;
      postgres_query  "SELECT count FROM presence_counters WHERE name = 'group_sittings' ORDER BY updated_at DESC LIMIT 1";
      postgres_output text;
    }

    location /count {
      default_type text/html;
      content_by_lua_block {
        local pgmoon = require("pgmoon")
        local pg = pgmoon.new({
          host     = "127.0.0.1",
          port     = "5432",
          user     = "counter",
          password = "counter",
          database = "portal_edge"
        })
        assert(pg:connect())
        local res = assert(pg:query("SELECT count FROM presence_counters WHERE name = 'group_sittings' ORDER BY updated_at DESC LIMIT 1"))
        pg:keepalive()
        pg = nil
        ngx.say(res[1]["count"])
      }
    }
  }
}

And the benchmarks:

tinvo@mangala:~$ rewrk -c 50 -t 2 -d 30s -h http://135.181.179.195:2222/count2
Beginning round 1...
Benchmarking 50 connections @ http://135.181.179.195:2222/count2 for 30 second(s)
  Latencies:
    Avg      Stdev    Min      Max      
    25.13ms  0.78ms   24.96ms  50.25ms  
  Requests:
    Total:  59630  Req/Sec: 1987.59
  Transfer:
    Total: 9.84 MB Transfer Rate: 335.79 KB/Sec

tinvo@mangala:~$ rewrk -c 50 -t 2 -d 30s -h http://135.181.179.195:2222/count 
Beginning round 1...
Benchmarking 50 connections @ http://135.181.179.195:2222/count for 30 second(s)
  Latencies:
    Avg      Stdev    Min      Max      
    25.19ms  1.49ms   24.94ms  71.72ms  
  Requests:
    Total:  59494  Req/Sec: 1983.05
  Transfer:
    Total: 9.84 MB Transfer Rate: 335.70 KB/Sec

tinvo@mangala:~$ rewrk -c 50 -t 2 -d 30s -h http://135.181.179.195:3000/count
Beginning round 1...
Benchmarking 50 connections @ http://135.181.179.195:3000/count for 30 second(s)
  Latencies:
    Avg      Stdev    Min      Max      
    25.26ms  0.19ms   25.05ms  31.87ms  
  Requests:
    Total:  59300  Req/Sec: 1976.60
  Transfer:
    Total: 6.62 MB Transfer Rate: 225.84 KB/Sec

Tiny improvement of postgres over lua; but Lua is more general purpose and more powerful.

Also I just noticed nginx output is much bigger in terms of bytes compared to Rust; nginx is returning more HTTP headers so the output is about 30% bigger.

[vtt]

FYI, testing from different data center will have latency (in this case about 25ms); so the benchmark number does not truly reflect the cost of instrumentation and internalization of these stacks. When I run the tests locally, here are the results:

tinvo@dev:~$ ./.cargo/bin/rewrk -c 50 -t 2 -d 30s -h http://localhost:2222/count
Beginning round 1...
Benchmarking 50 connections @ http://localhost:2222/count for 30 second(s)
  Latencies:
    Avg      Stdev    Min      Max      
    0.61ms   0.44ms   0.08ms   27.59ms  
  Requests:
    Total: 2459208 Req/Sec: 81972.04
  Transfer:
    Total: 412.35 MB Transfer Rate: 13.74 MB/Sec

tinvo@dev:~$ ./.cargo/bin/rewrk -c 50 -t 2 -d 30s -h http://localhost:2222/count2
Beginning round 1...
Benchmarking 50 connections @ http://localhost:2222/count2 for 30 second(s)
  Latencies:
    Avg      Stdev    Min      Max      
    0.55ms   0.58ms   0.08ms   25.84ms  
  Requests:
    Total: 2746213 Req/Sec: 91537.43
  Transfer:
    Total: 458.37 MB Transfer Rate: 15.28 MB/Sec

tinvo@dev:~$ ./.cargo/bin/rewrk -c 50 -t 2 -d 30s -h http://localhost:3000/count
Beginning round 1...
Benchmarking 50 connections @ http://localhost:3000/count for 30 second(s)
  Latencies:
    Avg      Stdev    Min      Max      
    0.72ms   0.21ms   0.19ms   5.38ms   
  Requests:
    Total: 2068225 Req/Sec: 68939.64
  Transfer:
    Total: 230.77 MB Transfer Rate: 7.69 MB/Sec

So you can clearly see the postgres module is 11% faster than lua module, and lua module is 19% faster than Rust

[greg-nagy]

I tried to replicate your measurements but ran into some issues.

• The test runs showed big variability between runs
• You have only tested a single implementation in rust
• The lua implementation had a strange behavior when they were tested after one another

So I have put in some work, to make the tests more robust and possible to reproduce.

• I have created a script to create the binaries
• I have scripted the test runs
• I have also reintroduced an optimization that I have dropped previously. (It didn't make sense in a single cpu environment but it did here)

You can see the detailed results on the edge tab in the spreadsheet

Everything is up in this repo, if you want to rerun the tests. (It would be good to test this measurement method to see if it's robust enough)

[vtt]

Thanks for the update; I noticed the big variability between runs too - depending on how busy the instance is (quite a bit of stuffs running on it).

The optimized Rust implementations using pre-forked workers are very impressive; Rust can be almost as fast as C; the nginx postgres module seems disappointing - not sure why it's even losing to Lua; my past tests showed postgres consistently beating Lua.

I'd be very interested to see the performance of the optimized Rust pre-forked implementations behind nginx reversed proxy. This is the real world usage since most of the time, nginx is prerequisite for the existing app (ie: Rails).

That comparison would show where the bottleneck is. Is it Lua? Or is it the overhead of full fledge http1/2 web server (nginx) vs basic tcp socket.