Performance Cowboy 2 vs Bandit Phoenix adapter

[svsool]

👋 Hello Mat!

I did a quick load test on an MBP M1 (elixir 1.14.0-otp-25 / erlang 25.0.4) with one Erlang scheduler online (+S = 1 in VM args) using k6, and wanted to share some results.

TL;DR: RPS seems to be lower (18k vs 23k) with Bandit vs Cowboy 2.7 and elixir 1.14.0-otp-25 / erlang 25.0.4, and similar (25k vs 24k) with elixir 1.17.1-otp-27 / erlang 27.0.

Phoenix with Cowboy Adapter (2 x runs)

Phoenix with Bandit Adapter (2 x runs)

Steps to reproduce:

• download original setup (generated using mix phx.new <app_name> --adapter <cowboy | bandit> + logging disabled and production release compiled)
  • phx_cowboy_test.zip
  • phx_bandit_test.zip
• run MIX_ENV=prod mix do deps.get, compile, release
• run PHX_SERVER=true SECRET_KEY_BASE=random PORT=4000 _build/prod/rel/<app_name>/bin/<app_name> start
• install k6 and run k6 run echo.test.js
  • test can be found here

I also tried setups from this repo (related post) except response modified to match original test, sleep removed and number of schedulers set to 1.

A bare Cowboy 1.x setup gave 40k+ RPS on a single core using the same echo test.

And with Cowboy 2.7 from that same repo.

With Bandit 1.5 + Plug 1.16, no Phoenix (bandit_test.zip)

Also tried to run Bandit's own benchmarks locally using mix benchmark cowboy bandit@main, but unfortunately got the following error:

Let me know if additional tweaking required to achieve better results with Bandit vs Cowboy and whether results are relatively comparable when reproduced using your hardware.

Thanks!

[svsool]

Did re-run tests with latest elixir 1.17.1-otp-27 / erlang 27.0, and with the same synthetic test Bandit performs slightly better than Cowboy 2.7 with http 1.1 when used as Phoenix adapter, and better than Cowboy 2.7 with a standalone setup.

Seems standalone Cowboy 1.x performs worse with the latest elixir / erlang build vs elixir 1.14.0-otp-25 / erlang 25.0.4 build overall, but better than standalone Bandit.

Phoenix with Cowboy Adapter

Phoenix with Bandit Adapter

With Bandit 1.5 + Plug 1.16, no Phoenix

With Cowboy 1.x

With Cowboy 2.7

[svsool]

Feel free to re-open if going to re-test and compare relative numbers yourself or if there are any other possible improvements. 👋

[mtrudel]

Thanks for the report! I'd not seen k6 before but it looks like exactly the client-side tool I'd wanted access to when I wrote the benchmark library (the client implementation there is pretty sad). What do you think would be involved in moving the mtrudel/benchmark suite over to use k6 on the client side?

[svsool]

hi @mtrudel, k6 from my experience is relatively easy to set up, and there are examples on running it with github actions (see related k6-action), what comes to test cases which bandit benchmark suit requires you'd probably know the details the best.