Load testing failure

[potatosalad]

First reported by @Gazler in phoenixframework/phoenix#2621.

Would you also be able to advise how to run the tester? I got this from my test (by the way how do you make the pretty charts from your output?!):

https://gist.github.com/Gazler/d8f7371e2fbabf5d2fac491e2ccd6464

However it is failing on:

 GEN    cowboy2-h1-test
[cowboy 2.x] load testing cowboy (h1) for 5s
Error response from daemon: Container 89171e9381fedb5e7ce3a602a5e78c504650db0cd6863125f4cb454c9fa90505 is not running
Makefile:108: recipe for target 'cowboy2-h1-test' failed
make: *** [cowboy2-h1-test] Error 1

The tests in (cowboy2-h2-test) work perfectly. make all implies that running make should work for testing everything.

[potatosalad]

@Gazler First off, I have no idea why cowboy2-h2-test would work while cowboy2-h1-test would not. Both tests use the exact same docker container.

So... ¯\_(ツ)_/¯

Secondly, you can manually run the tests using priv/test.sh directly on your machine (without using docker).

In one terminal, start the server:

$ MIX_ENV=prod PORT=29595 elixir --erl "+K true" -S mix phx.server

If this is cowboy2 mode (if COWBOY_VERSION=1 isn't specified), the following services will be available:

• 29591 — h2o (only starts if H2O_VERSION=2 is set)
• 29592 — Ace (only starts if COWBOY_VERSION is undefined; only supports TLS mode)
• 29593 — cowboy
• 29594 — plug
• 29595 — phoenix

So, to test cowboy, for example, you would run in another terminal:

$ priv/test.sh 60s 5s h2 "http://127.0.0.1:29593/"

The arguments format is:

$ priv/test.sh DURATION WARM_UP_TIME H1_OR_H2 ENDPOINT

[potatosalad]

by the way how do you make the pretty charts from your output?!

@Gazler I missed this question initially. The answer is: it's super clunky.

I mentioned really briefly in the cowboy 2 pull request how the graphs are created, but you will probably need to modify priv/h2load.escript manually to have it work with your timezone:

Explanation of Process

To repeat the graphical load tests, you'll need to clone potatosalad/phoenix_h2load and build a custom version of h2load from potatosalad/nghttp2@cowboy2:

git clone https://github.com/potatosalad/nghttp2.git
cd nghttp2
git checkout cowboy2
git submodule update --init
autoreconf -i
automake
autoconf
./configure --enable-app --disable-silent-rules --disable-python-bindings --enable-asio-lib --with-boost --without-jemalloc
make -j8

Note: You might have to adjust the configure options based on your system. For example, on macOS with homebrew, I had to use --with-boost=$(brew --prefix boost).

You should end up with an executable located at src/h2load where you can run one of the following:

# For testing cowboy 1.x
export COWBOY_VERSION=1
export MIX_ENV=prod
export PORT=29595
mix do deps.get, deps.compile, compile
elixir --erl "+K true" --name phoenix_h2load@127.0.0.1 --cookie mycookie -S mix phx.server

In a separate terminal, the separate stacks can be load tested:

cd priv
export H2LOAD_BIN="~/path/to/nghttp2/src/h2load"
./h2load.sh $H2LOAD_BIN c1_h1_cowboy h1 29593
./h2load.sh $H2LOAD_BIN c1_h1_plug h1 29594
./h2load.sh $H2LOAD_BIN c1_h1_phoenix h1 29595

The cowboy 2.x tests are identical, with the exception of the removal of the export COWBOY_VERSION=1 line above. Also, the h1 can be replaced with h2 to perform HTTP/2 load testing.

It uses the ancient and fantastically difficult to work with RRDtool to generate the graphs, but I'd love to find something a little easier to work with if you know of something. I wound up using it due to the time-series nature of the data that needs to be graphed.

The scripts output a bash script that shows the actual command run to generate the SVG image (for example, priv/c2_h2_cowboy.req.sh). I'll then manually take the generated time offsets and inputs and manually make a comparison graph script (for example, priv/c2_h2_h2o.sh). I'll then shove them through SVGOMG to minify them.

If you happen to know of a better way to get that done on the command line, I'm all for it. I kind of hate the current solution I've come up with 😬

[Gazler]

EDIT Discard everything below because timezones. I hadn't set the correct value in the escript.

@potatosalad Thanks for the super detailed instructions. I'm getting somewhere with this. After running the script though, all my SVGs come out looking like:

Here's an example RRD:

c1_h1_cowboy.rrd.gz

I tried adjusting the axis so it'd go up to 300k btw, but that didn't seem to change anything,

[Gazler]

Success!

[Gazler]

[Gazler]

I applied the changes from this commit, but it doesn't seem to have made a massive impact:

https://github.com/Gazler/plug/commit/524cd970984e61dcb56739fdfceba4465ce74eee

[potatosalad]

@Gazler Awesome! Yeah, rrdtool is a little finicky to work with, especially with time zones. The SHIFT amount can also be adjusted by subtracting 1 from the numbers which will close the gap on the left, but that's the only additional thing I've done with mine. There's probably some way to get similar results that are easier to play with using R, but I'm no R expert.

In my original load tests back in August, I saw a ~40% performance increase by bypassing cowboy_stream_h and implementing the cowboy_stream behaviour in Contention.StreamHandler. I think this is mostly due to the bypassed version not spawning an additional child process per stream and instead processing the request under the same PID as the connection itself (similar to how cowboy 1.x handles things). There might be some room there for experimentation that might yield better results.

[Gazler]

The Plug.RequestId plug is the cause of a lot of the slowdown with Phoenix. I'll raise an issue over on Plug. Thanks for this tool @potatosalad - wouldn't have found it so easily without it! :)