[DNM] Run targeted apps into containers

[waghanza]

Hi,

This PR run all frameworks into docker containers.

In order to build and run :

shards install && shards build --release --no-debug # <- build all tools
bin/neph # <- build and run all containers
bin/benchmarker # <- run benchmarks

@tbrand I have disabled crillo, since only on OS X.

Regards,

[OvermindDL1]

@tbrand I have disabled crillo, since only on OS X.

Who on earth runs Mac's as servers?!? o.O

bin/neph # <- build and run all containers

OgodScreenSpammingTerminal!

Why doesn't it move the cursor to update values as necessary? Right now it is spamming a very large screen very very quickly.... My poor terminal buffer over a remote SSH connection... that is painful... :-(

However, it seems to actually do nothing, yet docker is eating a few cores worth of CPU, however they all remain at 0%, been waiting over 10 minutes now...

And after 15 minutes it finally states:

[Error] 'perfect' failed with status code (256)
[Error]  -- STDOUT(.neph/perfect/log/log.out) -- 

[Error]  -- STDERR(.neph/perfect/log/log.err) -- 
d174e0623e4b: Verifying Checksum
d174e0623e4b: Download complete
1a8b3c87dba3: Pull complete
91390a1c435a: Pull complete
07844b14977e: Pull complete
b78396653dae: Pull complete
d174e0623e4b: Pull complete
69ea49c72ebc: Pull complete
Digest: sha256:3b0aecbdccf47692cb701c9a2fa031280f11c0aad7667fdf2485e92422a8aff8
Cannot overwrite digest sha256:3b0aecbdccf47692cb701c9a2fa031280f11c0aad7667fdf2485e92422a8aff8

log.out is empty, but log.err contains:

Sending build context to Docker daemon 8.704 kB^M^M
Step 1/8 : FROM swift
latest: Pulling from library/swift
22dc81ace0ea: Pulling fs layer
1a8b3c87dba3: Pulling fs layer
91390a1c435a: Pulling fs layer
07844b14977e: Pulling fs layer
b78396653dae: Pulling fs layer
d174e0623e4b: Pulling fs layer
69ea49c72ebc: Pulling fs layer
69ea49c72ebc: Waiting
91390a1c435a: Waiting
b78396653dae: Waiting
22dc81ace0ea: Waiting
07844b14977e: Waiting
d174e0623e4b: Waiting
1a8b3c87dba3: Waiting
1a8b3c87dba3: Verifying Checksum
1a8b3c87dba3: Download complete
22dc81ace0ea: Download complete
91390a1c435a: Verifying Checksum
91390a1c435a: Download complete
07844b14977e: Verifying Checksum
07844b14977e: Download complete
b78396653dae: Verifying Checksum
b78396653dae: Download complete
22dc81ace0ea: Pull complete
69ea49c72ebc: Verifying Checksum
69ea49c72ebc: Download complete
d174e0623e4b: Verifying Checksum
d174e0623e4b: Download complete
1a8b3c87dba3: Pull complete
91390a1c435a: Pull complete
07844b14977e: Pull complete
b78396653dae: Pull complete
d174e0623e4b: Pull complete
69ea49c72ebc: Pull complete
Digest: sha256:3b0aecbdccf47692cb701c9a2fa031280f11c0aad7667fdf2485e92422a8aff8
Cannot overwrite digest sha256:3b0aecbdccf47692cb701c9a2fa031280f11c0aad7667fdf2485e92422a8aff8

So it seems a bit... buggy? ^.^;

[OvermindDL1]

Oooo I see what is going on. It's compiling serially. Instead of it compiling them all across many cores it is instead only compiling one thing at a time (which doesn't match with what is being displayed on the screen that implies they are all in parallel). Where is this bug at? It is only using 1 core of CPU power to compile these things...

[OvermindDL1]

...until C++ and swift and rust started compiling, then all cores activated, because they can compile in parallel... Eyup 1600% cpu usage on those, always nice.

[OvermindDL1]

So yeah, it seems that single-core compilers (like crystal and ruby and other such poor languages) should probably all be compiled in parallel where multi-core compilers should run in serial...

[OvermindDL1]

Still though, the neph thing is not changing from 0% even though the logs show that the docker compilation has significantly progressed...

[OvermindDL1]

Also, if make cpp is run before bin/neph then the C++ cache directory will be wrong, you should add a rm -rf cpp/evhtp/_builds or so as the first command just to make sure that will be properly cleared out (or make it build into a different cache directory, which would probably be better for faster repeated builds).

[OvermindDL1]

And the rust ones are failing because rust nightly is not being used (why not compile on the host then just run that in docker via the local binary, there is no point to compile these inside docker, that just wastes massive amounts of space).

[waghanza]

@OvermindDL1 sorry you have trouble. I had to warn you before effectively, an intensive usage of docker COULD freeze a workstation ;-) for test, I run

make clean && shards install && shards build --release --no-debug && bin/neph && time bin/benchmarker --record --requests=100

and it is impossible to use any app during the build :warning: but for me this stop a while after

just after build (~20m) and run, I remove all containers :

docker ps -aq | xargs docker rm -f
docker images -aq | xargs docker rmi -f

PS : @OvermindDL1 you can specify a DOCKER_HOST as environment variable to target a remote docker server

[waghanza]

@OvermindDL1 using docker is only a first step, since I'm targeted using a cloud provider to compute results :stuck_out_tongue_winking_eye:

[OvermindDL1]

an intensive usage of docker COULD freeze a workstation ;-)

Well it's an new empty multi-core server, so... ;-)

(Who would benchmark on a workstation?!)

and it is impossible to use any app during the build warning but for me this stop a while after

I'm... not sure what you are referencing?

PS : @OvermindDL1 you can specify a DOCKER_HOST as environment variable to target a remote docker server

It is the dedicated server it is already running on (honestly don't even need docker, that just adds useless overhead).

@OvermindDL1 using docker is only a first step, since I'm targeted using a cloud provider to compute results stuck_out_tongue_winking_eye

Using cloud providers does not seem terribly smart if one is wanting reliable results, they incur a fluctuating overhead that would make such tests unreliable, so I'm not sure what the purpose of wanting to do that would be?

[waghanza]

an intensive usage of docker COULD freeze a workstation ;-)

Well it's an new empty multi-core server, so... ;-)

my quad-core does not seems to freeze any more -> needs to investigate

Using cloud providers does not seem terribly smart if one is wanting reliable results, they incur a fluctuating overhead that would make such tests unreliable, so I'm not sure what the purpose of wanting to do that would be?

sure, we had argue about that. a full dedicated / bare-metal server is a solution. however, with cloud adoption in companies, it seems accurate to deliver results in cloud, like TFB is done on dedicated and cloud (azure)

[tbrand]

@OvermindDL1

Why doesn't it move the cursor to update values as necessary?

You can set --mode=CI option to prevent it.

[waghanza]

as I have mentioned, TFB run on physical hardware and cloud (azure).

For me this is a very good approach, we could run which_is_the_fastest on docker, cloud or dedicated :stuck_out_tongue:

[OvermindDL1]

sure, we had argue about that. a full dedicated / bare-metal server is a solution. however, with cloud adoption in companies, it seems accurate to deliver results in cloud, like TFB is done on dedicated and cloud (azure)

Delivering to the cloud is fine, but testing in the cloud adds randomness to otherwise stable tests, and as you are wanting to benchmark the servers and not wanting to benchmark the randomness of the cloud servers, then... ;-)

You can set --mode=CI option to prevent it.

If this fixes it then this really needs to be default when an interactive terminal is detected on stdin! ^.^;

For me this is a very good approach, we could run which_is_the_fastest on docker, cloud or dedicated stuck_out_tongue

Well I have a good dedicated I can use for it. It's mostly used for CI work so it's fairly idle when not building something. :-)

[waghanza]

@tbrand @OvermindDL1 on my workstation (Core i5 quad-core), I have (in about 16h) some totally different results (than thos in README.md)

Language (Runtime)	Framework (Middleware)	Max [sec]	Min [sec]	Ave [sec]
ruby	rails	2108.742953	2004.721948	2060.131203
ruby	sinatra	360.827634	346.042592	353.378847
ruby	roda	123.527559	121.024325	121.581133
ruby	rack-routing	173.125427	164.716904	166.488733
crystal	kemal	30.243713	30.039599	30.132498
crystal	router.cr	24.508544	23.806082	24.249192
crystal	raze	24.683095	24.366317	24.482483
crystal	lucky	27.881339	26.940995	27.557054
crystal	amber	29.397903	28.413274	29.132793
crystal	spider-gazelle	25.964635	25.090902	25.749765
go	echo	24.320525	24.148351	24.226269
go	gorilla_mux	24.203060	23.606375	23.980805
go	iris	24.380352	24.124813	24.228804
go	fasthttprouter	23.292494	22.735249	22.995820
go	gin	59.772351	46.903066	54.841485
rust	actix	19.565631	19.296647	19.433929
rust	iron	20.317056	20.024369	20.144105
rust	nickel	21.047431	20.700917	20.820582
rust	rocket	24.966503	24.554333	24.741285
node	express	88.503014	86.857210	87.254594
node	clusterexpress	41.945097	39.872721	40.367525
node	polka	54.691880	52.998575	53.351919
node	clusterpolka	25.990100	24.098153	24.835647
elixir	plug	56.776633	53.875485	55.269400
elixir	phoenix	48.017713	46.805425	47.317403
swift	vapor	4201.599323	4200.607891	4200.962846
swift	perfect	27.657632	26.682522	27.137907
swift	kitura	174.009430	42.869625	93.882398
scala	akkahttp	37.378097	35.833233	36.472551
python	sanic	77.881731	46.727616	56.330798
python	japronto	23.857229	23.061519	23.606308
python	flask	1336.259429	1289.386087	1304.090857
python	django	1555.140515	1493.452898	1516.118073
python	tornado	714.357536	712.207327	713.373401
nim	jester	248.239479	247.157360	247.523019
nim	mofuw	65.357445	46.420157	50.497525

[OvermindDL1]

@tbrand @OvermindDL1 on my workstation (Core i5 quad-core), I have (in about 16h) some totally different results (than thos in README.md)

A couple of reasons:

• You are using the included crystal-based benchmarker, which is not at all accurate as the tests are throttled by the crystal benchmarking client and not the servers (watch your CPU usage and IO count, most servers barely even notice the absolute piddly amount of data that the crystal benchmarker gives them).
• Working over docker means going over the TCP stack in the kernel in full instead of local TCP sockets, this will incur a latency that would not exist otherwise (though it is consistent).

I really should PR my simple stats script in for now (it's not pretty code as it kind of 'grew' to what it is now instead of being properly designed, but it works well for me) as you will get significantly more accurate results... >.>

[OvermindDL1]

For note, even I get very different results with the crystal benchmarker than both you and the readme with very different ordering of the servers (and crystal is not on top locally here either like you, and unlike the readme). It is a very unreliable and absolutely should not be used.

Also wow! Why is Swift that slow on yours?!? o.O

/me still needs to figure out how to install swift on linux again...

EDIT: And Obj-C for that matter, I've never used either of them, or even ever seen a project that uses either of them... >.>

[waghanza]

A couple of reasons:

You are using the included crystal-based benchmarker, which is not at all accurate as the tests are throttled by the crystal benchmarking client and not the servers (watch your CPU usage and IO count, most servers barely even notice the absolute piddly amount of data that the crystal benchmarker gives them).
Working over docker means going over the TCP stack in the kernel in full instead of local TCP sockets, this will incur a latency that would not exist otherwise (though it is consistent).

I really should PR my simple stats script in for now (it's not pretty code as it kind of 'grew' to what it is now instead of being properly designed, but it works well for me) as you will get significantly more accurate results... >.>

I agree, but I do not think docker and the crystal based benchmarker are the only reasons

Feel free to PR your script with wrk, even code is not clean, I'll review it

Also wow! Why is Swift that slow on yours?!? o.O

Swift docker is based on ubuntu (not accurate as a server), the purpose is to test on either centos or debian for the cloud (or even freebsd) :stuck_out_tongue:

[OvermindDL1]

I agree, but I do not think docker and the crystal based benchmarker are the only reasons

The crystal based benchmarker I'd say is 80% of the fluctuation reasons and why tests are not reliable or consistent.

Swift docker is based on ubuntu (not accurate as a server), the purpose is to test on either centos or debian for the cloud (or even freebsd) stuck_out_tongue

Well I have a dedicated 8/16 core with 16gigs ram (soon 32, but no server is hitting that anyway) here to test on, no docker overhead even needed. ;-)

Feel free to PR your script with wrk, even code is not clean, I'll review it

Yeah, let me go ahead and add a --help menu (and default show it if no options given) and PR it in. It's not pretty, but it works well at least. It will require both elixir and wrk to be installed. I usually write such things in shell scripts but I wanted to see how well elixir writes command-line apps (not as bad as I expected actually). I'm really tempted to rewrite it as a shell script though just because I know them so much better... >.>

[waghanza]

Yeah, let me go ahead and add a --help menu (and default show it if no options given) and PR it in. It's not pretty, but it works well at least. It will require both elixir and wrk to be installed

I do not understand. Is your script made in elixir ? It'll be accurate to rewrite it into crystal (I can take this part) to keep a consistency in this project

[OvermindDL1]

I do not understand. Is your script made in elixir ? It'll be accurate to rewrite it into crystal (I can take this part) to keep a consistency in this project

Correct, it's in Elixir (as stated, I was playing around to see how well elixir works for command-line scripts). Crystal would be fine for this part as it's not needing multiple threads (wrk handles that part) but honestly a shell script should be best, it would be significantly easier to parse out the wrk information and run the tests anyway as wrk was designed for such usage, plus then you wouldn't need to compile anything on the local system if you happen to be running things over ssh for example.

I do use a shell script part to start and stop the actual servers as it will ensure that they will always die, regardless of if the benchmarker is killed or not (unlike currently where killing the current crystal benchmarker while it is running will leave some servers running unkilled, it is quite irritating). I'm unsure how well crystal can respond to kill signals (elixir can't really for example, hence the shell script) so it may or may not be good to use.

At a higher level though, Crystal really needs to stop being used for the tooling, it is a single-core-only language so it is not at all even remotely good to use for anything but simple single-core shell scripts without slaving out many processes (which at that point why use crystal).

[waghanza]

@OvermindDL1 I understand your points This is not a place to run a battle elixir vs crystal. I think both languages have their pro / cons.

[OvermindDL1]

This is not a place to run a battle elixir vs crystal. I think both languages have their pro / cons.

This is definitely not the use-case for Elixir for sure! Elixir is designed for long-running systems that never go down, so writing a shell script in it is... odd, but I was curious how it was. ^.^;

I'd opt entirely for a standard sh shell script to manage wrk though.

[waghanza]

I'd opt entirely for a standard sh shell script to manage wrk though. I prefer crystal or any programming language. It could be useful to puslish results in HTML (or else) ... to add more feature

[OvermindDL1]

Shell scripts are a programming language, at the same power as any other normal dynamically typed language, except that the interpreter already exists on about every computer out, while the syntax and functionality is specifically optimized for work just like this, don't discount them that fast. Proper tool for the proper job after all (otherwise I'd just use C++ for everything).

[tbrand]

@waghanza Hey, thanks again for this. I'm trying your branch, but i don't understand why you launch the container when I run neph.

For me, I want to keep clean my environment. So, for ideal, they sould be launched when I try to get the result of them. And after taking the results, clean the docker containers.(stop and remove)

Is there any reasons for launching them at neph?

[tbrand]

I mean

• just build container when run neph.
• start running a named container before taking the result.
• stop and remove them after recording it.

[waghanza]

@tbrand I see you point. There is not task for removing containers (I made it by hand).

My purpose my running with neph is to create think in // -> not waiting behind my screen ^^

[tbrand]

But each container consume resources. That affect for the performace right?

[tbrand]

stop and remove them after recording it.

Oops, not removing the containers but just stopping them. If we remove it, we have to rebuild them... 😱

[waghanza]

@tbrand ok. I see, There is 3 steps :

• build
• run
• stop

The 2 first are made by neph (in order that crystal could get the IP to benchmark) and the containers has to run to get the IP

Actual benchmarker get IP from running container, so I think I have to add some code to run from benchmarker (no neph)

[tbrand]

@waghanza

Actual library, docker.cr could only get IP from running container (not only created one), so I think I have to add some code to run from benchmarker (no neph)

I see. Running it by command line it acceptable for me.

# Execcute container
`docker run -d --name=...`

[waghanza]

I prefer no to ... if not finding how to, I'll PR this ;-)

[tbrand]

Awesome.

As another problem, crystal frameworks (kemal, raze, ...) cannot be successfully compiled since pulling image is too old. Specifying the image version by

FROM crystallang/crystal:0.24.2

solve the problem for me.

[waghanza]

FROM crystallang/crystal:0.24.2

I did not notice that.

@jhass @waj @will @matiasgarciaisaia The default tag used (latest) is too old

@see https://hub.docker.com/r/crystallang/crystal/tags/

[tbrand]

Use nightly? lol

[waghanza]

I can, but it could break everything :wink:

[tbrand]

Agree. Specify the version for now.

[waghanza]

I have an issue removing run step from neph for nodejs.

For example, polka and clusterpolka could not share the same Dockerfile but the source code reside in the same folder.

I'll have to find a workaround

[matiasgarciaisaia]

@waghanza I've just updated Crystal's latest tag to be 0.24.2 - let me know if it didn't work. And thanks for the ping :)

[waghanza]

thanks @matiasgarciaisaia, it works

[waghanza]

@tbrand Have you tried ?

[tbrand]

Yeah I've tried once but it's not latest one. (may be around #50b370b) I got an error context deadline exceeded then.

I think too many docker build made this error. Because when I build each language separately, the error had disappeared.

Can we build them sequentially for each language? Or do you have any ideas for fixing it?

I mean like this.

- ruby
  - rails
    - sinatora
      - ...
- crystal
   - kemal
      - amber
         - ...
...

[waghanza]

Hi @tbrand,

We could be containers sequentially (without neph but with make).

I did not anticipate those errors (docker freeze the workstation, or there is sha256 conflicts ... due to parallel container creation). @OvermindDL1 and you (even travis fails sometimes) but build was working for me and https://travis-ci.org/waghanza/which_is_the_fastest.

If you want to create containers sequentially, we have to use make (I can work on it :stuck_out_tongue:), neph will be used when creating on the cloud.

Ok for you ?

Regards,

[tbrand]

Hmm, OK.

neph is not good at executing jobs sequentially for now. I've opened the issue about it. here

So please fix Makefile for building them sequentially? Sorry for the duplicated works... :pray:

[waghanza]

done

Perhaps, we also SHOULD redesign framework listing, to avoid maintaining (by hand)

• Makefile
• neph.yaml

[tbrand]

Thanks! Im out without any PCs, so let me check it tomorrow.

[waghanza]

sure, I'll continue working / testing on my branch

[OvermindDL1]

Technically CMake could be used to do this, at which point it can use any system to do the rest of the work, of which I've never seen anything faster than ninja (I so love using ninja as a build system, it is SO fast!). ^.^

/me is perfectly fine with make files or neph, whichever, though wish neph was system standard, and since it's not would prefer make overall as it actually is standard

[tbrand]

@waghanza

For rails image, I got

Sending build context to Docker daemon   24.9GB

Obviously it's too big. I was digging for it and finally found that it also copy the log file (production.log) to the image. So please fix here

Other images have built successfully. 💯