search

TechEmpower / FrameworkBenchmarks

Source for the TechEmpower Framework Benchmarks project
https://www.techempower.com/benchmarks/
Other
7.59k stars 1.94k forks source link

Logs not capturing all framework output #1504

Closed hamiltont closed 9 years ago

hamiltont commented 9 years ago

This is a pretty critical one. When you fork a process (like we do at the end of every setup.sh) there is a 4KB output buffer added to that forked process. If you SIGKILL a process (like we do to stop all frameworks), then that output buffer is destroyed without being flushed. This means that we are not capturing this output in any log files. Some examples using the R10 final peak results:

lwan output ends with [100%] Built target techempower, but if you run it manually you'll see stdout happening after the build stage e.g. 

Loading configuration file: techempower.conf.
Using 32 threads, maximum 2000 sockets per thread.
Listening on http://0.0.0.0:8080.
TCP_FASTOPEN not supported by the kernel.
Ready to serve.

Similarly for haywire, duda, and multiple others - we are losing small bits of output that are probably very critical to the framework devs when trying to debug their system (e.g. TCP_FASTOPEN not supported by the kernel. would probbaly be a pretty important message if this wasn't output from travis-CI).

Granted, this is a small amount of output - no framework I could find is sending a lot of information to stdout - but the information they do send is likely critical to successful debugging.

methane commented 9 years ago

In common case, stdout is block buffered and stderr is line buffered (or non buffered) when they are not terminal (piped). So I think they should use stderr instead of stdout.

methane commented 9 years ago

I'm sorry, I hadn't read #1505.

msmith-techempower commented 9 years ago

One option here is to send a SIGTERM first, which gives applications time to properly shut down, and then after ~5s send the SIGKILL just to be sure.