Broken Frameworks

[NateBrady23]

The following frameworks are not passing Travis in one or both of Round 13 (Master) and our Round 14 target branch. There could be a number of reasons. The goal is to get all frameworks included in the current benchmarks to pass both Travis and local tests. We are considering removing frameworks that are not passing and not being maintained. This would speed up Travis runs and give more accurate snapshots for pull requests.

:heavy_multiplication_x: means the Framework is not passing Travis for that round. :heavy_check_mark:, it has. Local refers to pulling and testing the Framework in its most recent contribution in a vagrant environment. A lot of these issues stem from dependency problems, while others are strictly Travis problems. Some of the Frameworks pass some of the tests. (A lot of the Ruby frameworks are failing on jruby tests). So like I said, there are many different reasons a Framework might appear on this list.

If we can get these all to :heavy_check_mark:, we will be able to better identify when a Framework breaks and if a toolset change may have caused it.

Please feel free to add anything here and if you have a fix for any of these, certainly submit a pull request targeting the next round.

Framework	Rnd 13	Rnd 14	Local	Notes
C/lwan	:heavy_multiplication_x:		:heavy_check_mark:
C/haywire	:heavy_multiplication_x:		:heavy_check_mark:	Submitted #2099 with a fix for round-14
CSharp/servicestack	:heavy_multiplication_x:		:heavy_multiplication_x:
C++/cpoll_cppsp	:heavy_check_mark:		:heavy_check_mark:
C++/treefrog	:heavy_multiplication_x:	:heavy_check_mark:	:heavy_check_mark:
C++/silicon	:heavy_check_mark:		:heavy_check_mark:	Submitted #2082 with a fix for round-14
Crystal/crystal-raw	:heavy_multiplication_x:	:heavy_check_mark:	:heavy_check_mark:
Crystal/kemal	:heavy_multiplication_x:		:heavy_check_mark:
Haskell/spock	:heavy_multiplication_x:	:heavy_check_mark:	:heavy_check_mark:
Haskell/yesod	:heavy_check_mark:		:heavy_check_mark:
Java/activeweb	:heavy_multiplication_x:		:heavy_check_mark:	Submitted #2118 with a fix for round-14
Java/play1siena	:heavy_multiplication_x:		:heavy_multiplication_x:	Submitted #2149 to remove this
JavaScript/ringojs-convenient	:heavy_multiplication_x:	:heavy_check_mark:	:heavy_check_mark:	#2049 fixes this
JavaScript/sailsjs	:heavy_multiplication_x:		:heavy_check_mark:
Nim/jester	:heavy_multiplication_x:		:heavy_check_mark:	Submitted #2081 with a fix for round-14
Nim/nawak	:heavy_multiplication_x:		:heavy_check_mark:	Submitted #2081 with a fix for round-14
PHP/cakephp	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	Submitted #2124 with a fix for round-14
PHP/hhvm	:heavy_multiplication_x:	:heavy_check_mark:	:heavy_check_mark:
PHP/cygnite-php-framework	:heavy_multiplication_x:	:heavy_check_mark:	:heavy_check_mark:
PHP/codeigniter	:heavy_multiplication_x:	:heavy_check_mark:	:heavy_check_mark:
PHP/clancats	:heavy_multiplication_x:	:heavy_check_mark:	:heavy_check_mark:	Submitted #2124 with a fix for round-14
PHP/laravel	:heavy_multiplication_x:	:heavy_check_mark:	:heavy_check_mark:
PHP/micromvc	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	Submitted #2124 with a fix for round-14
PHP/php-phalcon	:heavy_multiplication_x:	:heavy_check_mark:	:heavy_check_mark:	Submitted #2124 with a fix for round-14
PHP/php-phalcon-micro	:heavy_multiplication_x:	:heavy_check_mark:	:heavy_check_mark:	Submitted #2124 with a fix for round-14
PHP/pimf	:heavy_check_mark:	:heavy_check_mark:	:heavy_check_mark:	Submitted #2124 with a fix for round-14
PHP/silex-orm	:heavy_multiplication_x:	:heavy_check_mark:	:heavy_check_mark:	Submitted #2109 with a fix for round-14
PHP/slim	:heavy_multiplication_x:	:heavy_check_mark:	:heavy_check_mark:
PHP/symfony2	:heavy_multiplication_x:	:heavy_check_mark:	:heavy_check_mark:
PHP/yaf	:heavy_multiplication_x:	:heavy_check_mark:	:heavy_check_mark:	Submitted #2124 with a fix for round-14
PHP/yii2	:heavy_multiplication_x:	:heavy_check_mark:	:heavy_check_mark:	Submitted #2124 with a fix for round-14
PHP/zend	:heavy_multiplication_x:	:heavy_check_mark:	:heavy_check_mark:
Racket/racket-ws	:heavy_multiplication_x:		:heavy_check_mark:	Currently pulling the latest racket source code. /toolset/setup/linux/languages/racket.sh should be updated to use a specific version of the language.
Ruby/grape	:heavy_multiplication_x:		:heavy_check_mark:
Ruby/padrino	:heavy_multiplication_x:		:heavy_check_mark:
Ruby/rack	:heavy_multiplication_x:		:heavy_check_mark:
Ruby/rails	:heavy_multiplication_x:		:heavy_check_mark:
Ruby/rails-stripped	:heavy_multiplication_x:		:heavy_check_mark:
Ruby/sinatra	:heavy_multiplication_x:		:heavy_check_mark:	#2085 fixes this
Ruby/sinatra-sequel	:heavy_multiplication_x:		:heavy_check_mark:
Rust/iron	:heavy_multiplication_x:	:heavy_check_mark:	:heavy_check_mark:
Rust/hyper	:heavy_multiplication_x:	:heavy_check_mark:	:heavy_check_mark:
Scala/lift-stateless	:heavy_multiplication_x:		:heavy_multiplication_x:
Scala/plain	:heavy_multiplication_x:		:heavy_multiplication_x:
Scala/scalatra	:heavy_multiplication_x:		:heavy_multiplication_x:
Scala/spray	:heavy_multiplication_x:		:heavy_check_mark:	#2087 fixes this
Scala/spray-es	:heavy_multiplication_x:		:heavy_check_mark:
Scala/unfiltered	:heavy_multiplication_x:		:heavy_multiplication_x:
Ur/urweb	:heavy_multiplication_x:		:heavy_check_mark:

[martin-g]

I'm glad to see that Java/Wicket is not in this list! But the last three rounds it works fine on Travis and locally but fails for the official run (#1403) :-/

[NateBrady23]

@martin-g These frameworks were pulled from Travis tests on the latest master run and the latest round 13 run. I wish I could say that's a guarantee that there won't be a problem with the official run, but I will pull Wicket and test locally tomorrow.

[martin-g]

Thank you, @nbrady-techempower ! Let me know if something has to be fixed! I really like to see Wicket results again.

[NateBrady23]

@martin-g I've tested wicket again in a local environment and all tests pass. There were some results for round-12, however there were some errors too. Here are the logs for wicket in Rnd 12 It appears things are working fine at the moment now.

[martin-g]

Great! Thanks!

[Rydgel]

Spock failed on 13 but works on 14 and local. So I suppose we are good for this one?

[NateBrady23]

@Rydgel I believe it should be.

[dashorst]

@nbrady-techempower @martin-g Unfortunately the logs only give a timeout for the database call. AFAIK Wicket can't do anything about that. Either the configuration for the database pool is incorrect, or another database connection pool should be used, or the database itself is unresponsive.

[NateBrady23]

@dashorst Thanks for that update! I've tested it in a few environments and can't reproduce. We will see what the official run shows and if the problem persists, try and solve it for good!

[achlipala]

Do you have any tips on how to get the Travis error log for a framework that isn't passing? I don't know how to find the first commit where urweb broke. (I'm generally in charge of that framework's code.)

[achlipala]

Never mind my last question; I just poked around a bit more and found a case where that test fails.

It looks like Ubuntu package mlton is no longer being installed as part of the baseline list of required packages. urweb requires this package. Is it possible to readd? (I'm not sure where is the canonical place to do so in the latest layout.)

[NateBrady23]

@achlipala It looks like you found the solution. Let us know if you need anything else.

misread. If you need to install packages that will only be part of one framework you can do it in toolset/linux/setup/frameworks/<frameworkname>.sh

[saturday06]

Crystal/moonshine is no longer maintained. I opened https://github.com/TechEmpower/FrameworkBenchmarks/pull/2127 to remove it.

[martin-g]

Unfortunately Wicket is still broken in the "production" environment: https://www.techempower.com/benchmarks/previews/round13/ :-( Only Plaintext and JSONSerialization passed, for all others "Did not complete".

[jkarni]

servant is listed in the preliminary round 13 results but not here (even though it failed some tests). The PR was merged into round-14, and discussion there seemed to suggest it wouldn't be included in round 13. Does the fact that it shows up in preliminary results mean it will?

[NateBrady23]

@jkarni Sorry for the late response, due to the delay in getting the new servers set up we decided to push through a lot of work toward round-14 in to round-13. So yes, it will. The preliminary results contain some issues with setup that may have affected some tests. We have resolved those issues as well.

[jknack]

Hi @nbrady-techempower,

Don't see http://jooby.org here but the tests in Round 13 preview are marked as did not complete. Is there anything I need to do to see the results in 13 preview?

Thanks

[NateBrady23]

@jknack Hey, so I don't think there will be an issue with jooby. A bug was introduced that affected some databases and caused global problems. That has been resolved. We should have some data available shortly.

[jknack]

@nbrady-techempower got it! thanks for your quick response!

[knewmanTE]

Updated the table now that the following tests are fixed:

• Crystal/kemal
• Ruby/grape
• Ruby/padrino
• Ruby/rack
• Ruby/rails
• Ruby/rails-fixed
• Ruby/sinatra-sequel
• Scala/spray

[mwpastore]

JRuby (9.1) appears to be working again after working around #2381.

[knewmanTE]

@mwpastore, It looks like JRuby was removed in #2277 because all of the JRuby tests were breaking. If it is working now, would you be able to open up a PR that adds the JRuby tests back in?

[zloster]

grizzly-jersey Plaintext seems to have runtime problems. See: https://github.com/TechEmpower/FrameworkBenchmarks/pull/2383#issue-192383588

[mwpastore]

@knewmanTE I'm happy to do so for the framework I maintain (sinatra-sequel), and in fact have a PR just about ready to go.

[knewmanTE]

@mwpastore awesome!

[msmith-techempower]

@nbrady-techempower What is the status of this task? Can we close this?

[NateBrady23]

Yessir!