search

TechEmpower / FrameworkBenchmarks

Source for the TechEmpower Framework Benchmarks project
https://www.techempower.com/benchmarks/
Other
7.66k stars 1.95k forks source link

Tests that need updating before next round #5189

Open NateBrady23 opened 5 years ago

NateBrady23 commented 5 years ago

This is a living list of tests that need to be updated or removed before we record our next official round results. I'll do my best to ping the most recent maintainers as well as list the problem. Tests that aren't updated/fixed prior to the next round will be removed to free up resources and speed up the testing cycle.

Tests that are broken

C++/ffead-cpp Crystal/spider-gazelle Dart/hunt-dmd Go/chi-* Go/revel Haskell/servant-beam Haskell/spock

Tests that are failing query verification #5145

C++/cpoll_cppsp-postgres-raw C++/cpoll_cppsp-postgres-raw-threadpool C++/cpoll_cppsp-raw C++/cppcms C++/silicon C++/treefrog-hybrid Dart/hunt-http Elixir/phoenix

Tests that are not reading the id column in the queries or db tests #5167

D/hunt D/vibed

This list is still growing. I'm not even a 3rd of the way through my audit. Please feel free to contribute in the comments section.

jcheron commented 5 years ago

I had already taken a little lead starting from the end of the Travis run:

Tests that are broken

Tests that are failing query verification https://github.com/TechEmpower/FrameworkBenchmarks/pull/5145

Tests that are not reading the id column in the queries or db tests https://github.com/TechEmpower/FrameworkBenchmarks/issues/5167

joanhey commented 5 years ago

All php frameworks are correct after PR #5192 for #5167.

The only that need the change is hhvm, but it is broken a long time ago.

@jcheron You marked frameworks that are correct. php, php-ngx, fat-fee, workerman, kumbiaphp, ...

They only omit the id in the update tests that is permited.

joanhey commented 5 years ago

php/amp is not broken, only hhvm.

joanhey commented 5 years ago

PHP/cakephp the fortunes test is broken after the last version update.

joanhey commented 5 years ago

https://www.techempower.com/benchmarks/#section=test&runid=59583d86-f7b6-4e86-b86a-a7e8eebf89fb&hw=ph&test=fortune&l=zik073-v the last row cakephp - Did not complete and https://tfb-status.techempower.com/unzip/results.2019-10-28-04-17-30-832.zip/results/20191024072803/cakephp/fortune/raw.txt

But the run log is empty https://tfb-status.techempower.com/unzip/results.2019-10-28-04-17-30-832.zip/results/20191024072803/cakephp/run

That happens after update the cakephp version.

jcheron commented 5 years ago

Maybe it was just an accident, it looks good on Travis.

joanhey commented 5 years ago

It looks good on travis, but the fortunes test always failed for the last 6 months in citrine and azure.

In azure: https://www.techempower.com/benchmarks/#section=test&runid=45c588f9-9968-4330-934f-bd39ed92442c&hw=ph&test=fortune&l=zik073-v

I think that we could create an issue for the php frameworks, to clean this issue. Created issue #5193

an-tao commented 5 years ago

@nbrady-techempower ,Hey, I have updated drogon with the PR #5191 , I think you can remove drogon from the list, thanks.

michaelhixson commented 5 years ago

@jcheron You commented in your PR that several Java frameworks using Hibernate were failing query verification, but they're not in your list in this thread. Why is that?

jcheron commented 5 years ago

@michaelhixson I continue the pointing, starting from the bottom, I'm close to java...

joanhey commented 5 years ago

Tests that are not reading the id column in the queries or db tests #5167

C++/cpoll_cppsp (all variants) C++/ulib Crystal/crystal Crystal/kemal Crystal/raze D/hunt (all variants) Perl/mojolicious Perl/plack Python/aiohttp Python/quart Racket/racket