TechEmpower / FrameworkBenchmarks

Source for the TechEmpower Framework Benchmarks project
https://www.techempower.com/benchmarks/
Other
7.67k stars 1.95k forks source link

Unable to run hexagon-resin test #2434

Closed jaguililla closed 6 years ago

jaguililla commented 7 years ago

@knewmanTE @nbrady-techempower @LadyMozzarella This is the issue for the problem with PR #2421

Description

The standalone test of Hexagon is working, however, the Resin one is not working.

Tapestry Resin test is working so I trimmed my test to be as close as possible to theirs.

Even in that case the test fail (timeout connecting to Resin: 127.0.0.1:8080).

I downloaded the generated WAR and it worked fine in a local Resin (4.0.49 on Linux).

I changed the hexagon directory permissions and added a+w (like in Tapestry) just in case.

One difference between woking Resin tests and Hexagon's is the fact that they live in different directories (Java and Kotlin) I do not know if this makes a difference.

I made changes in my development vagrant machine. So the latest test details are below:

Setup

vagrant@tfb-all:~/FrameworkBenchmarks$ cat frameworks/Kotlin/hexagon/benchmark_config.json 
{
    "framework" : "hexagon",
    "tests" : [
        {
            "default" : {
                "json_url" : "/hexagon/json",
                "db_url" : "/hexagon/db",
                "query_url" : "/hexagon/query?queries=",
                "fortune_url" : "/hexagon/fortunes",
                "update_url" : "/hexagon/update?queries=",
                "plaintext_url" : "/hexagon/plaintext",

                "port" : 8080,

                "approach" : "Realistic",
                "classification" : "Micro",
                "database" : "MongoDB",
                "framework" : "Hexagon",
                "language" : "Kotlin",
                "orm" : "Raw",
                "platform" : "Servlet",
                "webserver" : "Resin",
                "os" : "Linux",
                "database_os" : "Linux",
                "display_name" : "Hexagon Resin",
                "notes" : "http://there4.co/hexagon",

                "setup_file" : "setup",
                "versus" : "servlet"
            }
        }
    ]
}

vagrant@tfb-all:~/FrameworkBenchmarks$ cat frameworks/Kotlin/hexagon/setup.sh 
#!/bin/bash

fw_depends mongodb java resin

gradle/wrapper

rm -rf $RESIN_HOME/webapps/*
cp build/libs/hexagon.war $RESIN_HOME/webapps
resinctl start

Building

vagrant@tfb-all:~/FrameworkBenchmarks$ ls -AlF frameworks/Kotlin/hexagon/build/libs/
total 17892
-rw-rw-r-- 1 vagrant vagrant    26052 Dec 27 23:22 hexagon-1.0.0.jar
-rw-rw-r-- 1 vagrant vagrant 18290814 Dec 27 23:22 hexagon.war

Execution

Setup hexagon: ~/FrameworkBenchmarks/installs ~/FrameworkBenchmarks/frameworks/Kotlin/hexagon
Setup hexagon: java is installed!
Setup hexagon: ~/FrameworkBenchmarks/frameworks/Kotlin/hexagon
Setup hexagon: java is installed!
Setup hexagon: :compileKotlin
Setup hexagon: :compileJava UP-TO-DATE
Setup hexagon: :copyMainKotlinClasses
Setup hexagon: :processResources
Setup hexagon: :classes
Setup hexagon: :compileTestKotlin
Setup hexagon: :compileTestJava UP-TO-DATE
Setup hexagon: :copyTestKotlinClasses
Setup hexagon: :processTestResources
Setup hexagon: :testClasses
Setup hexagon: :test
Setup hexagon: :check
Setup hexagon: :jar
Setup hexagon: :startScripts
Setup hexagon: :war
Setup hexagon: :installDist
Setup hexagon: :install
Setup hexagon: 
Setup hexagon: BUILD SUCCESSFUL
Setup hexagon: 
Setup hexagon: Total time: 20.781 secs
Setup hexagon: Resin/4.0.48 launching watchdog at 127.0.0.1:6600
Setup hexagon: Resin/4.0.48 started -server 'default' with watchdog at 127.0.0.1:6600
Setup hexagon: Status: Poll: None, Port 8080 bound: True, Time Left: 1:44:25.757481
Setup hexagon: Bound port detected on 8080
INFO:root:Executed setup.sh, returning 0
INFO:root:Sleeping 60 seconds to ensure framework is ready
INFO:root:Verifying framework URLs
Accessing URL http://127.0.0.1:8080/hexagon/db:
WARNING:root:Verifying test db for hexagon caused an exception: timed out
   FAIL for http://127.0.0.1:8080

     Caused Exception in TFB
                 This almost certainly means your return value is incorrect,
                 but also that you have found a bug. Please submit an issue
                 including this message: timed out
     Traceback (most recent call last):
       File "/home/vagrant/FrameworkBenchmarks/toolset/benchmark/framework_test.py", line 395, in verify_type
         results = test.verify(base_url)
       File "/home/vagrant/FrameworkBenchmarks/toolset/benchmark/test_types/db_type.py", line 27, in verify
         headers, body = self.request_headers_and_body(url)
       File "/home/vagrant/FrameworkBenchmarks/toolset/benchmark/test_types/framework_test_type.py", line 85, in request_headers_and_body
         r = requests.get(url, timeout=15, headers=headers)
       File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
         return request('get', url, **kwargs)
       File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
         return session.request(method=method, url=url, **kwargs)
       File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 455, in request
         resp = self.send(prep, **send_kwargs)
       File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 558, in send
         r = adapter.send(request, **kwargs)
       File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 394, in send
         r.content
       File "/usr/lib/python2.7/dist-packages/requests/models.py", line 679, in content
         self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
       File "/usr/lib/python2.7/dist-packages/requests/models.py", line 616, in generate
         decode_content=True):
       File "/usr/lib/python2.7/dist-packages/urllib3/response.py", line 225, in stream
         data = self.read(amt=amt, decode_content=decode_content)
       File "/usr/lib/python2.7/dist-packages/urllib3/response.py", line 174, in read
         data = self._fp.read(amt)
       File "/usr/lib/python2.7/httplib.py", line 602, in read
         s = self.fp.read(amt)
       File "/usr/lib/python2.7/socket.py", line 380, in read
         data = self._sock.recv(left)
     timeout: timed out
     See http://frameworkbenchmarks.readthedocs.org/en/latest/Project-Information/Framework-Tests/#specific-test-requirements

Resin logs

vagrant@tfb-all:~/FrameworkBenchmarks$ tail -n 55 /home/vagrant/FrameworkBenchmarks/installs/resin-4.0.48/log/jvm-default.log   
[19:34:38.153] http listening to *:8080
[19:34:38.155] 
[19:34:38.156] Resin[id=default] started in 3692ms
[19:35:39.472] HCANN000001: Hibernate Commons Annotations {4.0.5.Final}
[19:35:39.479] HHH000412: Hibernate Core {4.3.6.Final}
[19:35:39.482] HHH000206: hibernate.properties not found
[19:35:39.485] HHH000021: Bytecode provider name : javassist
[19:35:39.505] HHH000043: Configuring from resource: /hibernate.cfg.xml
[19:35:39.505] HHH000040: Configuration resource: /hibernate.cfg.xml
[19:35:39.558] HHH000223: Recognized obsolete hibernate namespace http://hibernate.sourceforge.net/. Use namespace http://www.hibernate.org/dtd/ instead. Refer to Hibernate 3.6 Migration Guide!
[19:35:39.583] HHH000041: Configured SessionFactory: null
[19:35:40.017] HHH000400: Using dialect: org.hibernate.dialect.MySQLDialect
[19:35:40.165] HHH000399: Using default transaction strategy (direct JDBC transactions)
[19:35:40.170] HHH000397: Using ASTQueryTranslatorFactory
[19:35:40.221] HV000001: Hibernate Validator 4.3.0.Final
Resin-4.0.48 (built Wed, 17 Feb 2016 11:01:17 PST)
Copyright(c) 1998-2012 Caucho Technology.  All rights reserved.

  Using Resin(R) Open Source under the GNU Public License (GPL).

  See http://www.caucho.com for information on Resin Professional,
  including caching, clustering, JNI acceleration, and OpenSSL integration.

Starting Resin on Tue, 27 Dec 2016 23:22:45 +0000 (UTC)

[23:22:45.904] 
[23:22:45.905] Resin-4.0.48 (built Wed, 17 Feb 2016 11:01:17 PST)
[23:22:45.905] 
[23:22:45.905] Linux 3.13.0-105-generic amd64
[23:22:45.905] Java(TM) SE Runtime Environment 1.8.0_111-b14, UTF-8, en
[23:22:45.905] Java HotSpot(TM) 64-Bit Server VM 25.111-b14, 64, mixed mode, Oracle Corporation
[23:22:45.905] 
[23:22:45.905] user.name  = vagrant
[23:22:46.013] Table[mnode:2,/home/vagrant/FrameworkBenchmarks/installs/resin-4.0.48/resin-data/default/distcache/mnode.db] validating indexes due to unclean shutdown.
[23:22:46.033] Table[data:3,/home/vagrant/FrameworkBenchmarks/installs/resin-4.0.48/resin-data/default/distcache/data.db] validating indexes due to unclean shutdown.
[23:22:46.046] 
[23:22:46.046] resin.home = /home/vagrant/FrameworkBenchmarks/installs/resin-4.0.48/
[23:22:46.046] resin.root = /home/vagrant/FrameworkBenchmarks/installs/resin-4.0.48
[23:22:46.046] resin.conf = /home/vagrant/FrameworkBenchmarks/installs/resin-4.0.48/conf/resin.xml
[23:22:46.046] 
[23:22:46.046] server    = 127.0.0.1:-1 (:default)
[23:22:46.046] stage      = production
[23:22:48.021] servlet-mapping *.jsp -> resin-jsp
[23:22:48.021] servlet-mapping *.jspf -> resin-jsp
[23:22:48.021] servlet-mapping *.jspx -> resin-jspx
[23:22:48.031] servlet-mapping *.php -> resin-php
[23:22:48.031] servlet-mapping / -> resin-file
[23:22:48.087] In-place class redefinition (HotSwap) is available.
[23:22:48.442] WebApp[production/webapp/default/hexagon] active
[23:22:48.443] Host[production/host/default] active
[23:22:48.444] ServletService[id=default,cluster=] active
[23:22:48.444] 
[23:22:48.450] http listening to *:8080
[23:22:48.458] 
[23:22:48.461] Resin[id=default] started in 3351ms

If you need any other info, just write me

NateBrady23 commented 7 years ago

Great information @jaguililla ! I will take a closer look at this today!

NateBrady23 commented 7 years ago

@jaguililla After running your resin test, I don't see any resin-web.xml or web.xml files in $RESIN_HOME/webapps/ROOT/WEB-INF/

How are you configuring the db / db drivers with resin?

jaguililla commented 7 years ago

It seems to be failing because of a timeout (does It reach the webapp?). Could it be the network interface of Resin? or the user that started the process? I don't know... just giving ideas.

The only thing I saw different from Tapestry (after I modified the tests locally) was the parent directory of the test (Kotlin instead Java)

NateBrady23 commented 7 years ago

The parent directory of the framework shouldn't matter here. Clojure/ tests also use resin successfully. The users that start the process for the other frameworks are the same here as well.

jaguililla commented 7 years ago

I have no clue :( it works in my local resin, but it fails in vagrant-development

NateBrady23 commented 7 years ago

Have you made any changes to your local resin conf files?

jaguililla commented 7 years ago

No, just unzipped Resin and deployed the war

NateBrady23 commented 7 years ago

@jaguililla I've tracked this down to an issue in the python requests module. I'm going to look in to making some quick changes. But this does not seem to be related to your framework. I'll ping you when I have more information.

NateBrady23 commented 7 years ago

@jaguililla Ok, so a little more digging. I thought it might be an issue with the headers. After looking at some verbose logging from resin, I'm getting headers back from you but the connection is being kept-alive when using python's requests module. However, when testing tapestry, the connection is closing properly.

Upon further inspection it looks like your response is actually disconnecting too fast. I'm not getting the entire response before the connection is closed. Below are two verbose curl responses. Every time your connection closes before getting the final }

vagrant@TFB-all:~$ curl -v http://127.0.0.1:8080/tapestry/hellojson
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /tapestry/hellojson HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 127.0.0.1:8080
> Accept: */*
>
< HTTP/1.1 200 OK
* Server Resin/4.0.48 is not blacklisted
< Server: Resin/4.0.48
< Content-Type: application/json; charset=UTF-8
< Transfer-Encoding: chunked
< Date: Thu, 29 Dec 2016 18:29:23 GMT
<
* Connection #0 to host 127.0.0.1 left intact
{"message":"Hello, World!"}

vagrant@TFB-all:~$ curl -v http://127.0.0.1:8080/json
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /json HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 127.0.0.1:8080
> Accept: */*
>
< HTTP/1.1 200 OK
* Server Servlet/3.1 is not blacklisted
< Server: Servlet/3.1
< Transfer-Encoding: chunked
< Content-Type: application/json
< Transfer-Encoding: chunked
< Date: Thu, 29 Dec 2016 18:34:37 GMT
<
{
  "message" : "Hello, World!"
* Connection #0 to host 127.0.0.1 left intact
}
jam-walkers commented 7 years ago

Regarding this problem, when I run curl -v http://127.0.0.1:8080/json >output.json the output.json file contains a valid response (done from the Resin directory with vagrant user in the Vagrant development machine).

NateBrady23 commented 7 years ago

@jam-walkers For the hexagon-resin test? Can you please create a sprunge or other of your output from running that test? And what commit are you running it from, since the test no longer exists in the repo.

jaguililla commented 7 years ago

The previous comment was me using a different account, sorry. The test I ran was over the latest version from master adding the test to the configuration manually.

On the Vagrant development machine. Running: ~/FrameworkBenchmarks/toolset/run-tests.py --mode verify --test hexagon-resin failed

But running Resin outside the toolkit it worked:

RESIN_HOME="/home/vagrant/FrameworkBenchmarks/installs/resin-4.0.48" /home/vagrant/FrameworkBenchmarks/installs/resin-4.0.48/bin/resinctl start

After starting up Resin. I executed:

curl -s http://127.0.0.1:8080/fortunes >>out.log && curl -s http://127.0.0.1:8080/db >>out.log && cat out.log

And the result was:

<!DOCTYPE html>

<html>
<head>
  <title>Fortunes</title>
</head>
<body>
<table>
  <tr>
    <th>id</th>
    <th>message</th>
  </tr>
    <tr>
    <td>11</td>
    <td>&lt;script&gt;alert(&quot;This should not be displayed in a browser alert box.&quot;);&lt;/script&gt;</td>
  </tr>
    <tr>
    <td>4</td>
    <td>A bad random number generator: 1, 1, 1, 1, 1, 4.33e+67, 1, 1, 1</td>
  </tr>
    <tr>
    <td>5</td>
    <td>A computer program does what you tell it to do, not what you want it to do.</td>
  </tr>
    <tr>
    <td>2</td>
    <td>A computer scientist is someone who fixes things that aren&#39;t broken.</td>
  </tr>
    <tr>
    <td>8</td>
    <td>A list is only as strong as its weakest link. — Donald Knuth</td>
  </tr>
    <tr>
    <td>0</td>
    <td>Additional fortune added at request time.</td>
  </tr>
    <tr>
    <td>3</td>
    <td>After enough decimal places, nobody gives a damn.</td>
  </tr>
    <tr>
    <td>7</td>
    <td>Any program that runs right is obsolete.</td>
  </tr>
    <tr>
    <td>10</td>
    <td>Computers make very fast, very accurate mistakes.</td>
  </tr>
    <tr>
    <td>6</td>
    <td>Emacs is a nice operating system, but I prefer UNIX. — Tom Christaensen</td>
  </tr>
    <tr>
    <td>9</td>
    <td>Feature: A bug with seniority.</td>
  </tr>
    <tr>
    <td>1</td>
    <td>fortune: No such file or directory</td>
  </tr>
    <tr>
    <td>12</td>
    <td>フレームワークのベンチマーク</td>
  </tr>
  </table>
</body>
</html>
{
  "id" : 1402,
  "randomNumber" : 252
}

Which seems good. This was with vagrant user. Any clue?

NateBrady23 commented 7 years ago

@jaguililla Can you do the same with verbos, -v, output for curl? I'm at a loss here, but I want to see what that shows as far as the connection goes.

jaguililla commented 7 years ago

This is the output with -vvv (and thanks for keeping an eye in this issue):

* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /fortunes HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 127.0.0.1:8080
> Accept: */*
> 
< HTTP/1.1 200 OK
* Server Servlet/3.1 is not blacklisted
< Server: Servlet/3.1
< Cache-Control: private
< Transfer-Encoding: chunked
< Set-Cookie: JSESSIONID=aaaPvj0REhS6TezwLhVNv; path=/
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Date: Mon, 30 Jan 2017 14:04:16 GMT
< 
{ [data not shown]
100  1491    0  1491    0     0   423k      0 --:--:-- --:--:-- --:--:--  485k
* Connection #0 to host 127.0.0.1 left intact
<!DOCTYPE html>

<html>
<head>
  <title>Fortunes</title>
</head>
<body>
<table>
  <tr>
    <th>id</th>
    <th>message</th>
  </tr>
    <tr>
    <td>11</td>
    <td>&lt;script&gt;alert(&quot;This should not be displayed in a browser alert box.&quot;);&lt;/script&gt;</td>
  </tr>
    <tr>
    <td>4</td>
    <td>A bad random number generator: 1, 1, 1, 1, 1, 4.33e+67, 1, 1, 1</td>
  </tr>
    <tr>
    <td>5</td>
    <td>A computer program does what you tell it to do, not what you want it to do.</td>
  </tr>
    <tr>
    <td>2</td>
    <td>A computer scientist is someone who fixes things that aren&#39;t broken.</td>
  </tr>
    <tr>
    <td>8</td>
    <td>A list is only as strong as its weakest link. — Donald Knuth</td>
  </tr>
    <tr>
    <td>0</td>
    <td>Additional fortune added at request time.</td>
  </tr>
    <tr>
    <td>3</td>
    <td>After enough decimal places, nobody gives a damn.</td>
  </tr>
    <tr>
    <td>7</td>
    <td>Any program that runs right is obsolete.</td>
  </tr>
    <tr>
    <td>10</td>
    <td>Computers make very fast, very accurate mistakes.</td>
  </tr>
    <tr>
    <td>6</td>
    <td>Emacs is a nice operating system, but I prefer UNIX. — Tom Christaensen</td>
  </tr>
    <tr>
    <td>9</td>
    <td>Feature: A bug with seniority.</td>
  </tr>
    <tr>
    <td>1</td>
    <td>fortune: No such file or directory</td>
  </tr>
    <tr>
    <td>12</td>
    <td>フレームワークのベンチマーク</td>
  </tr>
  </table>
</body>
</html>
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /db HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 127.0.0.1:8080
> Accept: */*
> 
< HTTP/1.1 200 OK
* Server Servlet/3.1 is not blacklisted
< Server: Servlet/3.1
< Transfer-Encoding: chunked
< Content-Type: application/json
< Transfer-Encoding: chunked
< Date: Mon, 30 Jan 2017 14:04:16 GMT
< 
{ [data not shown]
100    42    0    42    0     0  15069      0 --:--:-- --:--:-- --:--:-- 21000
* Connection #0 to host 127.0.0.1 left intact
{
  "id" : 6221,
  "randomNumber" : 4429
}
NateBrady23 commented 7 years ago

That's looks different from the response I was getting. If you have the time, go ahead and open up a PR with the resin test added back in and I'll see if I can wrangle some people here in to look at it again.

jaguililla commented 7 years ago

Done (#2531)

msmith-techempower commented 6 years ago

Resolved by #2531