Jetty does not respond after being under load / no selector available?

[SimonK15]

Jetty Version: 9.3.6.v20151106

When putting an embedded jetty server under load the server end ups in a state where jetty can not process further requests.

The problem might be that the JDK selector has been closed, or some Jetty component has been stopped.

A dump of jetty shows the following:

|   += org.eclipse.jetty.server.ServerConnector$ServerConnectorManager@30b34287 - STARTED
|   |   +- org.eclipse.jetty.io.ManagedSelector@251bd1d8 id=0 keys=497 selected=0 id=0
|   |   +- org.eclipse.jetty.io.ManagedSelector@1f6cf02f id=1 keys=266 selected=0 id=1
|   |   +- org.eclipse.jetty.io.ManagedSelector@6402770b id=2 keys=273 selected=0 id=2
|   |   +- org.eclipse.jetty.io.ManagedSelector@7c89f880 id=3 keys=1042 selected=1 id=3
|   +~ org.eclipse.jetty.jmx.MBeanContainer@12299890

while a normal dump is supposed to contain some more informations below the Selector:

|   |   += org.eclipse.jetty.io.ManagedSelector@3c88191b id=0 keys=1 selected=0 id=0
|   |       +- sun.nio.ch.EPollSelectorImpl@a6116b3 keys=1
|   |       +-SelectionKey@47c7c0bd{i=1}->SocketChannelEndPoint@4bd83734{/127.0.0.1:43128<->/127.0.0.1:6443,OPEN,fill=FI,flush=-,to=14608/30000}{io=1/1,kio=1,kro=1}-

A sample webapp project can be found here: https://github.com/SimonK15/jetty-test-server Check the README.md for more information on how to reproduce the error. A tsung config file for a load tests is included as well.

Refer to the [jetty-users] mailinglist topic 'Jetty LoadTests and no available selectors' for the discussion.

[joakime]

If you use the latest Jetty does it still fail? Jetty 9.3.6 is on the old side now.

[sbordet]

@SimonK15 can you please detail how to setup tsung ? How do I install it, run it, etc ? Assume Linux.

[sbordet]

Managed to get tsung installed, but cannot get it to run and perform requests.

$ tsung -f etc/tsung.xml start
Starting Tsung
Log directory is: /home/simon/.tsung/log/20170616-1657
ssh: connect to host localhost port 22: Connection refused
[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed

Why does it need to connect to port 22 ?

Also, your etc/tsung.xml in your sample project is missing a closing } in the JSON that is sent on the request POST.

[SimonK15]

Why does it need to connect to port 22 ?

As far as I remember, the tsung 'master' distributes the tasks to the tsung 'clients' via ssh. It also does some monitoring. If you go for that setup, you will need to make the ssh keys of client and master known to each other.

Also, your etc/tsung.xml in your sample project is missing a closing } in the JSON that is sent on the request POST.

Fixed that typo.

[SimonK15]

If you use the latest Jetty does it still fail? Jetty 9.3.6 is on the old side now.

I did some test with the newest version '9.4.6.v20170531' and jetty does not seem to get stuck anymore. So I can't say whether something was fixed or its now just less likely to happen.

[sbordet]

Feel free to close the issue then.

[SimonK15]

I had hoped somebody could figure out what the reason behind this behavious was (or is?), but it seems unlikely that this will ever happen. So I close the ticket ...