search

jetty / jetty.project

Eclipse Jetty® - Web Container & Clients - supports HTTP/2, HTTP/1.1, HTTP/1.0, websocket, servlets, and more
https://eclipse.dev/jetty
Other
3.84k stars 1.91k forks source link

WebSocket hangs in blockingWrite #272

Closed jmcc0nn3ll closed 5 years ago

jmcc0nn3ll commented 8 years ago

migrated from Bugzilla #484698 status NEW severity normal in component websocket for 9.3.x Reported in version 9.3.5 on platform PC Assigned to: Joakim Erdfelt

On 2015-12-18 10:48:14 -0500, Ben Tatham wrote:

My application hangs in the following state. This is usually triggered by some underlying network change (like ip address change of the host running jetty), but it might happen in other cases too.

java.lang.Thread.State: WAITING at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <1edddee> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:210) at org.eclipse.jetty.websocket.common.BlockingWriteCallback$WriteBlocker.block(BlockingWriteCallback.java:82) at org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.blockingWrite(WebSocketRemoteEndpoint.java:107) at org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendBytes(WebSocketRemoteEndpoint.java:252)

Based on the code, it looks like the m_complete.await() call should be signalled on close, failure, or success, but never is.

dreis2211 commented 6 years ago

@sbordet & @joakime The issue doesn't seem to be solved unfortunately. With Jetty 9.4.8 it seems to be even worse and more frequent, but this might be just bad luck.

Nonetheless, I'd appreciate if this could be under investigation again.

joakime commented 6 years ago

@dreis2211 please file a new issue, and if you have any new (up to date) stacktraces with 9.4.8, that would be useful.

cdacvikram commented 6 years ago

Hi @joakime, we are also getting the same type of error ""pool-18-thread-42" #6179 prio=5 os_prio=0 tid=0x00007f7be0262800 nid=0x1741 waiting on condition [0x00007f7b1a21a000] java.lang.Thread.State: WAITING (parking)"

It never ends and when ever we load with more data the server with websocket is unresponsive at a particular time. Is there any direct fix, do we need to configure the server or java code. We are using spring boot with websocket.

sbordet commented 6 years ago

@cdacvikram you did not provide enough information. It's not clear if the stack trace is the same of this issue, and you don't specify what Jetty version you use.

Please provide a full stack trace and a server dump.

fredericofagundes commented 6 years ago

Good morning,

I'm experiencing issues with leased threads and the dump of the application behind the following code snippet, currently using the jetty version 9.3.11.v20160721, would anyone have a solution?

"jetty-queued-54-acceptor-1@9a0429-ServerConnector@390d87cf{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}" #54 prio=3 os_prio=0 tid=0x00007f9f25f08dd0 nid=0x43df waiting for monitor entry [0x00007f9fb11cd000] java.lang.Thread.State: BLOCKED (on object monitor) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:233)

joakime commented 6 years ago

@fredericofagundes that is unrelated to this issue/bug.

sbordet commented 6 years ago

@fredericofagundes that thread in BLOCKED state is normal, there are 2 acceptor threads that are competing to accept.

fredericofagundes commented 6 years ago

@sbordet Thanks for the reply, my project uses Jetty in version 9.3.11.v20160721, at some point in the day some of these threads are blocked and the machine needs to be restarted to return to normal. I saw some suggestions to increase the timeout but it does not solve the problem. This information I passed above was taken from the thread dump. Would you have any other suggestions for what to do?

sbordet commented 6 years ago

@fredericofagundes what you report is too vague.

We need the JVM thread dump and the Jetty server dump.

artem-emelin commented 6 years ago

Getting this issue in production environment: Jetty 9.4.11.v20180605 with Spring Boot 1.5.14

Thread stacktrace:

"outgoing-message-sender-0" prio=5 WAITING
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:210)
    org.eclipse.jetty.websocket.common.BlockingWriteCallback$WriteBlocker.block(BlockingWriteCallback.java:90)
    org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.blockingWrite(WebSocketRemoteEndpoint.java:107)
    org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendString(WebSocketRemoteEndpoint.java:394)
    org.springframework.web.socket.adapter.jetty.JettyWebSocketSession.sendTextMessage(JettyWebSocketSession.java:273)
    org.springframework.web.socket.adapter.AbstractWebSocketSession.sendMessage(AbstractWebSocketSession.java:101)
    com.magdata.exchange.service.websocket.host.model.connection.TextWebSocketConnection.sendMessage(TextWebSocketConnection.java:34)
    com.magdata.exchange.service.websocket.host.model.connection.AccountFundsWebSocketDecorator.sendMessage(AccountFundsWebSocketDecorator.java:33)
    com.magdata.exchange.service.websocket.host.model.connection.BetUpdatesWebSocketDecorator.sendMessage(BetUpdatesWebSocketDecorator.java:41)
    com.magdata.exchange.service.websocket.host.model.connection.ConcurrentWebSocketConnection.tryFlushMessageBuffer(ConcurrentWebSocketConnection.java:72)
    com.magdata.exchange.service.websocket.host.model.connection.ConcurrentWebSocketConnection.sendMessage(ConcurrentWebSocketConnection.java:54)
    com.magdata.exchange.service.websocket.host.sender.AbstractOutgoingMessageProcessor.sendServerMessage(AbstractOutgoingMessageProcessor.java:38)
    com.magdata.exchange.service.websocket.host.sender.ErrorOutgoingMessageProcessor.processMessage(ErrorOutgoingMessageProcessor.java:24)
    com.magdata.exchange.service.websocket.host.sender.WebSocketOutgoingMessageProcessor$Sender.run(WebSocketOutgoingMessageProcessor.java:108)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    java.lang.Thread.run(Thread.java:745)

All 8 threads got stuck with the same issue.

wsh_thread_dump.txt wsh_thread_dump_2.txt

Issues occurs quite periodically on both production WebSocket servers.

joakime commented 6 years ago

@artem-emelin what are your idle timeouts? (connector, http, and websocket)

artem-emelin commented 6 years ago

@joakime Sorry didn't grasp the question. You want to know what are other timeouts for other thread in the stack trace?

djenning90 commented 5 years ago

I'm also experiencing this problem. The steps are easy to repro:

  1. On the client side, create an @ClientEndpoint websocket, having an @onClose method that logs a debug message when the websocket closes.
  2. On the server side, inside the onWebSocketConnect() method, call session.setIdleTimeout(60000) to create a one minute timeout.
  3. The client application is written so that main() connects to the server, creates the client websocket, and then exits, leaving the websocket thread running after the application thread exits.
  4. Launch the client and let it run. Main completes almost immediately.
  5. After one minute you'll see the debug message from the @onClose being displayed, showing that the client websocket was closed: "CloseReason[1001,Idle Timeout]"
  6. Pause the debugger, and observe that it breaks at jdk.internal.misc,park(-1)

Results: The stack trace is: park:-1, Unsafe (jdk.internal.misc) parkNanos:234, LockSupport (java.util.concurrent.locks) awaitNanos:2123, AbstractQueuedSynchronizer$ConditionObject (java.util.concurrent.locks) poll:392, BlockingArrayQueue (org.eclipse.jetty.util) idleJobPoll:656, QueuedThreadPool (org.eclipse.jetty.util.thread) access$800:46, QueuedThreadPool (org.eclipse.jetty.util.thread) run:720, QueuedThreadPool$2 (org.eclipse.jetty.util.thread) run:834, Thread (java.lang)

Note: This is not the same as Bugzilla [#484698] because no underlying network change is required to reproduce it. The problem I'm seeing is a simple case of the websocket handling thread seemingly outliving the websocket.

Any update on when this will be fixed, or advice on how to work around it? Thanks!

sbordet commented 5 years ago

@djenning90 what you report is not a thread hanging in blocking write, but an idle thread in the thread pool, so everything is normal.

djenning90 commented 5 years ago

One more comment: If I program the client side to close the websocket before it exits main(), no threads are left running. The problem only happens (threads are left running after the only websocket in use has been closed) when the websocket that was left open after main() terminated gets closed by the idle timeout.

I'm concerned that every time a websocket times out that a thread might be leaked, as the partially-closed websocket sits in limbo with its thread still running but unable to do anything. For example I can't try to close it forcibly because the websocket no longer has a session.

I'd really appreciate some help resolving this.

djenning90 commented 5 years ago

@sbordet I expected that once the websocket was closed due to idle timeout, that it would no longer exist in the threadpool. If instead I close the websocket programmatically, I observe that it's no longer in the threadpool (because no threads are left running after main() exits in that case). I'm trying to understand why behavior is different if the websocket is closed by idle timeout.

I want to be sure that when my client application quits, that all websockets are terminated and that the threadpool is no longer running. When the application terminates I want there to be no orphaned processes left running.

Perhaps as a workaround, can you show me how at the application level I can issue a call into Jetty to have it flush all resources and close down the threadpool and any other running processes? I don't want there to be any resources left allocated or threads left running when the application terminates.

joakime commented 5 years ago
  • On the client side, create an @ClientEndpoint websocket, having an @onclose method that logs a debug message when the websocket closes.
  • On the server side, inside the onWebSocketConnect() method, call session.setIdleTimeout(60000) to create a one minute timeout.
  • The client application is written so that main() connects to the server, creates the client websocket, and then exits, leaving the websocket thread running after the application thread exits.

This starts a threadpool in daemon mode, threads are still present and running (even the WebSocketClient is still in memory and present)

  • Launch the client and let it run. Main completes almost immediately.

Since you didn't stop() the WebSocketClient it will still have all of the resources you created during start() still present on the JVM.

  • After one minute you'll see the debug message from the @onclose being displayed, showing that the client websocket was closed: "CloseReason[1001,Idle Timeout]"

The thread / jvm connection / jetty connection / jetty endpoint / websocket connection / websocket session are all still present and running.

  • Pause the debugger, and observe that it breaks at jdk.internal.misc,park(-1)

Your example is equivalent to this ...

  1. Start a web browser. (aka websocketclient.start())
  2. Open a web page to a live stock ticker. (aka client.connect())
  3. Alt-tab out of the browser (aka exit the main thread)
sbordet commented 5 years ago

@djenning90 threads and TCP sockets (to which eventually you refer to when you say "close the websocket", etc.) are not related. When you say:

I expected that once the websocket was closed due to idle timeout, that it would no longer exist in the threadpool.

That does not make sense, as the websocket is not pooled in the thread pool.

Seems to me that you are worrying of things that you think are happening but they are not happening because the system works differently than what you think.

Perhaps as a workaround, can you show me how at the application level I can issue a call into Jetty to have it flush all resources and close down the threadpool and any other running processes?

If you are done for good using the WebSocketClient, then WebSocketClient.stop() is what you need.

If you're not convinced and you still have a problem, then we need a clearer description of what you're doing: code snippets, what you do, what happens and what you expect should happen instead. You can call WebSocketClient.dump() to print a detailed representation of the object graph owned by WebSocketClient. This is very helpful to us to understand what's going on.

One last thing: you don't have a problem about blocking writes (whihc is what this issue is about), so would be great that if you still have a problem, you open a new issue and we can continue the discussion there.

djenning90 commented 5 years ago

I guess never mind... obvious workaround is simply for main() to terminate with: System.exit(exitCode);

It still seem reasonable, though, that Jetty should leave no threads running as the application terminates if there are no open/active websocket threads. But workaround is easy.

djenning90 commented 5 years ago

@sbordet and @joakime - I just saw your replies. Thank you for offering that useful information. I didn't know about stop(). I am new to websocket programming and you all have been very helpful in my learning. Thank you!

In my implementation of the websocket POJO, I create the session like this: 

WebSocketContainer container = ContainerProvider.getWebSocketContainer();
Session session = container.connectToServer(this, endpointURI);

And then my websocket is alive. I am not holding a WebSocketClient object in this case. It appears to be created internally by the call to getWebSocketContainer(). I don't know how to get access to the inner WebSocketClient object to close it.

sbordet commented 5 years ago

The JSR API do not define any lifecycle, unfortunately, so you need to use Jetty's APIs:

WebSocketContainer container = ContainerProvider.getWebSocketContainer();
Session session = container.connectToServer(this, endpointURI);

// Use the session

// When done, stop the container. 
LifeCycle.stop(container);
joakime commented 5 years ago

Closing (as duplicate), as all effort here is occurring on Issue #2061