Repeated Warning and system lag during client stop and close

[kprabha7]

Jetty version(s) 9.3.0

Java version/vendor (use: java -version) >=8

OS type/version Android 9

Description QueuedThreadPool: WebSocketClient@122503328{STOPPING,8<=8<=200,i=5,q=7} Couldn't stop Thread[WebSocketClient@122503328-1556,5,main] QueuedThreadPool: WebSocketClient@122503328{STOPPING,8<=8<=200,i=4,q=7} Couldn't stop Thread[WebSocketClient@122503328-1557,5,main] QueuedThreadPool: WebSocketClient@122503328{STOPPING,8<=8<=200,i=4,q=7} Couldn't stop Thread[WebSocketClient@122503328-1560,5,main] QueuedThreadPool: WebSocketClient@122503328{STOPPING,8<=8<=200,i=4,q=7} Couldn't stop Thread[WebSocketClient@122503328-1561,5,main] QueuedThreadPool: WebSocketClient@122503328{STOPPING,8<=8<=200,i=4,q=7} Couldn't stop Thread[WebSocketClient@122503328-1563,5,main]

The above warning log is seen repeatedly and system performance is impacted when we try to stop the client using WebSocketClient stop() method. The stop timeout is set to 0.

This is occurring when server application on 3rd party machine is down and connection is refused since no server is listening on destination port. ConnectException is seen in onError callback

How to reproduce? client must try to open connection with remote machine that is up and server application is not up in remote machine on required port. client object clean up should be done in onError callback using client.stop() and client.destroy().

[joakime]

Jetty 9.3.0 is deprecated / end of life.

Can you try it again with (the still supported) Jetty 9.4.x tree? (version 9.4.42.v20210604 was released last month) The WebSocketClient has some major improvements in that version tree.

[joakime]

client object clean up should be done in onError callback using client.stop() and client.destroy().

If you are doing this from the websocket's onError callback, that's the reason for the failure to stop. You cannot use a thread from the threadpool to kill/stop that same threadpool. You would need to be using a different thread entirely.

[kprabha7]

Thanks @joakime

by onError, I mean the method with @OnWebSocketError annotation. Are you also referring to the same method? Just want to confirm

[joakime]

Yes, I'm referring to the same thing.

That annotated method is being called by WebSocketClient from a thread that comes from the Thread Pool.

[joakime]

Also, keep in mind that WebSocketClient should be treated the same way you would a Web Browser. It's not meant to be started/stopped over and over and over again. If you are just wanting to recover and try again, just tell the WebSocket Session to disconnect() (note: this disconnect() feature might not exist on Jetty 9.3.x, but definitely does on 9.4.x) which will kill that connection, and then you issue another websocket request to start a new websocket session.

[kprabha7]

Till now, in our debugging efforts, it looks like Jetty Websocket version is the reason. Upgrading to version 9.4.4, I was able to see that this warning is not seen in similar scenario or test case.

We also made some low level design changes when it comes to clean up after onError. Client disconnect and stop are now done from the main service thread (this is different from thread spawned by Jetty client). But, doing only this change didn't help. We were able to see this warning again and multiple threads remained in system. So, root cause is unclear to me.

Now, moving the close or clean up logic looks like code or design improvement and Jetty version upgrade seems to fix the issue.

[janbartel]

Closing this issue, as it seems to have been addressed. If you still have concerns, please reopen with fresh info.