search

inspectIT / inspectit-ocelot

inspectIT Ocelot - Java agent for collecting application performance, tracing and behavior data
http://www.inspectit.rocks/
Apache License 2.0
204 stars 69 forks source link

[Bug] - AgentCommandService throws "Read timed out" exception after agent commands have been disabled #1542

Open heiko-holz opened 1 year ago

heiko-holz commented 1 year ago

Describe the bug When I enable the agent commands (and/or log-preloading) in the configuration server and then later disable them, I receive the following exception:

2022-11-15 11:40:51,139 ERROR 382527 --- [inspectIT] [pectit-thread-1] r.i.o.core.command.HttpCommandFetcher    : An error occurred while fetching an agent command.
java.net.SocketTimeoutException: Read timed out
    at java.base/java.net.SocketInputStream.socketRead0(Native Method)
    at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
    at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
    at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
    at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
    at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
    at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
    at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
    at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
    at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
    at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
    at rocks.inspectit.ocelot.core.command.HttpCommandFetcher.fetchCommand(HttpCommandFetcher.java:129)
    at rocks.inspectit.ocelot.core.command.CommandHandler.nextCommand(CommandHandler.java:72)
    at rocks.inspectit.ocelot.core.command.CommandHandler.nextCommand(CommandHandler.java:57)
    at rocks.inspectit.ocelot.core.command.AgentCommandService.run(AgentCommandService.java:93)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)

The exceptions are also thrown for a while, even if I re-enable the agent commands.

To Reproduce Steps to reproduce the behavior:

  1. enable the agent commands
  2. wait until the agent commands are enabled (e.g., wait until you can get the logs from the Agent Status Table)
  3. disable the agent commands
  4. See error

Expected behavior I would not expect exceptions to be thrown if I disable agent commands

danipaniii commented 1 year ago

While trying to find the cause of the problem, I figured out that the problem lays most likely at the "Show Logs" button. It is possible for me to switch the agent commands on/off without any error message. Then I noticed that the above error message always gets thrown approximately 30 seconds after pushing the "Show Logs" button.