search

kruize / autotune

Autonomous Performance Tuning for Kubernetes!
Apache License 2.0
160 stars 54 forks source link

Stress test failed with java.io.UncheckedIOException: org.eclipse.jetty.io.EofException: Early EOF #974

Open chandrams opened 1 year ago

chandrams commented 1 year ago

Stress test failed on openshift with 100 users 30 days bulk results upload with the below exception in the kruize pod log

Failing build - 0.0.19.2_mvp kruize image (Below exception is not seen with 50 users with this image) Issue is not seen with 0.0.19.1_mvp kruize image

2023-09-2907:42:49.303 ERROR [qtp88397182-44][UpdateResults.java(106)]-Exception: org.eclipse.jetty.io.EofException: Early EOF
2023-09-2907:42:49.304 ERROR [qtp88397182-44][UpdateResults.java(133)]-java.io.UncheckedIOException: org.eclipse.jetty.io.EofException: Early EOF
java.io.UncheckedIOException: org.eclipse.jetty.io.EofException: Early EOF
    at java.base/java.io.BufferedReader$1.hasNext(Unknown Source)
    at java.base/java.util.Iterator.forEachRemaining(Unknown Source)
    at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Unknown Source)
    at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
    at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
    at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
    at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source)
    at com.autotune.analyzer.services.UpdateResults.doPost(UpdateResults.java:70)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
    at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1631)
    at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:319)
    at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:273)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.Server.handle(Server.java:516)
    at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
    at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
    at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.eclipse.jetty.io.EofException: Early EOF
    at org.eclipse.jetty.server.HttpInput$3.getError(HttpInput.java:1207)
    at org.eclipse.jetty.server.HttpInput$3.noContent(HttpInput.java:1195)
    at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:333)
    at java.base/sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
    at java.base/sun.nio.cs.StreamDecoder.implRead(Unknown Source)
    at java.base/sun.nio.cs.StreamDecoder.read(Unknown Source)
    at java.base/java.io.InputStreamReader.read(Unknown Source)
    at java.base/java.io.BufferedReader.fill(Unknown Source)
    at java.base/java.io.BufferedReader.readLine(Unknown Source)
    at java.base/java.io.BufferedReader.readLine(Unknown Source)
    ... 37 more

Steps to recreate the issue:

git clone git@github.com:kruize/autotune.git
cd autotune/tests/scripts/remote_monitoring_tests/stress_test
./remote_monitoring_stress_test.sh -c openshift  -i quay.io/kruize/autotune_operator:0.0.19.2_mvp -u 100 -e 30 -d 120 -t 1200 -r /tmp/stress_test_0.0.19.2_mvp
rbadagandi1 commented 11 months ago

is this still happening @chandrams ?

chandrams commented 11 months ago

is this still happening @chandrams ?

Yes, still seen.