search

SumoLogic / sumologic-collector-docker

A Sumo Logic collector for Docker.
Apache License 2.0
69 stars 55 forks source link

java.net.SocketTimeoutException: Resource temporarily unavailable #103

Open axelwass opened 1 year ago

axelwass commented 1 year ago

Hi

I have installed the Sumologic collector as a Docker container.

I'm running docker on an EC2 instance on AWS. When this exception shows, I stop receiving logs from only one of the containers on the Sumologic console, this application is a Java server using spring. I can see the logs of this application using the docker logs command.

I keep getting this error:

jvm 1 | Exception in thread "onError:DockerLogManager:0000000001565CB5:'Docker-logs':44" java.lang.RuntimeException: java.net.SocketTimeoutException: Resource temporarily unavailable

Full log (until first time the error shows, then the error keeps repeating):

Running SumoLogic Collector...
wrapper  | The version of the script (3.5.49.1) doesn't match the version of this Wrapper (3.5.49). This might cause some problems
wrapper  | --> Wrapper Started as Console
wrapper  | Java Service Wrapper Standard Edition 64-bit 3.5.49
wrapper  |   Copyright (C) 1999-2022 Tanuki Software, Ltd. All Rights Reserved.
wrapper  |     http://wrapper.tanukisoftware.com
wrapper  |   Licensed to Sumo Logic Inc. for Collector
wrapper  | 
wrapper  | Launching a JVM...
jvm 1    | WrapperManager: Initializing...
jvm 1    |    . .       . .       . .    .       . .
jvm 1    | .+'|=|`+. .+'| |`+. .+'|=|`+.=|`+. .+'|=|`+.
jvm 1    | |  | `+.| |  | |  | |  | `+ | `+ | |  | |  |
jvm 1    | |  | .    |  | |  | |  |  | |  | | |  | |  |
jvm 1    | `+.|=|`+. |  | |  | |  |  | |  | | |  | |  |
jvm 1    | .    |  | |  | |  | |  |  | |  | | |  | |  |
jvm 1    | |`+. |  | |  | |  | |  |  | |  | | |  | |  |
jvm 1    | `+.|=|.+' `+.|=|.+' `+.|  |.|  |+' `+.|=|.+'
jvm 1    | Sumo Logic Collector Version 19.403-1
jvm 1    | Sumo Logic Build Hash a125300a564d
jvm 1    | current folder:/opt/SumoCollector
jvm 1    |   * See /opt/SumoCollector/./logs for more details.
jvm 1    | 2022-09-20 14:30:55,787 WrapperSimpleAppMain ERROR Appender Console cannot be located. Route ignored
jvm 1    | 2022-09-20 14:30:55,788 WrapperSimpleAppMain ERROR Appender Console cannot be located. Route ignored
jvm 1    | oslib (Utils.readProcess): Cannot run program "lsb_release": error=2, No such file or directory
jvm 1    | oslib (Utils.readProcess): Cannot run program "lsb_release": error=2, No such file or directory
jvm 1    | (oslib.linux.CommandContainsType) cmd=lsb_release -sd > null
jvm 1    |   * Connecting to https://collectors.de.sumologic.com.
jvm 1    | minor version mismatch: (7!=6) java=1.7.0.0, native=1.6.4.129
jvm 1    | org.hyperic.sigar.SigarException: minor version mismatch: (7!=6) java=1.7.0.0, native=1.6.4.129
jvm 1    |      at org.hyperic.sigar.Sigar.checkVersion(Sigar.java:148)
jvm 1    |      at org.hyperic.sigar.Sigar.<clinit>(Sigar.java:107)
jvm 1    |      at com.sumologic.collector.monitoring.SigarCollectorMetricsReporter.<init>(MetricsReporter.scala:46)
jvm 1    |      at com.sumologic.collector.monitoring.CollectorMetricsLogDumper.<init>(MetricsLogDumper.scala:49)
jvm 1    |      at com.sumologic.scala.collector.Collector$.bootstrap(Collector.scala:1176)
jvm 1    |      at com.sumologic.scala.collector.Collector$.main(Collector.scala:945)
jvm 1    |      at com.sumologic.scala.collector.Collector.main(Collector.scala)
jvm 1    |      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jvm 1    |      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
jvm 1    |      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
jvm 1    |      at java.lang.reflect.Method.invoke(Method.java:498)
jvm 1    |      at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:349)
jvm 1    |      at java.lang.Thread.run(Thread.java:750)
jvm 1    |   * Retrieved configuration from service.
jvm 1    | Exception in thread "onError:DockerLogManager:0000000001565CB5:'Docker-logs':44" java.lang.RuntimeException: java.net.SocketTimeoutException: Resource temporarily unavailable
jvm 1    |      at com.sumologic.collector.blade.docker.DockerMonitor$NativeResultCallback.onError(DockerMonitor.scala:56)
jvm 1    |      at com.github.dockerjava.core.exec.AbstrAsyncDockerCmdExec$1.onError(AbstrAsyncDockerCmdExec.java:46)
jvm 1    |      at com.github.dockerjava.jaxrs.async.AbstractCallbackNotifier.call(AbstractCallbackNotifier.java:54)
jvm 1    |      at java.lang.Thread.run(Thread.java:750)
jvm 1    | Caused by: java.net.SocketTimeoutException: Resource temporarily unavailable
jvm 1    |      at org.newsclub.net.unix.NativeUnixSocket.read(Native Method)
jvm 1    |      at org.newsclub.net.unix.AFUNIXSocketImpl$AFUNIXInputStream.read(AFUNIXSocketImpl.java:325)
jvm 1    |      at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
jvm 1    |      at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
jvm 1    |      at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
jvm 1    |      at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:261)
jvm 1    |      at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:222)
jvm 1    |      at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:183)
jvm 1    |      at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
jvm 1    |      at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
jvm 1    |      at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
jvm 1    |      at java.io.FilterInputStream.read(FilterInputStream.java:83)
jvm 1    |      at org.glassfish.jersey.message.internal.EntityInputStream.isEmpty(EntityInputStream.java:165)
jvm 1    |      at org.glassfish.jersey.message.internal.InboundMessageContext.hasEntity(InboundMessageContext.java:780)
jvm 1    |      at com.github.dockerjava.jaxrs.filter.LoggingFilter.filter(LoggingFilter.java:249)
jvm 1    |      at org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:171)
jvm 1    |      at org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:159)
jvm 1    |      at org.glassfish.jersey.process.internal.Stages.process(Stages.java:147)
jvm 1    |      at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:301)
jvm 1    |      at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:609)
jvm 1    |      at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
jvm 1    |      at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
jvm 1    |      at org.glassfish.jersey.internal.Errors.process(Errors.java:205)
jvm 1    |      at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390)
jvm 1    |      at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:608)
jvm 1    |      at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:395)
jvm 1    |      at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:295)
jvm 1    |      at com.github.dockerjava.jaxrs.async.GETCallbackNotifier.response(GETCallbackNotifier.java:26)
jvm 1    |      at com.github.dockerjava.jaxrs.async.AbstractCallbackNotifier.call(AbstractCallbackNotifier.java:51)
jvm 1    |      ... 1 more
jvm 1    | Exception in thread "onError:DockerLogInput:0000000001565CB5:'Docker-logs':d47cd9470db84675a2e44b21159992f5710bbf6a287e6d684415eadffc657264:sumologic_docker_agent_staging:50" Exception in thread "onError:DockerStatsManager:0000000001565CB6:'Docker-stats':56" java.lang.RuntimeException: java.lang.RuntimeException: java.net.SocketTimeoutException: Resource temporarily unavailable
jvm 1    |      at com.sumologic.collector.blade.docker.DockerMonitor$NativeResultCallback.onError(DockerMonitor.scala:56)
jvm 1    |      at com.github.dockerjava.core.exec.AbstrAsyncDockerCmdExec$1.onError(AbstrAsyncDockerCmdExec.java:46)
jvm 1    |      at com.github.dockerjava.jaxrs.async.AbstractCallbackNotifier.call(AbstractCallbackNotifier.java:76)
jvm 1    |      at java.lang.Thread.run(Thread.java:750)
jvm 1    | Caused by: java.lang.RuntimeException: java.net.SocketTimeoutException: Resource temporarily unavailable
jvm 1    |      at com.sumologic.collector.blade.docker.DockerMonitor$NativeResultCallback.onError(DockerMonitor.scala:56)
jvm 1    |      at com.github.dockerjava.core.exec.AbstrAsyncDockerCmdExec$1.onError(AbstrAsyncDockerCmdExec.java:46)
jvm 1    |      at com.github.dockerjava.core.async.FrameStreamProcessor.processResponseStream(FrameStreamProcessor.java:39)
jvm 1    |      at com.github.dockerjava.jaxrs.async.AbstractCallbackNotifier.call(AbstractCallbackNotifier.java:70)
jvm 1    |      ... 1 more
jvm 1    | Caused by: java.net.SocketTimeoutException: Resource temporarily unavailable
jvm 1    |      at org.newsclub.net.unix.NativeUnixSocket.read(Native Method)
jvm 1    |      at org.newsclub.net.unix.AFUNIXSocketImpl$AFUNIXInputStream.read(AFUNIXSocketImpl.java:325)
jvm 1    |      at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
jvm 1    |      at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
jvm 1    |      at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
jvm 1    |      at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:261)
jvm 1    |      at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:222)
jvm 1    |      at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:183)
jvm 1    |      at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
jvm 1    |      at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
jvm 1    |      at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
jvm 1    |      at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
jvm 1    |      at java.io.FilterInputStream.read(FilterInputStream.java:133)
jvm 1    |      at org.glassfish.jersey.message.internal.EntityInputStream.read(EntityInputStream.java:79)
jvm 1    |      at com.github.dockerjava.jaxrs.util.WrappedResponseInputStream.read(WrappedResponseInputStream.java:45)
jvm 1    |      at com.github.dockerjava.core.command.FrameReader.readFrame(FrameReader.java:64)
jvm 1    |      at com.github.dockerjava.core.async.FrameStreamProcessor.processResponseStream(FrameStreamProcessor.java:35)
jvm 1    |      ... 2 more
jvm 1    | java.lang.RuntimeException: java.net.SocketTimeoutException: Resource temporarily unavailable
jvm 1    |      at com.sumologic.collector.blade.docker.DockerMonitor$NativeResultCallback.onError(DockerMonitor.scala:56)
jvm 1    |      at com.github.dockerjava.core.exec.AbstrAsyncDockerCmdExec$1.onError(AbstrAsyncDockerCmdExec.java:46)
jvm 1    |      at com.github.dockerjava.jaxrs.async.AbstractCallbackNotifier.call(AbstractCallbackNotifier.java:54)
jvm 1    |      at java.lang.Thread.run(Thread.java:750)
jvm 1    | Caused by: java.net.SocketTimeoutException: Resource temporarily unavailable
jvm 1    |      at org.newsclub.net.unix.NativeUnixSocket.read(Native Method)
jvm 1    |      at org.newsclub.net.unix.AFUNIXSocketImpl$AFUNIXInputStream.read(AFUNIXSocketImpl.java:325)
jvm 1    |      at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
jvm 1    |      at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
jvm 1    |      at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
jvm 1    |      at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:261)
jvm 1    |      at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:222)
jvm 1    |      at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:183)
jvm 1    |      at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
jvm 1    |      at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
jvm 1    |      at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
jvm 1    |      at java.io.FilterInputStream.read(FilterInputStream.java:83)
jvm 1    |      at org.glassfish.jersey.message.internal.EntityInputStream.isEmpty(EntityInputStream.java:165)
jvm 1    |      at org.glassfish.jersey.message.internal.InboundMessageContext.hasEntity(InboundMessageContext.java:780)
jvm 1    |      at com.github.dockerjava.jaxrs.filter.LoggingFilter.filter(LoggingFilter.java:249)
jvm 1    |      at org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:171)
jvm 1    |      at org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:159)
jvm 1    |      at org.glassfish.jersey.process.internal.Stages.process(Stages.java:147)
jvm 1    |      at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:301)
jvm 1    |      at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:609)
jvm 1    |      at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
jvm 1    |      at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
jvm 1    |      at org.glassfish.jersey.internal.Errors.process(Errors.java:205)
jvm 1    |      at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390)
jvm 1    |      at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:608)
jvm 1    |      at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:395)
jvm 1    |      at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:295)
jvm 1    |      at com.github.dockerjava.jaxrs.async.GETCallbackNotifier.response(GETCallbackNotifier.java:26)
jvm 1    |      at com.github.dockerjava.jaxrs.async.AbstractCallbackNotifier.call(AbstractCallbackNotifier.java:51)
jvm 1    |      ... 1 more
that-kampe-steak commented 1 year ago

seeing the same issues with ubuntu:22.10 as there are many vulnerabilities in that ubuntu:18 base image, can we get an update on this?

dsjoolbl commented 10 months ago

has there been any progress on the resolution of this issue? We are also experiencing problems with the same error in the logs on just one sumo logic container on one out of three swarm nodes