search

wso2 / analytics-apim

Analytics for APIM
Apache License 2.0
55 stars 126 forks source link

Unable to send events to the endpoint Read timeout #1133

Open grego9 opened 4 years ago

grego9 commented 4 years ago

Description: Approximately ten minutes after starting the analytics worker module, the following error appears in APIM log:

TID: [-1] [] [2020-05-14 17:35:50,803] ERROR {org.wso2.carbon.databridge.agent.endpoint.DataEndpoint} -  Unable to send events to the endpoint.  {org.wso2.carbon.databridge.agent.endpoint.DataEndpoint}
org.wso2.carbon.databridge.agent.exception.DataEndpointException: Cannot send Events
        at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.send(ThriftDataEndpoint.java:88)
        at org.wso2.carbon.databridge.agent.endpoint.DataEndpoint$EventPublisher.publish(DataEndpoint.java:314)
        at org.wso2.carbon.databridge.agent.endpoint.DataEndpoint$EventPublisher.run(DataEndpoint.java:272)
        at java.lang.Thread.run(Thread.java:748)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
        at org.wso2.carbon.databridge.commons.thrift.service.general.ThriftEventTransmissionService$Client.recv_publish(ThriftEventTransmissionService.java:168)
        at org.wso2.carbon.databridge.commons.thrift.service.general.ThriftEventTransmissionService$Client.publish(ThriftEventTransmissionService.java:155)
        at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.send(ThriftDataEndpoint.java:81)
        ... 8 more
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
        ... 16 more

I've tried increasing the parameters as described in Tuning Performance with no luck

Affected Product Version: 2.6

OS, DB, other environment details and versions:
CentOS Linux release 7.6.1810 (Core) MySQL 8.0.15 java version "1.8.0_201"

Thanks

ruks commented 4 years ago

Have you checked by increasing the socket timeout in data bridge agent config?