search

BingAds / BingAds-Java-SDK

Other
43 stars 48 forks source link

Connections time out rather often #158

Open markusheiden opened 1 year ago

markusheiden commented 1 year ago

For some time we already noticed that the connections to the Bing Ads API time out rather often. Is that a load-limiting behavior or a bug? Why isn't that handled by the ReportingServiceManager via a meaningful exception?

Are there any limits on how many connections can be opened in parallel? What are the exact restrictions?

Stacktrace:

com.microsoft.bingads.v13.reporting.CouldNotSubmitReportingDownloadException: java.util.concurrent.ExecutionException: com.sun.xml.ws.client.ClientTransportException: HTTP transport error: java.net.ConnectException: Connection timed out
    at com.microsoft.bingads.v13.reporting.ReportingServiceManager$4.handleResponse(ReportingServiceManager.java:247)
    at com.sun.xml.ws.client.AsyncResponseImpl.set(AsyncResponseImpl.java:104)
    at com.sun.xml.ws.client.sei.AsyncMethodHandler$SEIAsyncInvoker$1.onCompletion(AsyncMethodHandler.java:187)
    at com.sun.xml.ws.client.Stub$1.onCompletion(Stub.java:535)
    at com.sun.xml.ws.api.pipe.Fiber.completionCheck(Fiber.java:895)
    at com.sun.xml.ws.api.pipe.Fiber.run(Fiber.java:792)
    at com.sun.xml.ws.api.server.ThreadLocalContainerResolver$2$1.run(ThreadLocalContainerResolver.java:89)
    ... 3 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.sun.xml.ws.client.ClientTransportException: HTTP transport error: java.net.ConnectException: Connection timed out
    at com.sun.xml.ws.util.CompletedFuture.get(CompletedFuture.java:50)
    at com.microsoft.bingads.v13.reporting.ReportingServiceManager$4.handleResponse(ReportingServiceManager.java:233)
    ... 9 common frames omitted
Caused by: com.sun.xml.ws.client.ClientTransportException: HTTP transport error: java.net.ConnectException: Connection timed out
    at com.sun.xml.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:103)
    at com.sun.xml.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:209)
    at com.sun.xml.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:131)
    at com.sun.xml.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:111)
    at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:1106)
    at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:1020)
    at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:989)
    at com.sun.xml.ws.api.pipe.Fiber.run(Fiber.java:782)
    ... 4 common frames omitted
Caused by: java.net.ConnectException: Connection timed out
    at java.base/sun.nio.ch.Net.connect0(Native Method)
    at java.base/sun.nio.ch.Net.connect(Unknown Source)
    at java.base/sun.nio.ch.Net.connect(Unknown Source)
    at java.base/sun.nio.ch.NioSocketImpl.connect(Unknown Source)
    at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
    at java.base/java.net.Socket.connect(Unknown Source)
    at java.base/sun.security.ssl.SSLSocketImpl.connect(Unknown Source)
    at java.base/sun.security.ssl.BaseSSLSocketImpl.connect(Unknown Source)
    at java.base/sun.net.NetworkClient.doConnect(Unknown Source)
    at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
    at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
    at java.base/sun.net.www.protocol.https.HttpsClient.<init>(Unknown Source)
    at java.base/sun.net.www.protocol.https.HttpsClient.New(Unknown Source)
    at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(Unknown Source)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(Unknown Source)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
    at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(Unknown Source)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(Unknown Source)
    at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(Unknown Source)
    at com.sun.xml.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:90)
    ... 11 common frames omitted
qitia commented 1 year ago

hi @markusheiden thanks for reporting this. I'm trying to repro it. Can you continuously reproduce this issue? and did you find and steps to repro?

markusheiden commented 1 year ago

We are using many (12) threads in parallel to access the API on each instance (> 10 independent instances). This most often happens when we download KeywordPerformanceReportRequests reports via the ReportingServiceManager. I reduced the parallelism to 6 threads and the problem was reduced significantly and currently just happens occasionally.

My problem is that I don't know what the limit for parallel access to the API is. What is the exact limit? How is the limit counted: Per developer token, per application (client ID), or per customer?

The current behavior (simply not answering) makes it hard to reliably use the API with parallelism. Without parallelism, the API access would take too long. The Google Ads API does this a bit better: They answer with an error that contains the delay that is needed before we may access the API again. Is there something similar in the Bing Ads API too?

qitia commented 1 year ago

hi there, this error does not seem like related to service side throttling, otherwise client will receive a RATE_EXCEEDED user error. A quick search shows there could be various reasons for a connection time out error: https://stackoverflow.com/questions/86824/why-would-a-java-net-connectexception-connection-timed-out-exception-occur-wh. Maybe setup a retry mechanism to work around this in the time we trouble shooting it?

markusheiden commented 1 year ago

@qitia Thanks for clarifying that.

We already retry 3 times. Because reducing the load reduces the problem, it has to be some kind of load problem.

Maybe the connection timeout is too low under load conditions. How can I increase that?

qitia commented 1 year ago

https://stackoverflow.com/questions/2148915/how-do-i-set-the-timeout-for-a-jax-ws-webservice-client

does above solution helps?

markusheiden commented 1 year ago

@qitia Thanks, we will try that.

But that solution will not suffice, because when the Bing Ads SDK uses the Apache HTTP Client for the final report download, there is the same problem:

com.microsoft.bingads.CouldNotDownloadResultFileException: org.apache.http.conn.ConnectTimeoutException: Connect to bingadsappsstorageprod.blob.core.windows.net:443 [bingadsappsstorageprod.blob.core.windows.net/20.150.126.4] failed: Connect timed out
    at com.microsoft.bingads.internal.utilities.HttpClientHttpFileService.downloadFile(HttpClientHttpFileService.java:65)
    at com.microsoft.bingads.v13.bulk.BulkOperation.downloadResultFileZip(BulkOperation.java:346)
    at com.microsoft.bingads.v13.bulk.BulkOperation.downloadFileWithFinalStatus(BulkOperation.java:326)
    at com.microsoft.bingads.v13.bulk.BulkOperation.downloadResultFileAsyncImpl(BulkOperation.java:299)
    at com.microsoft.bingads.v13.bulk.BulkOperation.downloadResultFileAsync(BulkOperation.java:275)
    at com.microsoft.bingads.v13.bulk.BulkServiceManager.downloadBulkFileAsync(BulkServiceManager.java:519)
    at com.microsoft.bingads.v13.bulk.BulkServiceManager.access$500(BulkServiceManager.java:63)
    at com.microsoft.bingads.v13.bulk.BulkServiceManager$5$1.onSuccess(BulkServiceManager.java:490)
    at com.microsoft.bingads.v13.bulk.BulkServiceManager$5$1.onSuccess(BulkServiceManager.java:487)
    at com.microsoft.bingads.internal.ParentCallback.onCompleted(ParentCallback.java:25)
    at com.microsoft.bingads.internal.ResultFuture.setResult(ResultFuture.java:26)
    at com.microsoft.bingads.v13.bulk.BulkOperation$1.onSuccess(BulkOperation.java:129)
    at com.microsoft.bingads.v13.bulk.BulkOperation$1.onSuccess(BulkOperation.java:124)
    at com.microsoft.bingads.internal.ParentCallback.onCompleted(ParentCallback.java:25)
    at com.microsoft.bingads.internal.ResultFuture.setResult(ResultFuture.java:26)
    at com.microsoft.bingads.v13.internal.bulk.PollingBulkOperationTracker.completeTaskWithResult(PollingBulkOperationTracker.java:197)
    at com.microsoft.bingads.v13.internal.bulk.PollingBulkOperationTracker.completeTaskIfOperationIsComplete(PollingBulkOperationTracker.java:192)
    at com.microsoft.bingads.v13.internal.bulk.PollingBulkOperationTracker.access$200(PollingBulkOperationTracker.java:25)
    at com.microsoft.bingads.v13.internal.bulk.PollingBulkOperationTracker$3.onCompleted(PollingBulkOperationTracker.java:107)
    at com.microsoft.bingads.internal.ResultFuture.setResult(ResultFuture.java:26)
    at com.microsoft.bingads.v13.internal.bulk.PollingBulkOperationTracker$6.accept(PollingBulkOperationTracker.java:221)
    at com.microsoft.bingads.v13.internal.bulk.PollingBulkOperationTracker$6.accept(PollingBulkOperationTracker.java:216)
    at com.microsoft.bingads.internal.OperationStatusRetry$1.onCompleted(OperationStatusRetry.java:49)
    at com.microsoft.bingads.internal.ResultFuture.setResult(ResultFuture.java:26)
    at com.microsoft.bingads.v13.internal.bulk.DownloadStatusProvider$1.handleResponse(DownloadStatusProvider.java:54)
    at org.apache.cxf.jaxws.JaxwsClientCallback.handleResponse(JaxwsClientCallback.java:44)
    at org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:855)
    at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1738)
    at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream$1.run(HTTPConduit.java:1216)
    at org.apache.cxf.workqueue.AutomaticWorkQueueImpl$3.run(AutomaticWorkQueueImpl.java:413)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at org.apache.cxf.workqueue.AutomaticWorkQueueImpl$AWQThreadFactory$1.run(AutomaticWorkQueueImpl.java:346)
    at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to bingadsappsstorageprod.blob.core.windows.net:443 [bingadsappsstorageprod.blob.core.windows.net/20.150.126.4] failed: Connect timed out
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
    at com.microsoft.bingads.internal.utilities.HttpClientHttpFileService.downloadFile(HttpClientHttpFileService.java:50)
    ... 33 common frames omitted
Caused by: java.net.SocketTimeoutException: Connect timed out
    at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(Unknown Source)
    at java.base/sun.nio.ch.NioSocketImpl.connect(Unknown Source)
    at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
    at java.base/java.net.Socket.connect(Unknown Source)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:368)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
    ... 43 common frames omitted
markusheiden commented 1 year ago

According to the CXF docs, the connection timeout can be configured via the cxf.xml in the root of the classpath.

See https://cxf.apache.org/docs/client-http-transport-including-ssl-support.html#ClientHTTPTransport(includingSSLsupport)-Theconduitelement for the config. For the default value of the "ConnectionTimeout" property (just 30 seconds) see org.apache.cxf.transports.http.configuration.HTTPClientPolicy#getConnectionTimeout() .

This configures the connection timeout to 15 minutes because 5 minutes were to be not enough for us:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:http="http://cxf.apache.org/transports/http/configuration"
       xsi:schemaLocation="
         http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
         http://cxf.apache.org/transports/http/configuration http://cxf.apache.org/schemas/configuration/http-conf.xsd">
  <http:conduit name="{https://bingads.microsoft.com/Reporting/v13}IReportingServicePort.http-conduit">
    <http:client ConnectionTimeout="900000" />
  </http:conduit>
</beans>

Instead of {https://bingads.microsoft.com/Reporting/v13}IReportingServicePort.http-conduit *.http-conduit could be used above, if no one else uses CXF. That would be more forward-compatible for new major API versions.

markusheiden commented 1 year ago

Another one that happened multiple times:

java.net.ConnectException: HTTP connect timed out
    at jdk.internal.net.http.ResponseTimerEvent.handle
    at jdk.internal.net.http.HttpClientImpl.purgeTimeoutsAndReturnNextDeadline
    at jdk.internal.net.http.HttpClientImpl$SelectorManager.run
java.net.http.HttpConnectTimeoutException: HTTP connect timed out
    at jdk.internal.net.http.ResponseTimerEvent.handle
    at jdk.internal.net.http.HttpClientImpl.purgeTimeoutsAndReturnNextDeadline
    at jdk.internal.net.http.HttpClientImpl$SelectorManager.run
org.apache.cxf.interceptor.Fault: Could not send Message.
    at org.apache.cxf.transport.http.HttpClientHTTPConduit$HttpClientWrappedOutputStream.isConnectionAttemptCompleted(HttpClientHTTPConduit.java:436)
    at org.apache.cxf.transport.http.HttpClientHTTPConduit$HttpClientWrappedOutputStream$1.canWrite(HttpClientHTTPConduit.java:464)
    at org.apache.cxf.transport.http.HttpClientHTTPConduit$HttpClientWrappedOutputStream$1.write(HttpClientHTTPConduit.java:474)
    at org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:51)
    at org.apache.cxf.io.AbstractThresholdOutputStream.write(AbstractThresholdOutputStream.java:69)
    at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1405)
    at org.apache.cxf.io.CacheAndWriteOutputStream.postClose(CacheAndWriteOutputStream.java:56)
    at org.apache.cxf.io.CachedOutputStream.close(CachedOutputStream.java:228)
    at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
    at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:717)
    at org.apache.cxf.transport.http.HttpClientHTTPConduit.close(HttpClientHTTPConduit.java:112)
    at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:63)
    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
    at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:528)
    at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:432)
    at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:410)
    at org.apache.cxf.jaxws.JaxWsClientProxy.invokeAsync(JaxWsClientProxy.java:326)
    at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:138)
    at jdk.proxy2.$Proxy388.downloadCampaignsByAccountIdsAsync
    at com.microsoft.bingads.v13.bulk.BulkServiceManager.submitDownloadAsync(BulkServiceManager.java:561)
    at com.microsoft.bingads.v13.bulk.BulkServiceManager.downloadFileAsyncImpl(BulkServiceManager.java:484)
    at com.microsoft.bingads.v13.bulk.BulkServiceManager.downloadEntitiesAsyncImpl(BulkServiceManager.java:435)
    at com.microsoft.bingads.v13.bulk.BulkServiceManager.downloadEntitiesAsync(BulkServiceManager.java:163)
java.util.concurrent.ExecutionException: org.apache.cxf.interceptor.Fault: Could not send Message.
    at org.apache.cxf.jaxws.JaxwsClientCallback$2.get(JaxwsClientCallback.java:103)
    at com.microsoft.bingads.v13.bulk.BulkServiceManager$7.handleResponse(BulkServiceManager.java:567)
    at org.apache.cxf.jaxws.JaxwsClientCallback.handleException(JaxwsClientCallback.java:90)
    at org.apache.cxf.interceptor.ClientOutFaultObserver.onMessage(ClientOutFaultObserver.java:61)
    at org.apache.cxf.endpoint.ClientImpl$2.onMessage(ClientImpl.java:519)
    at org.apache.cxf.phase.PhaseInterceptorChain.wrapExceptionAsFault(PhaseInterceptorChain.java:373)
    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:331)
    at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:528)
    at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:432)
    at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:410)
    at org.apache.cxf.jaxws.JaxWsClientProxy.invokeAsync(JaxWsClientProxy.java:326)
    at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:138)
    at jdk.proxy2.$Proxy388.downloadCampaignsByAccountIdsAsync
    at com.microsoft.bingads.v13.bulk.BulkServiceManager.submitDownloadAsync(BulkServiceManager.java:561)
    at com.microsoft.bingads.v13.bulk.BulkServiceManager.downloadFileAsyncImpl(BulkServiceManager.java:484)
    at com.microsoft.bingads.v13.bulk.BulkServiceManager.downloadEntitiesAsyncImpl(BulkServiceManager.java:435)
    at com.microsoft.bingads.v13.bulk.BulkServiceManager.downloadEntitiesAsync(BulkServiceManager.java:163)
com.microsoft.bingads.v13.bulk.CouldNotSubmitBulkDownloadException: java.util.concurrent.ExecutionException: org.apache.cxf.interceptor.Fault: Could not send Message.
    at com.microsoft.bingads.v13.bulk.BulkServiceManager$7.handleResponse(BulkServiceManager.java:582)
    at org.apache.cxf.jaxws.JaxwsClientCallback.handleException(JaxwsClientCallback.java:90)
    at org.apache.cxf.interceptor.ClientOutFaultObserver.onMessage(ClientOutFaultObserver.java:61)
    at org.apache.cxf.endpoint.ClientImpl$2.onMessage(ClientImpl.java:519)
    at org.apache.cxf.phase.PhaseInterceptorChain.wrapExceptionAsFault(PhaseInterceptorChain.java:373)
    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:331)
    at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:528)
    at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:432)
    at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:410)
    at org.apache.cxf.jaxws.JaxWsClientProxy.invokeAsync(JaxWsClientProxy.java:326)
    at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:138)
    at jdk.proxy2.$Proxy388.downloadCampaignsByAccountIdsAsync
    at com.microsoft.bingads.v13.bulk.BulkServiceManager.submitDownloadAsync(BulkServiceManager.java:561)
    at com.microsoft.bingads.v13.bulk.BulkServiceManager.downloadFileAsyncImpl(BulkServiceManager.java:484)
    at com.microsoft.bingads.v13.bulk.BulkServiceManager.downloadEntitiesAsyncImpl(BulkServiceManager.java:435)
    at com.microsoft.bingads.v13.bulk.BulkServiceManager.downloadEntitiesAsync(BulkServiceManager.java:163)
markusheiden commented 1 year ago

Another one:

java.net.ConnectException: HTTP connect timed out
    at jdk.internal.net.http.ResponseTimerEvent.handle
    at jdk.internal.net.http.HttpClientImpl.purgeTimeoutsAndReturnNextDeadline
    at jdk.internal.net.http.HttpClientImpl$SelectorManager.run
java.net.http.HttpConnectTimeoutException: HTTP connect timed out
    at jdk.internal.net.http.ResponseTimerEvent.handle
    at jdk.internal.net.http.HttpClientImpl.purgeTimeoutsAndReturnNextDeadline
    at jdk.internal.net.http.HttpClientImpl$SelectorManager.run
org.apache.cxf.interceptor.Fault: Could not send Message.
    at org.apache.cxf.transport.http.HttpClientHTTPConduit$HttpClientWrappedOutputStream.isConnectionAttemptCompleted(HttpClientHTTPConduit.java:436)
    at org.apache.cxf.transport.http.HttpClientHTTPConduit$HttpClientWrappedOutputStream$1.canWrite(HttpClientHTTPConduit.java:464)
    at org.apache.cxf.transport.http.HttpClientHTTPConduit$HttpClientWrappedOutputStream$1.write(HttpClientHTTPConduit.java:474)
    at org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:51)
    at org.apache.cxf.io.AbstractThresholdOutputStream.unBuffer(AbstractThresholdOutputStream.java:89)
    at org.apache.cxf.io.AbstractThresholdOutputStream.write(AbstractThresholdOutputStream.java:63)
    at org.apache.cxf.io.CacheAndWriteOutputStream.write(CacheAndWriteOutputStream.java:81)
    at org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:51)
    at com.ctc.wstx.io.UTF8Writer.write(UTF8Writer.java:143)
    at com.ctc.wstx.sw.BufferingXmlWriter.writeRaw(BufferingXmlWriter.java:286)
    at com.ctc.wstx.sw.BufferingXmlWriter.writeCharacters(BufferingXmlWriter.java:600)
    at com.ctc.wstx.sw.BaseStreamWriter.writeCharacters(BaseStreamWriter.java:467)
    at org.apache.cxf.staxutils.StaxUtils.copy(StaxUtils.java:759)
    at org.apache.cxf.staxutils.StaxUtils.copy(StaxUtils.java:707)
    at org.apache.cxf.binding.soap.saaj.SAAJOutInterceptor$SAAJOutEndingInterceptor.handleMessage(SAAJOutInterceptor.java:213)
    at org.apache.cxf.binding.soap.saaj.SAAJOutInterceptor$SAAJOutEndingInterceptor.handleMessage(SAAJOutInterceptor.java:173)
    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
    at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:528)
    at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:432)
    at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:410)
    at org.apache.cxf.jaxws.JaxWsClientProxy.invokeAsync(JaxWsClientProxy.java:326)
    at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:138)
    at jdk.proxy2.$Proxy388.uploadEntityRecordsAsync
    at com.microsoft.bingads.v13.bulk.BulkServiceManager.uploadEntityRecordsImpl(BulkServiceManager.java:197)
    at com.microsoft.bingads.v13.bulk.BulkServiceManager.uploadEntitiesAsync(BulkServiceManager.java:182)
markusheiden commented 1 year ago

Another one:

java.net.ConnectException: HTTP connect timed out
    at jdk.internal.net.http.ResponseTimerEvent.handle
    at jdk.internal.net.http.HttpClientImpl.purgeTimeoutsAndReturnNextDeadline
    at jdk.internal.net.http.HttpClientImpl$SelectorManager.run
java.net.http.HttpConnectTimeoutException: HTTP connect timed out
    at jdk.internal.net.http.ResponseTimerEvent.handle
    at jdk.internal.net.http.HttpClientImpl.purgeTimeoutsAndReturnNextDeadline
    at jdk.internal.net.http.HttpClientImpl$SelectorManager.run
org.apache.cxf.interceptor.Fault: Could not send Message.
    at org.apache.cxf.transport.http.HttpClientHTTPConduit$HttpClientWrappedOutputStream.isConnectionAttemptCompleted(HttpClientHTTPConduit.java:436)
    at org.apache.cxf.transport.http.HttpClientHTTPConduit$HttpClientWrappedOutputStream$1.canWrite(HttpClientHTTPConduit.java:464)
    at org.apache.cxf.transport.http.HttpClientHTTPConduit$HttpClientWrappedOutputStream$1.write(HttpClientHTTPConduit.java:474)
    at org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:51)
    at org.apache.cxf.io.AbstractThresholdOutputStream.unBuffer(AbstractThresholdOutputStream.java:89)
    at org.apache.cxf.io.AbstractThresholdOutputStream.write(AbstractThresholdOutputStream.java:63)
    at org.apache.cxf.io.CacheAndWriteOutputStream.write(CacheAndWriteOutputStream.java:81)
    at org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:51)
    at com.ctc.wstx.io.UTF8Writer.flush(UTF8Writer.java:100)
    at com.ctc.wstx.sw.BufferingXmlWriter.flush(BufferingXmlWriter.java:242)
    at com.ctc.wstx.sw.BaseStreamWriter.flush(BaseStreamWriter.java:260)
    at org.apache.cxf.binding.soap.saaj.SAAJOutInterceptor$SAAJOutEndingInterceptor.handleMessage(SAAJOutInterceptor.java:214)
    at org.apache.cxf.binding.soap.saaj.SAAJOutInterceptor$SAAJOutEndingInterceptor.handleMessage(SAAJOutInterceptor.java:173)
    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
    at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:528)
    at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:432)
    at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:410)
    at org.apache.cxf.jaxws.JaxWsClientProxy.invokeAsync(JaxWsClientProxy.java:326)
    at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:138)
    at jdk.proxy2.$Proxy385.submitGenerateReportAsync
    at com.microsoft.bingads.v13.reporting.ReportingServiceManager$submitDownloadAsync$1(ReportingServiceManager.java:xxx)
qitia commented 1 year ago

hi @markusheiden does it help to config the timeout per your previous post?

markusheiden commented 1 year ago

No, the connection timeout had already been set to 15 minutes for the above requests in the cxf.xml:

<beans ...>
  <http:conduit name="*.http-conduit">
    <http:client ConnectionTimeout="900000" />
  </http:conduit>
</beans>

This config is just for the SOAP requests. I have not changed the connection timeout for the Apache HTTP client used for all other Bing Ads API requests yet. E.g. for the connection to bingadsappsstorageprod.blob.core.windows.net as reported above.

These connection timeouts happen just for a few of our requests. But they are annoying because they happen several times a week. I wasn't able to determine a request that fails reproducibly yet.