java.io.IOException: Closed

[ashawley]

I have an AWS Lambda function running in us-east-1 that occasionally fails to connect to GitHub's HTTP API. It most often fails on the first API call, which is the endpoint /repos/SpeakEZ/speakez-rails/pulls. After that failure, the AWS Lambda function halts.

START RequestId: ed8e9386-4c53-11e7-8916-3be6d53816d3 Version: $LATEST
14:08:21.199 <ed8e9386-4c53-11e7-8916-3be6d53816d3> [main] INFO  prs.Main$:109 - Starting aws-gh-prs 0.1-SNAPSHOT
14:08:21.199 <ed8e9386-4c53-11e7-8916-3be6d53816d3> [main] INFO  prs.Main$:111 - Inspecting SNS notification...
14:08:21.242 <ed8e9386-4c53-11e7-8916-3be6d53816d3> [main] INFO  prs.Main$:120 - Processed 1 event(s)
14:08:21.260 <ed8e9386-4c53-11e7-8916-3be6d53816d3> [main] INFO  prs.Main$:165 - Pull request was SpeakEZ/master..ashawley/CAP-2052
14:08:21.260 <ed8e9386-4c53-11e7-8916-3be6d53816d3> [main] INFO  prs.Main$:239 - Querying GitHub for open pull request(s)...
14:08:21.261 <ed8e9386-4c53-11e7-8916-3be6d53816d3> [main] DEBUG com.ning.http.client.AsyncCompletionHandler:60 - Closed
14:08:21.358 <ed8e9386-4c53-11e7-8916-3be6d53816d3> [main] ERROR prs.Main$:233 - Failed for unexpected reason
java.io.IOException: Closed
    at com.ning.http.client.providers.netty.request.NettyRequestSender.sendRequest(NettyRequestSender.java:98)
    at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.execute(NettyAsyncHttpProvider.java:87)
    at com.ning.http.client.AsyncHttpClient.executeRequest(AsyncHttpClient.java:506)
    at com.ning.http.client.AsyncHttpClient$BoundRequestBuilder.execute(AsyncHttpClient.java:225)
    at codecheck.github.transport.asynchttp19.AsyncHttp19Request.execute(AsyncHttp19Transport.scala:41)
    at codecheck.github.api.GitHubAPI.exec(GitHubAPI.scala:66)
    at codecheck.github.operations.PullRequestOp$class.listPullRequests(PullRequestOp.scala:28)
    at codecheck.github.api.GitHubAPI.listPullRequests(GitHubAPI.scala:21)
    at prs.Main$.mergeFor(Main.scala:247)
    at prs.Main$$anonfun$2$$anonfun$5.apply(Main.scala:175)
    at prs.Main$$anonfun$2$$anonfun$5.apply(Main.scala:172)
    at scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:683)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:682)
    at prs.Main$$anonfun$2.apply(Main.scala:172)
    at prs.Main$$anonfun$2.apply(Main.scala:106)
    at scala.util.Try$.apply(Try.scala:192)
    at prs.Main$.handler(Main.scala:106)
    at prs.Main.handler(Main.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at lambdainternal.EventHandlerLoader$PojoMethodRequestHandler.handleRequest(EventHandlerLoader.java:456)
    at lambdainternal.EventHandlerLoader$PojoHandlerAsStreamHandler.handleRequest(EventHandlerLoader.java:375)
    at lambdainternal.EventHandlerLoader$2.call(EventHandlerLoader.java:1139)
    at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:278)
    at lambdainternal.AWSLambda.<clinit>(AWSLambda.java:62)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:94)
END RequestId: ed8e9386-4c53-11e7-8916-3be6d53816d3
REPORT RequestId: ed8e9386-4c53-11e7-8916-3be6d53816d3  Duration: 1134.79 ms
       Billed Duration: 1200 ms     Memory Size: 384 MB Max Memory Used: 384 MB 

It seems the HTTP client is not retrying the request or waiting on a timeout. It is pretty sudden. I've inspected the source code of the Ning/AHC library:

https://github.com/AsyncHttpClient/async-http-client/blob/1.9.x/src/main/java/com/ning/http/client/providers/netty/request/NettyRequestSender.java

https://github.com/AsyncHttpClient/async-http-client/blob/1.9.x/src/main/java/com/ning/http/client/providers/netty/NettyAsyncHttpProvider.java

For some reason the connection is in a closed state right away.

Coincidentally, AWS Lambda will re-run a function if it fails. Typically it will retry twice. However, the subsequent tries by AWS Lambda doesn't usually improve anything.

[ashawley]

One solution is trying AHC 2.0, or switching to another GitHub API library, such as https://github.com/eed3si9n/gigahorse-github/

[ashawley]

There's another Scala GitHub API library in town

https://github.com/47deg/github4s