Closed junhoi-pk closed 5 years ago
lazy val xxxClient = XXXHttpClientBuilder.build(
conf.getString("services.xxx.label"),
conf.getString("services.xxx.host"),
conf.getInt("services.xxx.port"),
conf.getBoolean("services.xxx.ssl")
)
-----------------------------------------------
object XXXHttpClientBuilder {
def build(label: String, host: String, port: Int = 80, ssl: Boolean = false, requestTimeout: Duration = 3.seconds) = {
val httpClientModule = new HttpClientModule {
override def dest = s"$host:$port"
override def defaultHeaders: Map[String, String] = Map("Host" -> host)
def buildHttpClient(mapper: FinatraObjectMapper, httpService: Service[Request, Response]): XXXHttpClient = {
new XXXHttpClient(
hostname = hostname,
httpService = httpService,
retryPolicy = retryPolicy,
defaultHeaders = defaultHeaders,
mapper = mapper
)
}
}
val httpClient = ssl match {
case false =>
Http.client
.withLabel(label)
.withTracer(XXXServerMain.injector.instance[HttpZipkinTracer])
.withSessionQualifier.noFailFast
.withRequestTimeout(requestTimeout)
.newClient(s"$host:$port")
.toService
case true =>
Http.client
.withLabel(label)
.withTracer(XXXServerMain.injector.instance[HttpZipkinTracer])
.withSessionQualifier.noFailFast
.withRequestTimeout(requestTimeout)
.withTls(host)
.newClient(s"$host:$port")
.toService
}
val masked = new MaskCancelFilter[Request, Response]
val maskedClient = masked.andThen(httpClient)
httpClientModule.buildHttpClient(
XXXServerMain.injector.instance[FinatraObjectMapper],
masked.andThen(maskedClient)
)
}
}
-----------------------------------------------
class XXXHttpClient(
hostname: String = "",
httpService: Service[Request, Response],
retryPolicy: Option[RetryPolicy[Try[Response]]] = None,
defaultHeaders: Map[String, String] = Map(),
mapper: FinatraObjectMapper) extends HttpClient(hostname, httpService, retryPolicy, defaultHeaders, mapper) {
override def execute(request: Request): Future[Response] = {
val res: Future[Response] = super.execute(request)
res.map {
httpResponse =>
if(httpResponse.status == Status.InternalServerError)
MDC.put("errorResponse", s"uri:${request.uri}, code:${httpResponse.statusCode}, body:${httpResponse.contentString} ")
}
res
}
}
@jungrammer thanks for the issue. Technically, this is something with Finagle you're experiencing. Channel closed generally indicates the connection has been closed. This could happen for any myriad of reasons not all of which are bad. Were you experiencing any network issues at the time?
@cacoco Thank you for the answer. It's not an error that's often happening, but because it's happening occasionally, I can only see it with error reporting.
@jungrammer you may be experiencing something similar to https://github.com/twitter/finagle/issues/517 which has come up again recently internally as well as we suspect that we may not be properly handling the Connection: Close
header. Any more information you can provide around the system when you observe this behavior would be helpful. Thanks!
@jungrammer as @cacoco mentioned, ChannelClosedException
is generic and could indicate that the server has hung up / close the session. Do you have any visibility on the remote server to try to correlate these errors with any such event on the server?
@roanta Hi roanta.
I've tested when the remote server died before. I've tested when the remote server died before. In other words, we tested the session disconnection on the remote server.
At that time, a ChannelClosedException
occurred, but it showed a different stack trace.
Guessing it seems to be caused by the http client
closing the connection and attempting to close the session but it is already closed.
In the code below, the above error occurred about 2%.
lazy val xxxClient = XXXHttpClientBuilder.build(
conf.getString("services.xxx.label"),
conf.getString("services.xxx.host"),
conf.getInt("services.xxx.port"),
conf.getBoolean("services.xxx.ssl")
)
def xxxx(...) = {
val cli = xxxClient;
cli.executeJson[XXX](Request("xxx"))
}
@jungrammer, we're unable to reproduce this locally so I'm going to close this for now. If you could write a failing unit test please do post it and reopen the ticket.
HttpClientModule rarely throws a
ChannelClosedException
. The finatra version I use is 18.1.0 (But version 18.12.0 also had issues.)It didn't happen when there was a lot of traffic, and the remote server didn't have a problem.
Below is the stack trace