search

opendiffy / diffy

Other
1.25k stars 142 forks source link

Exception propagated to the default monitor #5

Closed chxy325 closed 2 years ago

chxy325 commented 5 years ago

hi, I pulled the latest code. But I have an error when i just use one request. What does the error means? How can i fix it？ Thank you！

18:25:37.902 [finagle/netty4-8] INFO ai.diffy.lifter.HttpLifter - Return(Response("HTTP/1.1 Status(200)")) 18:25:38.954 [finagle/netty4-8] WARN com.twitter.finagle.util.DefaultMonitor - Exception propagated to the default monitor (upstream address: /10.10.35.183:50718, downstream address: n/a, label: ). java.lang.StackOverflowError: null at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658) at java.util.regex.Pattern$Loop.match(Pattern.java:4785) at java.util.regex.Pattern$GroupTail.match(Pattern.java:4717) at java.util.regex.Pattern$BranchConn.match(Pattern.java:4568) at java.util.regex.Pattern$CharProperty.match(Pattern.java:3777) at java.util.regex.Pattern$Branch.match(Pattern.java:4604) at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658) at java.util.regex.Pattern$Loop.match(Pattern.java:4785) at java.util.regex.Pattern$GroupTail.match(Pattern.java:4717) at java.util.regex.Pattern$BranchConn.match(Pattern.java:4568) at java.util.regex.Pattern$CharProperty.match(Pattern.java:3777) at java.util.regex.Pattern$Branch.match(Pattern.java:4604) at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658) at java.util.regex.Pattern$Loop.match(Pattern.java:4785) at java.util.regex.Pattern$GroupTail.match(Pattern.java:4717) at java.util.regex.Pattern$BranchConn.match(Pattern.java:4568) at java.util.regex.Pattern$CharProperty.match(Pattern.java:3777) at java.util.regex.Pattern$Branch.match(Pattern.java:4604) at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658) at java.util.regex.Pattern$Loop.match(Pattern.java:4785) at java.util.regex.Pattern$GroupTail.match(Pattern.java:4717) at java.util.regex.Pattern$BranchConn.match(Pattern.java:4568) at java.util.regex.Pattern$CharProperty.match(Pattern.java:3777) at java.util.regex.Pattern$Branch.match(Pattern.java:4604)

puneetkhanduri commented 5 years ago

This is showing that your downstream services are not reachable. i.e. At least one of primary, secondary, or candidate is not configured correctly. Can you share your diffy deploy script.

chxy325 commented 5 years ago

This is showing that your downstream services are not reachable. i.e. At least one of primary, secondary, or candidate is not configured correctly. Can you share your diffy deploy script.

Thank you for your reply. This is my deploy script: java -jar diffy-server.jar \ -candidate='10.26.9.124:8080' \ -master.primary='10.26.64.20:8080'\ -master.secondary='10.26.64.20:8080'\ -service.protocol=http \ -serviceName=My-Service \ -proxy.port=:8880 \ -admin.port=:8881 \ -http.port=:8882 \ -rootUrl='localhost:8882'

And my request : curl "http://10.10.35.183:8880/test/test.jpg"

Both candidate and master.primary are stable service.

puneetkhanduri commented 5 years ago

are you able to curl candidate, primary, and secondary from the machine where you are running diffy?

puneetkhanduri commented 5 years ago

Closing old issues. Please comment if needs to be re-opened.

sergioasantiago commented 5 years ago

I have a similar issue, and it is not related to not being able to connect to master/cand1/cand2, because it works for some cases, but out of a sudden diffy dies with this error:

09:50:06.467 [main] INFO ai.diffy.Main$ - Startup complete, server awaiting.
09:50:06.469 [main] INFO com.twitter.inject.server.Awaiter$ - Awaiting 2 awaitables:
com.twitter.finagle.server.ListeningStackServer$$anon$1
com.twitter.finagle.server.ListeningStackServer$$anon$1
09:50:15.353 [finagle/netty4-2-7] WARN com.twitter.finagle.util.DefaultMonitor - Exception propagated to the default monitor (upstream address: /100.96.8.84:48638, downstream address: n/a, label: ).
java.lang.StackOverflowError: null
        at java.util.regex.Pattern$Branch.match(Pattern.java:4606)
        at java.util.regex.Pattern$GroupHead.match(Pattern.java:4660)
        at java.util.regex.Pattern$Loop.match(Pattern.java:4787)
        at java.util.regex.Pattern$GroupTail.match(Pattern.java:4719)
        at java.util.regex.Pattern$BranchConn.match(Pattern.java:4570)
        at java.util.regex.Pattern$CharProperty.match(Pattern.java:3779)
        at java.util.regex.Pattern$Branch.match(Pattern.java:4606)

Any idea?

PS: We have some responses that returns the body using protbuf

puneetkhanduri commented 5 years ago

@sergioasantiago : Please share the following details:

  1. Diffy release version you are running.
  2. The protocol setting you are using. (http?)
  3. An example response with protobuf body.
sergioasantiago commented 5 years ago

1 - I'm using: FROM diffy/diffy:latest -> diffy/diffy latest 4d3a7c0343dc 12 days ago 2 - http: these are my parameters:

CMD [ "-candidate=localhost:9992", \ "-master.primary=localhost:9990", \ "-master.secondary=localhost:9991", \ "-service.protocol=http", \ "-serviceName=DiffyService", \ "-proxy.port=:8880", \ "-admin.port=:8881", \ "-http.port=:8888", \ "-rootUrl=localhost:8888", \ "-allowHttpSideEffects=true", \ "-apiRoot=/diffyservice", \ "-summary.delay=360" \ ]

this address is from a haproxy in order to rewrite the url before forwarding to master/cand1/cand2

3 - Unfortunately I don't have and example of a request that kills diffy, because I don't know which one is since we trigger thousands per sec.

puneetkhanduri commented 5 years ago

It is hard for me to imagine what kind of response would kill Diffy. You mentioned that the body is protobuf. So if you can provide any example of a concrete response with a protobuf body- even one that is not guranteed to kill Diffy, then I can try recreating it on my end and tracing through the code to see if it touches any of the lines in the stacktrace ...

sergioasantiago commented 5 years ago

I could get this one:

Headers:

HTTP/2 200 
date: Thu, 12 Sep 2019 11:38:02 GMT
content-type: application/x-protobuf
content-length: 308
server: Apache-Coyote/1.1
x-request-id: ab18a52e-52e0-4e81-8a10-72d2e9edd3dd
expires: 0
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
x-frame-options: DENY
cache-control: no-cache, no-store, max-age=0, must-revalidate
pragma: no-cache
strict-transport-security: max-age=31536000 ; includeSubDomains

Body:


K���������  HH���J@���W'�#@������$@  (�����-0�����-8��������GA�� �Ɨp@
K���������  HH���J@���W'�#@������$@  (�����-0�����-8��������GA�� �Ɨp@
K��ӹ�����   t��>�J@b�T��#@������$@ �(�����-0�����-8��������GA������[@
K���������  t��>�J@b�T��#@������$@ �(�����-0�����-8��������GA������[@
puneetkhanduri commented 5 years ago

The headers suggest you are using http/2. Diffy currently does not have the bindings for http/2 yet. That might explain the finagle monitor exceptions we are seeing. Are you able to downgrade to http?

sergioasantiago commented 5 years ago

The thing is, some requests were able to be compared, and all of them uses HTTP/2.

E.G: Headers:

HTTP/2 401
date: Thu, 12 Sep 2019 13:15:46 GMT
content-type: application/json
content-length: 113
cache-control: no-cache, no-store, max-age=0, must-revalidate
expires: 0
pragma: no-cache
x-content-type-options: nosniff
x-frame-options: DENY
x-xss-protection: 1; mode=block

Body:

{
  "status": "ERROR",
  "comment": "HTTP Status 401 - Full authentication is required to access this resource"
}
puneetkhanduri commented 5 years ago

Yup. The underlying networking library used by Diffy is finagle which has some experimental support for http2. See https://github.com/twitter/finagle/tree/develop/finagle-http2.

sergioasantiago commented 5 years ago

I did a small change and force the header Accept: application/json, and diffy is not been killed anymore.

I notice that we are getting from time to time 2 warns:

12:32:08.346 [finagle/netty4-2-4] WARN com.twitter.finagle.util.DefaultMonitor - Exception propagated to the default monitor (upstream address: /100.96.5.211:53892, downstream address: n/a, label: http).
java.util.NoSuchElementException: null
    at scala.collection.mutable.LinkedListLike.head(LinkedListLike.scala:78)
    at scala.collection.mutable.LinkedListLike.head$(LinkedListLike.scala:77)
    at scala.collection.mutable.LinkedList.head(LinkedList.scala:81)
    at scala.collection.mutable.MutableList.head(MutableList.scala:57)
    at scala.collection.generic.Growable.loop$1(Growable.scala:56)
    at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:61)
    at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
    at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:244)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at scala.collection.TraversableLike.flatMap(TraversableLike.scala:244)
    at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:241)
    at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
    at ai.diffy.analysis.InMemoryDifferenceCollector.$anonfun$prefix$1(InMemoryDifferenceCollector.scala:109)
    at com.twitter.util.Try$.apply(Try.scala:26)
    at com.twitter.util.Future$.apply(Future.scala:154)
    at ai.diffy.analysis.InMemoryDifferenceCollector.prefix(InMemoryDifferenceCollector.scala:113)
    at ai.diffy.ApiController.$anonfun$new$13(ApiController.scala:99)
    at com.twitter.finatra.http.internal.marshalling.CallbackConverter.$anonfun$createResponseCallback$5(CallbackConverter.scala:124)
    at com.twitter.finagle.Service$$anon$2.apply(Service.scala:29)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at ai.diffy.AllowLocalAccess.apply(ApiController.scala:211)
    at ai.diffy.AllowLocalAccess.apply(ApiController.scala:209)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:90)
    at com.twitter.finatra.http.internal.routing.Route.handleMatch(Route.scala:80)
    at com.twitter.finatra.http.internal.routing.Route.handle(Route.scala:71)
    at com.twitter.finatra.http.internal.routing.Routes.handle(Routes.scala:61)
    at com.twitter.finatra.http.internal.routing.RoutingService.route(RoutingService.scala:35)
    at com.twitter.finatra.http.internal.routing.RoutingService.apply(RoutingService.scala:30)
    at com.twitter.finatra.http.internal.routing.RoutingService.apply(RoutingService.scala:11)
    at com.twitter.finagle.ServiceProxy.apply(ServiceProxy.scala:12)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.tracing.AnnotatingTracingFilter.apply(TraceInitializerFilter.scala:140)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.tracing.ServerDestTracingFilter.apply(DestinationTracing.scala:38)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.service.DeadlineFilter.apply(DeadlineFilter.scala:265)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.filter.ExceptionSourceFilter.apply(ExceptionSourceFilter.scala:50)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.filter.MkJvmFilter$$anon$1.apply(MkJvmFilter.scala:30)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.tracing.AnnotatingTracingFilter.apply(TraceInitializerFilter.scala:140)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.http.filter.HttpNackFilter.apply(HttpNackFilter.scala:156)
    at com.twitter.finagle.http.filter.HttpNackFilter.apply(HttpNackFilter.scala:113)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.http.filter.PayloadSizeFilter.apply(PayloadSizeFilter.scala:123)
    at com.twitter.finagle.http.filter.PayloadSizeFilter.apply(PayloadSizeFilter.scala:47)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.server.BackupRequest$$anon$1$$anon$2.apply(BackupRequest.scala:31)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.http.filter.ServerContextFilter.$anonfun$apply$1(ContextFilter.scala:15)
    at com.twitter.util.Local.let(Local.scala:4979)
    at com.twitter.finagle.context.MarshalledContext.letLocal(MarshalledContext.scala:157)
    at com.twitter.finagle.context.MarshalledContext.let(MarshalledContext.scala:104)
    at com.twitter.finagle.http.codec.HttpContext$.read(HttpContext.scala:90)
    at com.twitter.finagle.http.filter.ServerContextFilter.apply(ContextFilter.scala:15)
    at com.twitter.finagle.http.filter.ServerContextFilter.apply(ContextFilter.scala:12)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:13)
    at com.twitter.finagle.http.HttpServerTraceInitializer.$anonfun$make$3(HttpServerTraceInitializer.scala:20)
    at com.twitter.finagle.http.TraceInfo$.$anonfun$letTraceIdFromRequestHeaders$1(TraceInfo.scala:105)
    at com.twitter.util.Local.let(Local.scala:4979)
    at com.twitter.finagle.context.MarshalledContext.letLocal(MarshalledContext.scala:157)
    at com.twitter.finagle.context.MarshalledContext.let(MarshalledContext.scala:90)
    at com.twitter.finagle.tracing.Trace$.letId(Trace.scala:104)
    at com.twitter.finagle.http.TraceInfo$.letTraceIdFromRequestHeaders(TraceInfo.scala:103)
    at com.twitter.finagle.http.HttpServerTraceInitializer.$anonfun$make$2(HttpServerTraceInitializer.scala:20)
    at com.twitter.util.Local.let(Local.scala:4979)
    at com.twitter.finagle.context.LocalContext.letLocal(LocalContext.scala:53)
    at com.twitter.finagle.context.LocalContext.let(LocalContext.scala:24)
    at com.twitter.finagle.tracing.Trace$.letTracer(Trace.scala:124)
    at com.twitter.finagle.http.HttpServerTraceInitializer.$anonfun$make$1(HttpServerTraceInitializer.scala:20)
    at com.twitter.finagle.Filter$$anon$16.apply(Filter.scala:406)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.filter.MonitorFilter.apply(MonitorFilter.scala:66)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.http.filter.DtabFilter.apply(DtabFilter.scala:25)
    at com.twitter.finagle.http.filter.DtabFilter.apply(DtabFilter.scala:12)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.http.codec.ResponseConformanceFilter$.apply(ResponseConformanceFilter.scala:23)
    at com.twitter.finagle.http.codec.ResponseConformanceFilter$.apply(ResponseConformanceFilter.scala:18)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.http.codec.HttpServerDispatcher.dispatch(HttpServerDispatcher.scala:42)
    at com.twitter.finagle.http.codec.HttpServerDispatcher.dispatch(HttpServerDispatcher.scala:23)
    at com.twitter.finagle.http.exp.GenStreamingSerialServerDispatcher.$anonfun$dispatchAndHandleFn$2(GenStreamingSerialServerDispatcher.scala:79)
    at com.twitter.util.Local.let(Local.scala:4979)
    at com.twitter.finagle.context.LocalContext.letLocal(LocalContext.scala:53)
    at com.twitter.finagle.context.LocalContext.let(LocalContext.scala:24)
    at com.twitter.finagle.http.exp.GenStreamingSerialServerDispatcher.$anonfun$dispatchAndHandleFn$1(GenStreamingSerialServerDispatcher.scala:77)
    at com.twitter.util.Future.$anonfun$flatMap$1(Future.scala:1798)
    at com.twitter.util.Promise$FutureTransformer.liftedTree1$1(Promise.scala:250)
    at com.twitter.util.Promise$FutureTransformer.k(Promise.scala:250)
    at com.twitter.util.Promise$Transformer.apply(Promise.scala:225)
    at com.twitter.util.Promise$WaitQueue.com$twitter$util$Promise$WaitQueue$$run(Promise.scala:101)
    at com.twitter.util.Promise$WaitQueue$$anon$1.run(Promise.scala:96)
    at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:167)
    at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:126)
    at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:243)
    at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:78)
    at com.twitter.util.Promise$WaitQueue.runInScheduler(Promise.scala:96)
    at com.twitter.util.Promise.updateIfEmpty(Promise.scala:797)
    at com.twitter.util.Promise.update(Promise.scala:769)
    at com.twitter.util.Promise.setValue(Promise.scala:745)
    at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:123)
    at com.twitter.finagle.netty4.transport.ChannelTransport$$anon$2.channelRead(ChannelTransport.scala:169)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at com.twitter.finagle.netty4.http.handler.UnpoolHttpHandler$.channelRead(UnpoolHttpHandler.scala:32)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at com.twitter.finagle.netty4.http.handler.BadRequestHandler.channelRead(BadRequestHandler.scala:41)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at com.twitter.finagle.netty4.http.handler.HeaderValidatorHandler$.channelRead(HeaderValidatorHandler.scala:51)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at com.twitter.finagle.netty4.http.handler.UriValidatorHandler$.channelRead(UriValidatorHandler.scala:29)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
    at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at com.twitter.finagle.util.BlockingTimeTrackingThreadFactory$$anon$1.run(BlockingTimeTrackingThreadFactory.scala:23)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)
12:26:55.837 [finagle/netty4-2-8] WARN com.twitter.finagle.util.DefaultMonitor - Exception propagated to the default monitor (upstream address: /100.96.5.211:47190, downstream address: n/a, label: http).
java.lang.Exception: Difference result not found
    at ai.diffy.analysis.InMemoryDifferenceCollector$.<init>(InMemoryDifferenceCollector.scala:82)
    at ai.diffy.analysis.InMemoryDifferenceCollector$.<clinit>(InMemoryDifferenceCollector.scala)
    at ai.diffy.analysis.InMemoryDifferenceCollector.apply(InMemoryDifferenceCollector.scala:124)
    at ai.diffy.ApiController.$anonfun$new$21(ApiController.scala:142)
    at scala.Option.map(Option.scala:163)
    at ai.diffy.ApiController.$anonfun$new$20(ApiController.scala:141)
    at com.twitter.finatra.http.internal.marshalling.CallbackConverter.$anonfun$createResponseCallback$5(CallbackConverter.scala:124)
    at com.twitter.finagle.Service$$anon$2.apply(Service.scala:29)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at ai.diffy.AllowLocalAccess.apply(ApiController.scala:211)
    at ai.diffy.AllowLocalAccess.apply(ApiController.scala:209)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:90)
    at com.twitter.finatra.http.internal.routing.Route.handleMatch(Route.scala:80)
    at com.twitter.finatra.http.internal.routing.Route.handle(Route.scala:71)
    at com.twitter.finatra.http.internal.routing.Routes.handle(Routes.scala:61)
    at com.twitter.finatra.http.internal.routing.RoutingService.route(RoutingService.scala:35)
    at com.twitter.finatra.http.internal.routing.RoutingService.apply(RoutingService.scala:30)
    at com.twitter.finatra.http.internal.routing.RoutingService.apply(RoutingService.scala:11)
    at com.twitter.finagle.ServiceProxy.apply(ServiceProxy.scala:12)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.tracing.AnnotatingTracingFilter.apply(TraceInitializerFilter.scala:140)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.tracing.ServerDestTracingFilter.apply(DestinationTracing.scala:38)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.service.DeadlineFilter.apply(DeadlineFilter.scala:265)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.filter.ExceptionSourceFilter.apply(ExceptionSourceFilter.scala:50)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.filter.MkJvmFilter$$anon$1.apply(MkJvmFilter.scala:30)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.tracing.AnnotatingTracingFilter.apply(TraceInitializerFilter.scala:140)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.http.filter.HttpNackFilter.apply(HttpNackFilter.scala:156)
    at com.twitter.finagle.http.filter.HttpNackFilter.apply(HttpNackFilter.scala:113)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.http.filter.PayloadSizeFilter.apply(PayloadSizeFilter.scala:123)
    at com.twitter.finagle.http.filter.PayloadSizeFilter.apply(PayloadSizeFilter.scala:47)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.server.BackupRequest$$anon$1$$anon$2.apply(BackupRequest.scala:31)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.http.filter.ServerContextFilter.$anonfun$apply$1(ContextFilter.scala:15)
    at com.twitter.util.Local.let(Local.scala:4979)
    at com.twitter.finagle.context.MarshalledContext.letLocal(MarshalledContext.scala:157)
    at com.twitter.finagle.context.MarshalledContext.let(MarshalledContext.scala:104)
    at com.twitter.finagle.http.codec.HttpContext$.read(HttpContext.scala:90)
    at com.twitter.finagle.http.filter.ServerContextFilter.apply(ContextFilter.scala:15)
    at com.twitter.finagle.http.filter.ServerContextFilter.apply(ContextFilter.scala:12)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:13)
    at com.twitter.finagle.http.HttpServerTraceInitializer.$anonfun$make$3(HttpServerTraceInitializer.scala:20)
    at com.twitter.finagle.http.TraceInfo$.$anonfun$letTraceIdFromRequestHeaders$1(TraceInfo.scala:105)
    at com.twitter.util.Local.let(Local.scala:4979)
    at com.twitter.finagle.context.MarshalledContext.letLocal(MarshalledContext.scala:157)
    at com.twitter.finagle.context.MarshalledContext.let(MarshalledContext.scala:90)
    at com.twitter.finagle.tracing.Trace$.letId(Trace.scala:104)
    at com.twitter.finagle.http.TraceInfo$.letTraceIdFromRequestHeaders(TraceInfo.scala:103)
    at com.twitter.finagle.http.HttpServerTraceInitializer.$anonfun$make$2(HttpServerTraceInitializer.scala:20)
    at com.twitter.util.Local.let(Local.scala:4979)
    at com.twitter.finagle.context.LocalContext.letLocal(LocalContext.scala:53)
    at com.twitter.finagle.context.LocalContext.let(LocalContext.scala:24)
    at com.twitter.finagle.tracing.Trace$.letTracer(Trace.scala:124)
    at com.twitter.finagle.http.HttpServerTraceInitializer.$anonfun$make$1(HttpServerTraceInitializer.scala:20)
    at com.twitter.finagle.Filter$$anon$16.apply(Filter.scala:406)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.filter.MonitorFilter.apply(MonitorFilter.scala:66)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.http.filter.DtabFilter.apply(DtabFilter.scala:25)
    at com.twitter.finagle.http.filter.DtabFilter.apply(DtabFilter.scala:12)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.Service$$anon$1.apply(Service.scala:16)
    at com.twitter.finagle.http.codec.ResponseConformanceFilter$.apply(ResponseConformanceFilter.scala:23)
    at com.twitter.finagle.http.codec.ResponseConformanceFilter$.apply(ResponseConformanceFilter.scala:18)
    at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:93)
    at com.twitter.finagle.http.codec.HttpServerDispatcher.dispatch(HttpServerDispatcher.scala:42)
    at com.twitter.finagle.http.codec.HttpServerDispatcher.dispatch(HttpServerDispatcher.scala:23)
    at com.twitter.finagle.http.exp.GenStreamingSerialServerDispatcher.$anonfun$dispatchAndHandleFn$2(GenStreamingSerialServerDispatcher.scala:79)
    at com.twitter.util.Local.let(Local.scala:4979)
    at com.twitter.finagle.context.LocalContext.letLocal(LocalContext.scala:53)
    at com.twitter.finagle.context.LocalContext.let(LocalContext.scala:24)
    at com.twitter.finagle.http.exp.GenStreamingSerialServerDispatcher.$anonfun$dispatchAndHandleFn$1(GenStreamingSerialServerDispatcher.scala:77)
    at com.twitter.util.Future.$anonfun$flatMap$1(Future.scala:1798)
    at com.twitter.util.Promise$FutureTransformer.liftedTree1$1(Promise.scala:250)
    at com.twitter.util.Promise$FutureTransformer.k(Promise.scala:250)
    at com.twitter.util.Promise$Transformer.apply(Promise.scala:225)
    at com.twitter.util.Promise$WaitQueue.com$twitter$util$Promise$WaitQueue$$run(Promise.scala:101)
    at com.twitter.util.Promise$WaitQueue$$anon$1.run(Promise.scala:96)
    at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:167)
    at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:126)
    at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:243)
    at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:78)
    at com.twitter.util.Promise$WaitQueue.runInScheduler(Promise.scala:96)
    at com.twitter.util.Promise.updateIfEmpty(Promise.scala:797)
    at com.twitter.util.Promise.update(Promise.scala:769)
    at com.twitter.util.Promise.setValue(Promise.scala:745)
    at com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:123)
    at com.twitter.finagle.netty4.transport.ChannelTransport$$anon$2.channelRead(ChannelTransport.scala:169)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at com.twitter.finagle.netty4.http.handler.UnpoolHttpHandler$.channelRead(UnpoolHttpHandler.scala:32)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at com.twitter.finagle.netty4.http.handler.BadRequestHandler.channelRead(BadRequestHandler.scala:41)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at com.twitter.finagle.netty4.http.handler.HeaderValidatorHandler$.channelRead(HeaderValidatorHandler.scala:51)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at com.twitter.finagle.netty4.http.handler.UriValidatorHandler$.channelRead(UriValidatorHandler.scala:29)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
    at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at com.twitter.finagle.util.BlockingTimeTrackingThreadFactory$$anon$1.run(BlockingTimeTrackingThreadFactory.scala:23)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)

Do you have an idea if this can impact the tests? or when it can happen?

puneetkhanduri commented 5 years ago

These exceptions are occasionally triggered by UI requests. I don't think these have anything to do with the header that you are forcing.

Was there any code-related insight that lead you to try forcing the Accept: application/json header? I am not sure why that would fix this problem.

sergioasantiago commented 5 years ago

Forcing this will prevent the response be sent using protobuf, which I believe is the problem here. When diffy tries to compare the responses in protobuf, it breaks somehow.

xiarilizi commented 4 years ago

I meet a similar problem, and diffy always die when I send one particular http request. (I also send some other http requests via postman and diffy is normal.)

the mistake is bellow: QQ图片20200229190927

any solutions?

sergioasantiago commented 4 years ago

In my case, the issue was that we used a binary protocol (protobuf) inside the body, which was probably causing issue to diffy and killing it.