search

kamon-io / Kamon

Distributed Tracing, Metrics and Context Propagation for applications running on the JVM
https://kamon.io
Other
1.41k stars 327 forks source link

Connection error, retrying... (11 left) timeout #801

Open AlexITC opened 4 years ago

AlexITC commented 4 years ago

I have seen this error almost 6k times in a single project on the last 15 days:

    at kamon.apm.KamonApmApiClient.postWithRetry(KamonApmApiClient.scala:101)
    at kamon.apm.KamonApmApiClient.postSpans(KamonApmApiClient.scala:32)
    at kamon.apm.KamonApm.$anonfun$reportSpans$2(KamonApm.scala:82)
    at kamon.apm.KamonApm.$anonfun$reportSpans$2$adapted(KamonApm.scala:82)
    at scala.Option.foreach(Option.scala:407)
    at kamon.apm.KamonApm.reportSpans(KamonApm.scala:82)
    at kamon.module.ModuleRegistry$$anon$2.$anonfun$run$4(ModuleRegistry.scala:214)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
bradrust commented 4 years ago

Although I don't have the stack trace to confirm it's the same problem, I frequently get what appears to be the same timeouts...

2020-08-04 14:08:34,420 [warn] k.a.KamonApmApiClient - Connection error, retrying... (20 left) timeout 2020-08-04 14:08:58,510 [warn] k.a.KamonApmApiClient - Connection error, retrying... (20 left) timeout . . . 2020-08-04 14:38:37,315 [error] k.a.KamonApmApiClient - Ingestion error, no retries, dropping snapshot... timeout

And it appears at that point that it isn't reporting and from the Kamon dashboard side, I get no statistics and alerts start firing that I don't have enough instances (based on my alert levels of course).

JyoshnaBoppidi commented 3 years ago

[2021-06-04 14:02:33,644] [WARN] [kamon.apm.KamonApmApiClient] [] [main] - Connection error, retrying... (1 left) sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target [2021-06-04 14:02:38,772] [ERROR] [kamon.apm.KamonApmApiClient] [] [main] - Ingestion error, no retries, dropping snapshot... sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

JyoshnaBoppidi commented 3 years ago

Please provide the solution for above issue