Library included: opentracing-spring-jaeger-web-starter
version: 3.1.1/3.2.2
implementation: Extended ServletFilterSpanDecorator & RestTemplateSpanDecorator to set tags and span details to log details related to our API calls.
Issue: Our API gets called twice in a business flow and we call 3 downstream systems to aggregate data. It was observed that data that was set to tags is being added to span as a hashmap. As a results, 2 calls to api is resulting in adding 12 hashmaps to heap. Since these hashmaps are still being referenced in the memory, minor/mixed GC is not collecting them. Below is histo of our heap with and without jaeger tracing. It's a 2-3 hour test.
Library included: opentracing-spring-jaeger-web-starter version: 3.1.1/3.2.2 implementation: Extended ServletFilterSpanDecorator & RestTemplateSpanDecorator to set tags and span details to log details related to our API calls. Issue: Our API gets called twice in a business flow and we call 3 downstream systems to aggregate data. It was observed that data that was set to tags is being added to span as a hashmap. As a results, 2 calls to api is resulting in adding 12 hashmaps to heap. Since these hashmaps are still being referenced in the memory, minor/mixed GC is not collecting them. Below is histo of our heap with and without jaeger tracing. It's a 2-3 hour test.
with jaeger
num #instances #bytes class name
1: 6209579 198706528 java.util.HashMap$Node 2: 6275334 150608016 java.lang.String 3: 490368 69699240 [Ljava.util.HashMap$Node; 4: 747753 47571272 [C 5: 1185323 37930336 io.opentracing.util.ThreadLocalScope 6: 474978 34198416 io.jaegertracing.internal.JaegerSpanContext 7: 474978 30398592 io.jaegertracing.internal.JaegerSpan 8: 480967 23086416 java.util.HashMap 9: 490463 11771112 java.util.ArrayList
without jaeger
num #instances #bytes class name
1: 148201 14553584 [C 2: 52875 11382616 [B 3: 262144 6291456 org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapper 4: 44951 4929096 [Ljava.lang.Object; 5: 147255 3534120 java.lang.String 6: 20508 3117216 sun.security.ssl.SSLSessionImpl 7: 13168 2704688 [I 8: 20646 2307752 java.lang.Class 9: 66099 2115168 java.util.concurrent.ConcurrentHashMap$Node 10: 51668 2066720 java.util.LinkedHashMap$Entry 11: 31884 1783368 [Ljava.lang.String; 12: 21543 1378752 java.util.concurrent.ConcurrentHashMap 13: 14062 1275808 [Ljava.util.HashMap$Node; 14: 52835 1268040 java.util.ArrayList