search

FrankChen021 / bithon

An observability platform mainly for Java
Apache License 2.0
14 stars 4 forks source link

Optimize the memory usage at the collector side #724

Closed FrankChen021 closed 3 months ago

FrankChen021 commented 4 months ago

In large traffic, the memory at the collector is under pressure which might cause frequent old gc. 

root@collector-6949bf6844-p49mb:/opt# jmap -histo 1

 num     #instances         #bytes  class name
----------------------------------------------
   1:      65012425     4554538328  [C
   2:      64970068     1559281632  java.lang.String
   3:       9158685      952503240  org.bithon.server.storage.tracing.TraceSpan
   4:        176603      657832968  [B
   5:       9204857      441833136  java.util.TreeMap
   6:       9319575      372783000  java.util.TreeMap$Entry
   7:       9193435      147094960  java.util.TreeMap$EntrySet
   8:       1232730       66622800  [Ljava.lang.Object;
   9:         19098       51258416  [I
  10:         13335       23747568  [S
  11:        466503       22392144  com.fasterxml.jackson.core.json.JsonWriteContext
  12:        720935       17302440  java.lang.Long
  13:        155818       14958528  com.fasterxml.jackson.core.json.UTF8JsonGenerator
  14:        155831       11219832  com.fasterxml.jackson.databind.ser.DefaultSerializerProvider$Impl
  15:        155870        8728720  com.fasterxml.jackson.core.io.IOContext
  16:         98354        7081488  com.fasterxml.jackson.databind.deser.DefaultDeserializationContext$Impl
  17:        156650        5012800  java.util.LinkedList
  18:        155818        4986176  com.fasterxml.jackson.core.util.ByteArrayBuilder
  19:        196861        4724800  [Lcom.fasterxml.jackson.databind.JavaType;
  20:         98405        3148960  com.fasterxml.jackson.databind.type.TypeBindings
  21:         30550        2688400  java.lang.reflect.Method
  22:        104944        2647976  [Ljava.lang.String;
  23:          3840        2580480  org.bithon.shaded.io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue
  24:        100055        2401112  [Ljava.lang.reflect.Type;
  25:         21524        2381592  java.lang.Class
  26:         98678        2370448  [Ljava.lang.reflect.TypeVariable;
  27:         98358        2360592  com.fasterxml.jackson.databind.type.TypeBindings$AsKey
  28:         72822        2330304  java.util.concurrent.ConcurrentHashMap$Node
  1. Optimize how the jackson is used. create JsonGenerator manually once for a send call instead of being created inside the writeValue implicitely.
  2. Optimize how serialization works. Serialize on byte buffer directly.
FrankChen021 commented 3 months ago

Related to #646