Closed GAtom22 closed 1 year ago
we've observed similar thing as well, but didn't investigated, I believe https://github.com/cosmos/iavl/pull/773 will fix this, can you give it a try?
@yihuang will try, thanks for the prompt reply!
Just manually created a backport PR to 0.19.x: https://github.com/cosmos/iavl/pull/774, I think that's what evmos using.
still waiting to finish the sync, but looks good! The memory issue seems to be resolved! Thanks @yihuang!!
gc 266 @626.403s 0%: 0.068+2.8+0.010 ms clock, 0.54+0.81/5.4/10+0.086 ms cpu, 3116->3116->2844 MB, 3120 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 267 @630.070s 0%: 0.060+2.8+0.009 ms clock, 0.48+0.22/5.5/10+0.078 ms cpu, 3125->3125->2848 MB, 3129 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 268 @631.041s 0%: 0.045+2.5+0.003 ms clock, 0.36+0.33/4.9/10+0.024 ms cpu, 3129->3129->2837 MB, 3133 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 269 @637.667s 0%: 0.081+2.8+0.009 ms clock, 0.65+0.46/5.4/9.9+0.074 ms cpu, 3117->3117->2849 MB, 3121 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 270 @638.373s 0%: 0.067+2.8+0.014 ms clock, 0.53+0.45/5.1/11+0.11 ms cpu, 3132->3132->2840 MB, 3134 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 271 @644.314s 0%: 0.097+3.0+0.004 ms clock, 0.77+0.70/5.8/11+0.037 ms cpu, 3121->3121->2845 MB, 3125 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 272 @646.159s 0%: 0.053+2.6+0.010 ms clock, 0.42+0.55/5.0/11+0.083 ms cpu, 3126->3127->2845 MB, 3130 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 273 @649.257s 0%: 0.056+3.0+0.009 ms clock, 0.45+0.52/5.8/10+0.078 ms cpu, 3126->3127->2849 MB, 3130 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 274 @653.745s 0%: 0.064+4.1+0.015 ms clock, 0.51+0.15/7.9/11+0.12 ms cpu, 3130->3131->2854 MB, 3134 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 275 @654.562s 0%: 0.047+2.4+0.003 ms clock, 0.37+0.14/4.7/10+0.031 ms cpu, 3136->3136->2840 MB, 3140 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 276 @661.407s 0%: 0.055+2.8+0.009 ms clock, 0.44+0.47/5.4/11+0.075 ms cpu, 3122->3122->2844 MB, 3125 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 277 @662.096s 0%: 0.082+2.9+0.012 ms clock, 0.65+0.32/5.5/10+0.096 ms cpu, 3124->3126->2838 MB, 3128 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 278 @667.137s 0%: 0.055+2.3+0.005 ms clock, 0.44+0.23/4.4/11+0.040 ms cpu, 3118->3119->2844 MB, 3122 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 279 @669.601s 0%: 0.047+2.3+0.008 ms clock, 0.37+0.16/4.4/11+0.065 ms cpu, 3125->3125->2851 MB, 3129 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 280 @672.212s 0%: 0.049+2.2+0.003 ms clock, 0.39+0.50/4.3/11+0.024 ms cpu, 3133->3133->2842 MB, 3137 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 281 @677.829s 0%: 0.047+3.7+0.016 ms clock, 0.38+1.9/5.4/9.4+0.13 ms cpu, 3122->3133->2857 MB, 3126 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 282 @678.498s 0%: 0.048+2.5+0.008 ms clock, 0.39+0.25/5.0/11+0.070 ms cpu, 3129->3129->2834 MB, 3143 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 283 @685.982s 0%: 0.053+3.8+0.011 ms clock, 0.43+0.16/7.3/15+0.094 ms cpu, 3103->3104->2847 MB, 3118 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 284 @689.288s 0%: 0.090+3.2+0.020 ms clock, 0.72+3.5/6.1/12+0.16 ms cpu, 3128->3129->2844 MB, 3132 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 285 @691.534s 0%: 0.082+4.2+0.017 ms clock, 0.65+1.4/8.3/12+0.14 ms cpu, 3125->3125->2833 MB, 3129 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 286 @699.978s 0%: 0.080+3.6+0.014 ms clock, 0.64+0.93/7.1/12+0.11 ms cpu, 3113->3113->2846 MB, 3117 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 287 @700.669s 0%: 0.093+2.6+0.024 ms clock, 0.74+0.75/5.1/10+0.19 ms cpu, 3127->3127->2842 MB, 3131 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 288 @704.776s 0%: 0.077+3.5+0.003 ms clock, 0.62+0.24/6.9/14+0.026 ms cpu, 3132->3132->2852 MB, 3132 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 289 @709.676s 0%: 0.051+2.8+0.010 ms clock, 0.40+0.14/5.2/11+0.083 ms cpu, 3133->3134->2847 MB, 3137 MB goal, 0 MB stacks, 1 MB globals, 8 P
gc 290 @710.394s 0%: 0.047+2.3+0.009 ms clock, 0.38+0.50/4.3/11+0.074 ms cpu, 3127->3127->2834 MB, 3131 MB goal, 0 MB stacks, 1 MB globals, 8 P
Description
Trying to perform the state sync on an Evmos node noticed that the process runs OOM.
Using
GODEBUG=gctrace=1
can be observed that the heap used increases while applying the snapshot chunks. Filtering only the gctrace logs when fetching and applying the snapshot chunks we can see the size of the heap continues to grow over time:The heap profile shows that the![snapshot_heap](https://github.com/cosmos/iavl/assets/54514587/a786830c-3516-4c9b-9a7a-93491004e330)
iavl (*Importer) Add
andtypes (*SnapshotIAVLItem) Unmarshal
functions are involved in this issuePortion of the raw log: