Mario-Hofstaetter
commented
2 years ago After changing config of your biggest instance to:
processors:
memory_limiter:
check_interval: 1s
limit_mib: 768
batch:
extensions:
memory_ballast:
size_mib: 512and it is still GC'ing in a short interval:
2022-05-08T22:40:03+02:00 {"level":"info","ts":1652042403.575912,"caller":"memorylimiterprocessor/memorylimiter.go:281","msg":"Memory usage after GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":159}
2022-05-08T22:40:03+02:00 {"level":"info","ts":1652042403.5041053,"caller":"memorylimiterprocessor/memorylimiter.go:310","msg":"Memory usage is above soft limit. Forcing a GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":702}
2022-05-08T22:39:30+02:00 {"level":"info","ts":1652042370.570825,"caller":"memorylimiterprocessor/memorylimiter.go:281","msg":"Memory usage after GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":163}
2022-05-08T22:39:30+02:00 {"level":"info","ts":1652042370.489251,"caller":"memorylimiterprocessor/memorylimiter.go:310","msg":"Memory usage is above soft limit. Forcing a GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":641}
2022-05-08T22:38:56+02:00 {"level":"info","ts":1652042336.586582,"caller":"memorylimiterprocessor/memorylimiter.go:281","msg":"Memory usage after GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":166}
2022-05-08T22:38:56+02:00 {"level":"info","ts":1652042336.5032008,"caller":"memorylimiterprocessor/memorylimiter.go:310","msg":"Memory usage is above soft limit. Forcing a GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":680}
2022-05-08T22:38:21+02:00 {"level":"info","ts":1652042301.5798883,"caller":"memorylimiterprocessor/memorylimiter.go:281","msg":"Memory usage after GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":186}
2022-05-08T22:38:21+02:00 {"level":"info","ts":1652042301.5080392,"caller":"memorylimiterprocessor/memorylimiter.go:310","msg":"Memory usage is above soft limit. Forcing a GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":677}
2022-05-08T22:37:48+02:00 {"level":"info","ts":1652042268.5550425,"caller":"memorylimiterprocessor/memorylimiter.go:281","msg":"Memory usage after GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":158}
2022-05-08T22:37:48+02:00 {"level":"info","ts":1652042268.492242,"caller":"memorylimiterprocessor/memorylimiter.go:310","msg":"Memory usage is above soft limit. Forcing a GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":645}
2022-05-08T22:37:14+02:00 {"level":"info","ts":1652042234.574366,"caller":"memorylimiterprocessor/memorylimiter.go:281","msg":"Memory usage after GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":159}
2022-05-08T22:37:14+02:00 {"level":"info","ts":1652042234.4942825,"caller":"memorylimiterprocessor/memorylimiter.go:310","msg":"Memory usage is above soft limit. Forcing a GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":643}
2022-05-08T22:36:40+02:00 {"level":"info","ts":1652042200.5750558,"caller":"memorylimiterprocessor/memorylimiter.go:281","msg":"Memory usage after GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":159}
2022-05-08T22:36:40+02:00 {"level":"info","ts":1652042200.501883,"caller":"memorylimiterprocessor/memorylimiter.go:310","msg":"Memory usage is above soft limit. Forcing a GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":714}
2022-05-08T22:36:06+02:00 {"level":"info","ts":1652042166.5794554,"caller":"memorylimiterprocessor/memorylimiter.go:281","msg":"Memory usage after GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":169}
2022-05-08T22:36:06+02:00 {"level":"info","ts":1652042166.4941897,"caller":"memorylimiterprocessor/memorylimiter.go:310","msg":"Memory usage is above soft limit. Forcing a GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":655}
2022-05-08T22:35:32+02:00 {"level":"info","ts":1652042132.5813339,"caller":"memorylimiterprocessor/memorylimiter.go:281","msg":"Memory usage after GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":160}
2022-05-08T22:35:32+02:00 {"level":"info","ts":1652042132.4957755,"caller":"memorylimiterprocessor/memorylimiter.go:310","msg":"Memory usage is above soft limit. Forcing a GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":619}
2022-05-08T22:35:06+02:00 {"level":"info","ts":1652042106.5584857,"caller":"memorylimiterprocessor/memorylimiter.go:281","msg":"Memory usage after GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":170}
2022-05-08T22:35:06+02:00 {"level":"info","ts":1652042106.4969077,"caller":"memorylimiterprocessor/memorylimiter.go:310","msg":"Memory usage is above soft limit. Forcing a GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":632}
2022-05-08T22:34:33+02:00 {"level":"info","ts":1652042073.5752532,"caller":"memorylimiterprocessor/memorylimiter.go:281","msg":"Memory usage after GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":148}
2022-05-08T22:34:33+02:00 {"level":"info","ts":1652042073.4907186,"caller":"memorylimiterprocessor/memorylimiter.go:310","msg":"Memory usage is above soft limit. Forcing a GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":620}
2022-05-08T22:33:59+02:00 {"level":"info","ts":1652042039.5686462,"caller":"memorylimiterprocessor/memorylimiter.go:281","msg":"Memory usage after GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":135}
2022-05-08T22:33:59+02:00 {"level":"info","ts":1652042039.4970694,"caller":"memorylimiterprocessor/memorylimiter.go:310","msg":"Memory usage is above soft limit. Forcing a GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":621}
2022-05-08T22:33:24+02:00 {"level":"info","ts":1652042004.566677,"caller":"memorylimiterprocessor/memorylimiter.go:281","msg":"Memory usage after GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":143}
2022-05-08T22:33:24+02:00 {"level":"info","ts":1652042004.5086606,"caller":"memorylimiterprocessor/memorylimiter.go:310","msg":"Memory usage is above soft limit. Forcing a GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":631}
2022-05-08T22:32:50+02:00 {"level":"info","ts":1652041970.5390697,"caller":"memorylimiterprocessor/memorylimiter.go:281","msg":"Memory usage after GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":147}
2022-05-08T22:32:50+02:00 {"level":"info","ts":1652041970.490576,"caller":"memorylimiterprocessor/memorylimiter.go:310","msg":"Memory usage is above soft limit. Forcing a GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":668}
2022-05-08T22:32:10+02:00 {"level":"info","ts":1652041930.5783658,"caller":"memorylimiterprocessor/memorylimiter.go:281","msg":"Memory usage after GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":112}
2022-05-08T22:32:10+02:00 {"level":"info","ts":1652041930.5014062,"caller":"memorylimiterprocessor/memorylimiter.go:310","msg":"Memory usage is above soft limit. Forcing a GC.","kind":"processor","name":"memory_limiter","pipeline":"metrics","cur_mem_mib":663}Is this normal behavior? IMHO 700MB of memory is a lot for scraping 8 MB of metrics.
Edit 2022-05-12
Been playing with memory limiter the last couple of days and still anxious about memory usage.
Currently running this config (CLICK ME)
```yaml exporters: prometheus: endpoint: 0.0.0.0:7299 metric_expiration: 5m # default = 5m send_timestamps: true receivers: prometheus: config: scrape_configs: - job_name: localmetrics scrape_interval: 17s tls_config: insecure_skip_verify: true static_configs: - targets: [localhost:8888] # Self diagnostic metrics of otelcol labels: app: otelcol file_sd_configs: - files: - "C:/Program Files/OpenTelemetry/OTEL Collector/metric-targets/*.yaml" processors: memory_limiter: check_interval: 1s limit_mib: 512 batch: extensions: health_check: endpoint: localhost:13133 memory_ballast: size_mib: 256 pprof: endpoint: "127.0.0.1:1777" zpages: endpoint: localhost:42424 service: extensions: [health_check,memory_ballast,pprof,zpages] pipelines: metrics: receivers: [prometheus] processors: [memory_limiter,batch] exporters: [prometheus] # Otel-Collector Self-Diagnostics telemetry: logs: level: info encoding: json output_paths: ["stdout"] error_output_paths: ["stderr"] metrics: address: localhost:8888 ```using only metrics pipeline and a imho reasonable amount of metrics, the logs of otelcol are full of info and warn messages showing GC actions, and even Dropping data, while consume > 600 MB of memory?
Currently the prometheus exporter exposes: 34377 lines of metrics, 4.39 MB. 600 MB of memory seems too much for this.
What am I doing wrong?
pprof outputs
Since github does not accept .txt or .gz at the moment, link to tar.gz with all files from /pprof: https://1drv.ms/u/s!AnvnX1Qo7mIHj6VrpUxcn95c8Ue7Tw?e=DfQS6J
mx-psi
dashpole
newly12
jpkrohling
RalphSu
holograph
kwiesmueller
Doron-Bargo
jmacd
gouthamve
CatherineF-dev
bogdandrutu
Describe the bug
I am infrequently observing high memory usage of
otelcol.exeon Windows Systems. The memory is however within the set bounds ofmemory_limiter(see config below). But I am not sure if this memory usage is intended behavior, since there is negligible load on the systems. I am not experienced in golang so please excuse my lack of knowledge.My planned actions to counteract this behavior is reduce the configured
limit_miband fix thememory_ballast.Steps to reproduce
?? Full config see below.
What did you expect to see?
otelcolshould not consume "much" memory given the workload is little, and/or memory should eventually be garbage collected?What did you see instead?
otelcolprocess uses (?) up to 2GB of memory. Currently on my local machine here it sits at ~ 1.5GB. Last stdout log message frommemory_limiterafter machine was started from hibernation:Is this the intended behavior of memory usage of the process?
What are recommended values for
memory_limiter? The documentation is a little vague.On my machine, the
prometheusexporter currently emits ~ 19362 lines of metrics, less than 3 MB in size. On our biggest instance, the prometheus exporter has 37654 metric lines , ~ 8MB. Is this a lot?The documentation on that page uses
limit_mib: 4000, which seems kinda HUGE for this kind of application?After re-reading those docs just now, this line caught my attension:
So it actually is the expected behavor of otelcol to stay around
limit_mibindefinetely?My trace queue size is currently at zero:
I had the suspicion the trace buffer was filling memory when VPN was disconnected at
jaegerserver not reachable, since the default queue size of 5000 (?) seemed rather high.Looking at logs from last night, it has been the
tracesresponsible for causing the memory increase:If the
sending_queuequeue_sizeof 5000 does not fit within memory limits, what is going to happen? Oldest trace spans are going to get dropped from queue?I did however also have on machine where otelcol used 2GB of memory suddenly, and no traces where being queued. No warnings at the time, unfortunately I have not info logs from that date (otelcol was still windows application event logging)
There are no apps emitting traces on that machine yet, so no Idea what has happend there.
I have collected and attached various debug information from http://localhost:1777/debug/pprof/, in case the memory usage is not okay.
What version did you use?
What config did you use?
I am using two config files (currently 3 two debug) consisting of the following parts:
ConfigFile 1 (common and metrics):
I just noticed an error in my config... the
memory_ballastextension is not used it seems..Second File used optionally (if traces are required):
Third config file currently used to add
pprof:Environment OS: Windows 10 21H2, Windows Server 2019