search

elastic / elastic-agent

Elastic Agent - single, unified way to add monitoring for logs, metrics, and other types of data to a host.
Other
124 stars 134 forks source link

Diagnostics rate limiting config doesn't seem to work #5570

Open jen-huang opened 3 hours ago

jen-huang commented 3 hours ago

Found while testing advanced monitoring settings for UI work in https://github.com/elastic/kibana/pull/193361

I applied agent.monitoring.diagnostics.limit options through the new UI which is translated to the following agent config (from inspect):

agent:
  monitoring:
    diagnostics:
      limit:
        burst: 100
        interval: 10m
      uploader:
        init_dur: 1m
        max_dur: 1h
        max_retries: 100

This seems to have been applied, from the logs:

[elastic_agent][info] Starting monitoring server with cfg &config.MonitoringConfig{Enabled:true, MonitorLogs:true, MonitorMetrics:true, MetricsPeriod:"", LogMetrics:true, HTTP:(*config.MonitoringHTTPConfig)(0x14001e33ce0), Namespace:"default", Pprof:(*config.PprofConfig)(0x14001d53145), MonitorTraces:true, APM:config.APMConfig{Environment:"", APIKey:"", SecretToken:"", Hosts:[]string(nil), GlobalLabels:map[string]string(nil), TLS:config.APMTLS{SkipVerify:false, ServerCertificate:"", ServerCA:""}}, Diagnostics:config.Diagnostics{Uploader:config.Uploader{MaxRetries:100, InitDur:1000000000, MaxDur:600000000000}, Limit:config.Limit{Interval:600000000000, Burst:100}}}

I then requested agent diagnostics through the UI a few times while waiting a few seconds in between, but it seems to only allow me 1 request per minute (the default value). Logs show messages like:

[elastic_agent][info] diagnostics action handler rate limited: rate limit exceeded

I'm not sure how to test the agent.monitoring.diagnostics.uploader options, but am hoping that whoever investigates this issue can also double check in the code that these options are applied as well.

elasticmachine commented 3 hours ago

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)