search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
12.11k stars 4.91k forks source link

[beats] Live reloading causes delay sending data based on reload.period if enabled #20847

Open mark54g opened 4 years ago

mark54g commented 4 years ago

Various beats seem to have an issue where reload.enabled being set to true will cause a delay that is roughly equal to the reload.period

Confirmed with master branch of metricbeat per @kaiyan-sheng

elasticmachine commented 4 years ago

Pinging @elastic/integrations (Team:Integrations)

botelastic[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

PresGas commented 3 years ago

@mark54g , @kaiyan-sheng . This is still a problem and I'm hoping this ticket gets reopened by a human.

I was the person that worked with elastic support initially to report this

kaiyan-sheng commented 2 years ago

Hi @PresGas, could you post with more info on this issue please? Thanks!!

PresGas commented 2 years ago

I opened a case that created this ticket.

It is still impacting filebeat version 7.10.x and likely newer since this was auto closed without any human interaction. Thanks for reopening!

PresGas commented 2 years ago

Please let me know if I can provide anything. The key bits for reproducing the issue is generally to have the filebeat.yml configed with this snippet:

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 30m

And then you should see some delays either going to logstash/kibana or putting filebeat to debug mode and watching the logs there.

andrewkroh commented 2 years ago

What's the use case for a 30m reload interval? Normally I've seen this in the range of seconds to a minute. In that case the initial delay is less obvious.

The code does appear to wait the full reload interval before triggering the initial load.

https://github.com/elastic/beats/blob/5d7e83b631854442912d887a04e10412b5876a04/libbeat/cfgfile/reload.go#L176-L193

PresGas commented 2 years ago

@andrewkroh - I have since switched roles and not longer on the elastic related project prompting this issue.

IIRC - one of those was likely a default? I honestly have slept since then.

Nevertheless, an initial start should not wait that long to load those settings. An initial start should load the settings immediately.

adrianchen-es commented 1 year ago

@andrewkroh

The same issue still occurs on metricbeat 8.5.3 Our use case is set at

reload.period: 300s

If the behaviour of waiting the same time before the initial loading is correct. This should be noted in the doco?

botelastic[bot] commented 8 months ago

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

LarryTurtis commented 3 months ago

👍 This issue burned my team today! Would be nice if, at a minimum, the docs reflected that live reload will delay the initial log shipping by reload.period

elasticmachine commented 3 months ago

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)