search

Graylog2 / collector-sidecar

Manage log collectors through Graylog
https://www.graylog.org/
Other
268 stars 56 forks source link

Sidecar renders configuration if Server response is not successfull #413

Closed HenryTheSir closed 2 years ago

HenryTheSir commented 3 years ago

Problem description

If the Server responds with an error/timeout for the configuration call. Sidecar detects this as a 'new' configuration and rewrites the local configuration and restarts the specific agent. In the next iteration it detects the correct configuration again as new configuration and restarts the agent.

Steps to reproduce the problem

  1. Make sure the request for the configuration times out or fails, without an HTTP Status Code or DNS Resolution times out via UDP.

Possible Solution

In this Line the Error gets detected from the sidecar, and the error gets logged to file: https://github.com/Graylog2/collector-sidecar/blob/cd538a158832ec48f257598fe44185ccfff84c4c/api/graylog.go#L109-L113

As the resp being nil following condition does not match: https://github.com/Graylog2/collector-sidecar/blob/cd538a158832ec48f257598fe44185ccfff84c4c/api/graylog.go#L115

So the last return statement triggers and returns an 'nil' Error, which leads the calling procedure to 'detect' a new config: https://github.com/Graylog2/collector-sidecar/blob/cd538a158832ec48f257598fe44185ccfff84c4c/api/graylog.go#L136

Logs


time="2021-05-20T02:39:05+02:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get GLHOST/api/sidecar/configurations/render/xxxxxxx dial tcp: lookup GLHOST  on IP:Port2 : write udp IP:Port1-> IP:Port2 : write: invalid argument" 
time="2021-05-20T02:39:05+02:00" level=info msg="[filebeat-wrapped] Configuration change detected, rewriting configuration file." 
time="2021-05-20T02:39:06+02:00" level=info msg="[filebeat-wrapped] Stopping" 
time="2021-05-20T02:39:08+02:00" level=info msg="[filebeat-wrapped] Starting (exec driver)" 
time="2021-05-20T02:39:09+02:00" level=error msg="[filebeat-wrapped] Unable to start collector after 3 tries, giving up!" 
time="2021-05-20T02:39:09+02:00" level=error msg="[filebeat-wrapped] Collector output: Exiting: No outputs are defined. Please define one under the output section.\nThu May 20 02:39:08 CEST 2021 /opt/graylog-sidecar/filebeat-sudo-wrapper.sh: started\nThu May 20 02:39:08 CE 
ST 2021 /opt/graylog-sidecar/filebeat-sudo-wrapper.sh: start collector: sudo /opt/filebeat/filebeat -c /opt/graylog-sidecar/generated/filebeat-wrapped.conf -strict.perms=false\n" 
time="2021-05-20T02:39:16+02:00" level=info msg="[filebeat-wrapped] Configuration change detected, rewriting configuration file." 
time="2021-05-20T02:39:17+02:00" level=info msg="[filebeat-wrapped] Starting (exec driver)" ```
### Environment

* Sidecar Version: 1.0.1
* Graylog Version: 4.x
* Operating System: RHEL