With every registration request, the Sidecar performs a version check against the Graylog server.
The result then decides which kind of config assignment is expected.
Servers >= 5.0 return a type that supports multiple configs per backend. (see #441)
As a precaution, a fallback result to version 4.0.0 was added in case the version
could not be parsed from the server.
This turns out to be problematic in the following scenario.
If a Sidecar is talking to a load balancer, which sits in front of multiple
Graylog nodes, the following race condition can happen:
one node in the graylog cluster is being shut down.
the sidecar performs the version api request, but gets an error from the LB, because the request was redirected
to the node that is being terminated.
The sidecar uses the fallback to version 4.0.0 and requests a config update.
mpfz0r
commented
10 months ago
Problem description
With every registration request, the Sidecar performs a version check against the Graylog server. The result then decides which kind of config assignment is expected. Servers >= 5.0 return a type that supports multiple configs per backend. (see #441)
As a precaution, a fallback result to version 4.0.0 was added in case the version could not be parsed from the server. This turns out to be problematic in the following scenario. If a Sidecar is talking to a load balancer, which sits in front of multiple Graylog nodes, the following race condition can happen:
one node in the graylog cluster is being shut down.
the sidecar performs the version api request, but gets an error from the LB, because the request was redirected to the node that is being terminated.
The sidecar uses the fallback to version 4.0.0 and requests a config update.
The config update is parsed with the assumption that it is in the <= 5.0.0 format see https://github.com/Graylog2/collector-sidecar/blob/master/api/graylog.go#L293
This leads the sidecar to requesting assignments with the wrong format.
The wrong request reaches another Graylog node through the LB (running version >=5)
The server responds with an empty config
The sidecar stops all collectors
The request is not repeated because of ETag caching
Sidecar Version: >= 1.3
Graylog Version: >= 5.0