Service crashes on duplicate IDs

[tptignor]

Hi. We just discovered this operator and it's been very impressive so far. However, we did see a bug through the use of ID fields which I wanted to report. The first error occurred when using similar filter blocks in two different namespaces, each of the form:

<filter kube.elasticsearch.dramacycle.dramacycle>
  @type concat
  @id filter_ml_drama
...
</filter>

After changing the IDs for one of the ConfigMaps, I found myself still facing this second error. This was caused by the universal output I had configured for the kube-system namespace.

<plugin es-local>
  @type elasticsearch
  @id es-local
...
</plugin>

Evidently while I had just specified the es-local ID in one ConfigMap, the reloader generated two fluentd config sections for it. Thus the second error (for what looks like the same bug.)

The effect of both errors is that the DaemonSet pods repeatedly crash until the config is fixed.

bos-mpt5c:cake ttignor$ k logs kfo-log-router-zrjcg fluentd
2020-04-17 13:40:15 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2020-04-17 13:40:15 +0000 [info]: adding rewrite_tag_filter rule: unit [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00007f713d8b90d8 @keys="unit">, /^\
(.+)$/, "", "systemd.$1"]
2020-04-17 13:40:15 +0000 [info]: adding rewrite_tag_filter rule: kubernetes_namespace_container_name [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x000\
07f713aab9b10 @keys="kubernetes_namespace_container_name">, /^(.+)$/, "", "kube.$1"]
2020-04-17 13:40:15 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Duplicated plugin id `filter_ml_drama`\
. Check whole configuration and fix it."
bos-mpt5c:cake ttignor$

ERROR Ex 2:
bos-mpt5c:cake ttignor$ k logs kfo-log-router-v9bp2 -c fluentd
2020-04-17 13:48:11 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2020-04-17 13:48:11 +0000 [info]: adding rewrite_tag_filter rule: unit [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x00007f5b640b9548 @keys="unit">, /^\
(.+)$/, "", "systemd.$1"]
2020-04-17 13:48:12 +0000 [info]: adding rewrite_tag_filter rule: kubernetes_namespace_container_name [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x000\
07f5b61758640 @keys="kubernetes_namespace_container_name">, /^(.+)$/, "", "kube.$1"]
2020-04-17 13:48:12 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Duplicated plugin id `es-local`. Check\
 whole configuration and fix it."
bos-mpt5c:cake ttignor$

[tjorourke]

We will try and recreate this - thank you

[Cryptophobia]

Unfortunately, this kind of configuration error is not caught by fluentd config validation command. The reason is that fluentd generates the main config based on the defined behaviour of the plugin and the IDs involved.