Allow control the order of Output Plugin

[sugaf1204]

Is your feature request related to a problem? Please describe.

In the Fluentd, I can't completely control the order of output plugin.

The order of plugins cannot be controlled between them that have different values for the tag field of output plugin. For example, when using rewrite_tag_filter, the problem arises because we cannot control the order of customPlugin that matches the tag before rewritten and s3 plugin that matches it after rewritten.

apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterOutput
metadata:
  name: example
spec: 
  outputs: 
  - tag: "k8s.log.*"
     customPlugin:
       config: |
         <match k8s.log.*>
           @type rewrite_tag_filter
           <rule>
             key $.type
             pattern ^(.+)$
             tag $1.log
           </rule>
         </match>
  - tag: "A.log"
     s3:
     ......
  - tag: "B.log" 
     forward:
     ......
  - tag: "*.log" 
     elasticsearch:
     ......
  - tag: "k8s.**" # catch all
     stdout:
     ......

The order in which these are written out to config is undifined.

Describe the solution you'd like

In the current implementation, within a group of plugins with the same tag field value, the order of them can probably be controlled by metadata.name of CR. (grouping by tag field value: https://github.com/fluent/fluent-operator/blob/v2.7.0/apis/fluentd/v1alpha1/helper.go#L296) (sorted by metadata.name: https://github.com/fluent/fluent-operator/blob/v2.7.0/apis/fluentd/v1alpha1/helper.go#L124)

I can solve this problem by writing everything in single customPlugin, but it is not good because it raises the possibility of managing a very huge single config in the future. I want to control the order of all plugins according to the order of the outputs array defined in one CR, not just the metadata.name.

Additional context

No response

[benjaminhuo]

@sugaf1204 https://github.com/fluent/fluent-operator/pull/1106 pls take a look if this PR from @antrema fixed the problem you mentioned

[sugaf1204]

Sorry, I just misunderstood the behavior of rewrite_tag_filter.

I don't know how to determine the order of the rewrite_tag_filter plugin and the plugins that catch the tags that did not match any of the others.

Is there way to resolve it?

[SvenThies]

@sugaf1204, the configuration will be built always in the same way, as #1106 describes. The execution of filters is then also always the same. So the order is controlled by your configuration.

Is this still relevant? Can we close this issue?