Rewrite_tag not working as expected when we used Namespaced Custom Resource Definitions

Describe the issue

I have been diligently following the documentation to establish Namespaced Custom Resource Definitions (CRDs) within our environment. However, during the setup process, I encountered an error message that reads:

"Created task=0x7f86a808f6a0 id=0 without routes, dropping."

This error suggests that Fluent Bit, the logging aggregator we're using, encountered a task without a defined route for processing. Despite following the setup instructions provided in the documentation, I'm uncertain about the root cause of this issue.

Context:

Fluent Operator is deployed via Helm. Fluent Bit is integrated as our logging aggregator and configured using the following namespace label:
```
namespaceFluentBitCfgSelector:
matchLabels:
onboardedFluentbit: "true"
```
ClusterInput looks like

apiVersion: fluentbit.fluent.io/v1alpha2
 kind: ClusterInput
metadata:
  name: test
   labels:
    fluentbit.fluent.io/enabled: "true"
    fluentbit.fluent.io/component: logging
spec:
  tail:
     tag: kube.var.log.container.test
     path: "/var/log/containers/*.log"
     excludePath: ""
     parser: cri
     refreshIntervalSeconds: 10
     memBufLimit: 100MB
     skipLongLines: true
     db: /fluent-bit/tail/pos.db
     dbSync: Normal

Fluentbit config for namespace app-1 looks like

apiVersion: fluentbit.fluent.io/v1alpha2
kind: FluentBitConfig
metadata:
  name: test
  namespace: app-1
   labels:
     onboardedFluentbit: "true"
spec:
   outputSelector:
     matchLabels:
       fluentbit.fluent.io/enabled: "true"
       app.kubernetes.io/name: "test"

output Crds looks like

apiVersion: fluentbit.fluent.io/v1alpha2
kind: Output
metadata:
  name: stdout
  namespace: app-1
  labels:
      fluentbit.fluent.io/enabled: "true"
      app.kubernetes.io/name: "test"
spec:
  match: kube.var.log.container.test
  stdout:
    format: msgpack

config which is stored in secret named fluent-bit-config looks like

[Service]
    Http_Server    true
    Parsers_File    parsers.conf
[Input]
    Name    tail
    Path    /var/log/containers/*.log
    Refresh_Interval    10
    Skip_Long_Lines    true
    DB    /fluent-bit/tail/pos.db
    DB.Sync    Normal
    Mem_Buf_Limit    100MB
    Parser    cri
    Tag    kube.var.log.container.test
[Filter]
    Name    kubernetes
    Match    kube.*
    Kube_URL    https://kubernetes.default.svc:443
    Kube_CA_File    /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File    /var/run/secrets/kubernetes.io/serviceaccount/token
    Labels    false
    Annotations    false
[Filter]
    Name    nest
    Match    kube.*
    Operation    lift
    Nested_under    kubernetes
    Add_prefix    kubernetes_
[Filter]
    Name    modify
    Match    kube.*
    Remove    stream
    Remove    kubernetes_pod_id
    Remove    kubernetes_host
    Remove    kubernetes_container_hash
[Filter]
    Name    nest
    Match    kube.*
    Operation    nest
    Wildcard    kubernetes_*
    Nest_under    kubernetes
    Remove_prefix    kubernetes_
[Filter]
    Name    rewrite_tag
    Match    kube.*
    Rule    $kubernetes['namespace_name'] ^(app-1)$ 5d24f63865268abd3e56d1e2b430162a.$TAG false
    Emitter_Name    re_emitted_5d24f63865268abd3e56d1e2b430162a
[Output]
    Name    stdout
    Match    5d24f63865268abd3e56d1e2b430162a.kube.var.log.container.test
    Format    msgpack

It got to do with rewrite_tag which is added after having namespaced crs. it's probably not able to tell the dynamic matching works

Thanks

How did you install fluent operator?

Fluent Operator is deployed via Helm

Additional context

No response

fluent / fluent-operator