[Fleet] Support injected routing rules during integration installation

[kpollich]

Ref https://github.com/elastic/package-spec/issues/514 Follow-up from https://github.com/elastic/kibana/issues/155910

When integrations are installed, Fleet should honor all injected routing rules defined by a given data stream manifest in addition to all local rules. The local rules support is handled by https://github.com/elastic/kibana/issues/155910

Given a data stream manifest as follows

# nginx/data_stream/nginx/manifest.yml
title: Nginx logs
type: logs

# This is a catch-all "sink" data stream that routes documents to 
# other datasets based on conditions or variables
dataset: nginx

# Ensures agents have permissions to write data to `logs-nginx.*-*`
elasticsearch.dynamic_dataset: true
elasticsearch.dynamic_namespace: true

routing_rules:
  # "Local" routing rules are included under this current dataset, not a special case
  nginx:
    # Route error logs to `nginx.error` when they're sourced from an error logfile
    - dataset: nginx.error
      if: "ctx?.file?.path?.contains('/var/log/nginx/error')"
      namespace:
        - {{labels.data_stream.namespace}}
        - default

    # Route access logs to `nginx.access` when they're sourced from an access logfile
    - dataset: nginx.access
      if: "ctx?.file?.path?.contains('/var/log/nginx/access')"
      namespace:
        - {{labels.data_stream.namespace}}
        - default

  # --- Rules below this line are what we're supporting here

  # Route K8's container logs to this catch-all dataset for further routing
  k8s.router: 
    - dataset: nginx
      if: "ctx?.container?.image?.name == 'nginx'"
      namespace:
        - {{labels.data_stream.namespace}}
        - default

  # Route syslog entries tagged with nginx to this catch-all dataset
  syslog:
    - dataset: nginx
      if: "ctx?.tags?.contains('nginx')"
      namespace:
        - {{labels.data_stream.namespace}}
        - default

The ingest pipelines for the k8s.router and syslog datastreams should be updated as follows

// logs-k8s.router-1.2.3
{
    "processors": [
        {
            "reroute": {
                "tag": "logs-k8s.router",
                "if": "ctx?.container?.image?.name == 'nginx'",
                "dataset": "{{container.image.name}}",
                "namespace": [
                    "{{labels.data_stream.namespace}}",
                    "default"
                ]
            }
        }
    ]
}

// logs-syslog-1.2.3
{
    "processors": [
        {
            "reroute": {
                "tag": "logs-syslog",
                "dataset": "{{container.image.name}}",
                "if": "ctx?.tags?.contains('nginx')",
                "namespace": [
                    "{{labels.data_stream.namespace}}",
                    "default"
                ]
            }
        }
    ]
}

One important detail to note: injected routing rules always appear AFTER local routing rules for a given data stream. This prevent unintended short-circuiting.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[joshdover]

@kpollich are we still doing this one for the MVP or are we only focusing on #155910?

[kpollich]

@joshdover We'll only be doing "local" routing rules for the MVP in 8.10. So yes, the scope has been pared down to https://github.com/elastic/kibana/issues/155910 for MVP.

[jlind23]

@juliaElastic Do we know how this should work with input-packages? Can we specify the input package dataset as target dataset for routing rules? Because the problem with input packages is that the dataset is set at the installation time by the user.

[juliaElastic]

@jlind23 I think dynamic dataset name in input packages was not considered to be supported by this enhancement. We could come up with a way to support it (in a new issue?), though it's tricky as the dataset name can be anything. E.g. use a wildcard to map to any dataset name under custom_logs input package

- source_dataset: k8s.router
  rules:
    - target_dataset: custom_logs.*
       if: "true == true"

[jlind23]

What about this one: https://github.com/elastic/package-spec/issues/566 ?

[juliaElastic]

What about this one: elastic/package-spec#566 ?

We can use that issue to add support for routing rules in input packages, that looks like a use case for local routing rules, not injected.

[joshdover]

See my comment on the other issue - I'm skeptical that we need routing rules in input packages at all.

[flash1293]

@kpollich Can this be closed or is there an aspect of this not implemented yet?