Dynamic routing based on namespace

A note for the community

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

We currently have an EKS cluster where each client has their own namespace. We want to be able to send each client logs to an EC2 instance that belongs to the client.

Attempted Solutions

We tried deploying a vector instance per namespace, however, we quickly realised that having 300 namespaces, it'll be a vector per node per namespace so that'll be 300 pods per node, which obviously isn't ideal.

Proposal

We're hoping we will be able to do some intelligent namespace based routing e.g.

data_dir: /vector-data-dir
sources:
  kube_logs:
    type: kubernetes_logs
    auto_partial_merge: true
    # extra_field_selector: "metadata.namespace=${namespace}"

transforms:
  add_metadata:
    inputs:
      - kube_logs
    type: remap
    source: |
      .namespace = .kubernetes.namespace
      .cluster_name = "${cluster_name}"

sinks:
  namespace_router:
    type: vector
    inputs:
      - add_metadata
    address: "{{ namespace }}-0.hosting.test.com:3150"

however type: vector doesn't have address template-able.

If this isn't the best way of handling, please let us know what the best practice is in such cases

References

No response

Version

0.40.1

vectordotdev / vector