Managing Vector configuration in Kubernetes environment via CRDs

MOZGIII commented 4 years ago

We can add support for managing Vector configuration via Custom Resouce Definitions when running in the Kubernetes environment.

Motivation

Allow separation of responsibilities - applications can ship with their own Vector config bits, prepared by the same teams that works on the service, rather than the ops team doing all the work
Rely on native Kubernetes capabilities for access control and configuration management - fewer headaches for the ops team.
Enable sharing a single Vector DaemonSet instance among multiple independent cluster users, utilizing resources more efficiently.

Context

This was explored briefly at the RFC.
There is the competition doing this already, for instance banzaicloud logging operator uses fluentd.
In the current state of things, a Vector instance is deployed into the cluster, and it has a centralized configuration, managed as part of the Vector deployment.
There currently are ways to enable sharing this Vector instance and allowing configuration to be gathered from multiple apps, but all of those are hacky; we can offer a proper solution.

How it would look like

This is just a rough example to illustrate the idea, NOT the proposed design for the actual architecture or the CRDs.

Let there be a Vector instance is deployed in a Kubernetes cluster, in a namespace vector.

Then in another namespace, custom resources are added as part of an app (let's say billing app) deployment:

apiVersion: vector.dev/v1beta1
kind: Sink
metadata:
  name: billing-s3
  namespace: billing
spec:
  type: aws_s3
  raw: |
    bucket = "billing-logs-bucket"
    compression = "gzip"
    region = "us-east-1"
    encoding.codec = "ndjson"
---
apiVersion: vector.dev/v1beta1
kind: Flow
metadata:
  name: billing-app
  namespace: billing
spec:
  transforms:
  - name: parse_json
    type: parse_json
    raw: |
      drop_invalid = true
  - name: mark_flow
    type: add_fields
    raw: |
      fields.flow = "billing"
  sinkRefs:
    - billing-s3
  match:
    - select:
        labels:
          app: billing

This would configure Vector to take logs with pod labels app: billing to go through a custom log aggregation flow, so that they arrive directly to the S3 bucket.

Effectively, Vector would build the following config after picking up the custom resources above:

[sources.kubernetes_logs]
  type = "kubernetes_logs"

# ... the rest of the config ...

[transforms.dynamic-billing-billing-app-json_parser]
  type = "json_parser"
  inputs = ["kubernetes_logs"]
  drop_invalid = true

[transforms.dynamic-billing-billing-app-mark_flow]
  type = "add_fields"
  inputs = ["dynamic-billing-billing-app-json_parser"]
  fields.flow = "billing"

[sinks.dynamic-billing-billing-s3]
  type = "aws_s3"
  inputs = ["dynamic-billing-billing-app-mark_flow"]
  bucket = "billing-logs-bucket"
  compression = "gzip"
  region = "us-east-1"
  encoding.codec = "ndjson"

Where names are roughly at the form of <crd>.dynamic-<namespace>-<unit-name>, like transforms.dynamic-billing-billing-app-json_parser or sinks.dynamic-billing-billing-s3.

Refs:

https://kubernetes.io/blog/2020/09/16/gsoc20-building-operators-for-cluster-addons/

afoninsky commented 4 years ago

Indeed, operator-way is more native for k8s and gives more flexibility.

Few more thoughts related to a topic:

Various flows may belong to different vector instances. It can be useful to have "Kind: Vector" with specification how to deploy Vector itself (instances count, daemonset/deployment/statefulset, antiaffinity, resources requests/limits, hpa) and specify it in sinks/flows. It brings more flexibility in deploying, ability to control vector components more granular, use "microservice-like" approach when broken components don't not affect other flows.
It's good to support clean yaml instead/together with described:
```
raw: |
  drop_invalid = true
```
=>
```
config:
  drop_invalid: true
```
Use case: ability to use native tools to generate manifests from templates (jsonnet/tanka, pulumi, etc...)
In case of operators it's possible to bring maturity model which will be useful for companies. Just an example of idea: https://sdk.operatorframework.io/docs/operator-capabilities/

binarylogic commented 4 years ago

Thanks @afonisky. @MOZGIII, where do you think this sits in our Kubernetes roadmap phases? 2?

MOZGIII commented 4 years ago

This would be phase 3. We'll probably need a full-blown RFC for this.

zvlb commented 1 year ago

Now you can use Kubernetes Vector Operator for configuring Vector in Kubernetes with CRDs

yuriy-yarosh commented 1 year ago

I think it would've make more sense to provide a backwards compatibility layer with Grafana Agent Operator instead, and reuse the respective CRD's that had been fairly widespread.

Also it would've been really nice if it was possible to use Vector as an agent for Grafana Faro,

spencergilbert commented 1 year ago

I think it would've make more sense to provide a backwards compatibility layer with Grafana Agent Operator instead, and reuse the respective CRD's that had been fairly widespread.

There's probably some amount of re-use that can be done, but a quick look at those CRD's it may be difficult to translate some things. Definitely worth consideration.

Also it would've been really nice if it was possible to use Vector as an agent for Grafana Faro,

Feel free to open a feature request for this!

yuriy-yarosh commented 1 year ago

Definitely worth consideration.

@spencergilbert Yes, backwards compat, with better perf, would make Vector into a drop-in replacement for grafana agent stack.

Added #17591

vectordotdev / vector