fluent / fluent-operator

Operate Fluent Bit and Fluentd in the Kubernetes way - Previously known as FluentBit Operator
Apache License 2.0
555 stars 229 forks source link

Adding Fluentbit's unified networking interface #1217

Closed localleon closed 6 days ago

localleon commented 1 week ago

What this PR does / why we need it:

This PR adds support for the fluentbit unified networking interface for many output plugins. See the offical docs for more information about the feature https://docs.fluentbit.io/manual/administration/networking

You can use the advanced networking options in most Clusteroutput-Plugins like this:

apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
  name: tcp
  namespace: fluent
  labels:
    fluentbit.fluent.io/enabled: "true"
spec:
  match: "*"
  tcp:
    host: 127.0.0.1
    port: 9090
    format: json_lines
    networking:
      connectTimeout: 20
      connectTimeoutLogError: true
      DNSMode: "UDP"
      DNSPreferIPv4: false
      DNSResolver: "ASYNC"
      keepalive: "on"
      keepaliveIdleTimeout: 40
      keepaliveMaxRecycle: 3000
      maxWorkerConnections: 10

The operator will then generate the fluentbit-configuration according to the documentation. When running kubectl get secrets -n fluent fluent-bit-config -o json | jq -r '.data["fluent-bit.conf"]' | base64 -d we can see that our configuration now includes everything that we defined in the CRD!

[Input]
    Name    dummy
    Tag    my_dummy
[Output]
    Name    stdout
    Match    *
[Output]
    Name    tcp
    Match    *
    Host    127.0.0.1
    Port    9090
    Format    json_lines
    net.connect_timeout    20
    net.connect_timeout_log_error    true
    net.dns.mode    UDP
    net.dns.prefer_ipv4    false
    net.dns.prefer_ipv4    ASYNC
    net.keepalive    on
    net.keepalive_idle_timeout    40
    net.keepalive_max_recycle    3000
    net.max_worker_connections    10

Which issue(s) this PR fixes:

Fixes #247

Does this PR introduced a user-facing change?

Yes, most CRDs are updated to include the net-options and have the corresponding documentation updated

Implement the unified networking interface (https://docs.fluentbit.io/manual/administration/networking) for many supported output plugins (e.g splunk, tcp, syslog). Most CRDs of Output plugins now support setting common network settings e.g net.keepalive

Additional documentation, usage docs, etc.:

This PR is a 1:1 implementation of the provided documentation at https://docs.fluentbit.io/manual/administration/networking

benjaminhuo commented 6 days ago

That's great, thanks!

benjaminhuo commented 6 days ago

That's great, Thank you! @localleon cc @wanjunlei

localleon commented 6 days ago

@benjaminhuo when do you guys usally do releases? I would like to get this feature into an official release soon because we need it for our company rollout of the operator ;-)