Dynamically include configuration files

Hey, I want to dynamically add config files to fluentbit using HOT_RELOAD and @INCLUDE functionallity. When creating empty directory and pointing @INCLUDE to it like this:

@INCLUDE /fluent-bit/config/*.conf

i receive the following error:

[2024/01/14 15:30:41] [ warn] [read_glob] glob: [/fluent-bit/config/*.conf] no match
[2024/01/14 15:30:41] [error] configuration file contains errors, aborting.

And when i try adding a temporary config file to that directory called temp.conf with the following configuration:

[OUTPUT]
    Name   stdout
    Match *

i get the following error:

[2024/01/14 10:48:59] [ warn] [config] Read twice. path=/fluent-bit/config/temp.conf
[2024/01/14 10:48:59] [error] configuration file contains errors, aborting.

Without @INCLUDE my configuration works correctly with no errors.

How can i add configuration files dynamically? how can i fix the @INCLUDE error? I am using the latest fluentbit version v2.2.1 on AWS EKS v1.24.

@LeonD9 it's not clear if you have any config at all in the first instance - if you just have a config file that only includes other files and those files do not exist then it is a legitimate problem - what config would it be?

In the second case where you create a file with an input, the config you posted will not work - there is no stdout input and an error is reported along those lines for me.

$ ls -lR
.:
total 14
drwxrwxr-x 2 pat pat  3 Jan 15 10:45 config
-rw-rw-r-- 1 pat pat 36 Jan 15 10:43 fluent-bit.conf

./config:
total 5
-rw-rw-r-- 1 pat pat 33 Jan 15 10:45 stdout.conf
$ cat fluent-bit.conf 
@INCLUDE /fluent-bit/config/*.conf

$ cat config/stdout.conf 
[INPUT]
  name stdout
  tag test

$ docker run --rm -it -v $PWD/config:/fluent-bit/config:ro -v $PWD/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro fluent/fluent-bit:2.2.2
Fluent Bit v2.2.2
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

____________________
< Fluent Bit v2.2.2 >
 -------------------
          \
           \
            \          __---__
                    _-       /--______
               __--( /     \ )XXXXXXXXXXX\v.
             .-XXX(   O   O  )XXXXXXXXXXXXXXX-
            /XXX(       U     )        XXXXXXX\
          /XXXXX(              )--_  XXXXXXXXXXX\
         /XXXXX/ (      O     )   XXXXXX   \XXXXX\
         XXXXX/   /            XXXXXX   \__ \XXXXX
         XXXXXX__/          XXXXXX         \__---->
 ---___  XXX__/          XXXXXX      \__         /
   \-  --__/   ___/\  XXXXXX            /  ___--/=
    \-\    ___/    XXXXXX              '--- XXXXXX
       \-\/XXX\ XXXXXX                      /XXXXX
         \XXXXXXXXX   \                    /XXXXX/
          \XXXXXX      >                 _/XXXXX/
            \XXXXX--__/              __-- XXXX/
             -XXXXXXXX---------------  XXXXXX-
                \XXXXXXXXXXXXXXXXXXXXXXXXXX/
                  ""VXXXXXXXXXXXXXXXXXXV""

[2024/01/15 10:47:13] [error] [config] section 'stdout' tried to instance a plugin name that don't exists
[2024/01/15 10:47:13] [error] configuration file contains errors, aborting.

If I use a valid input like cpu then it works fine:

$ cat config/stdout.conf 
[INPUT]
  name cpu
  tag test
$ docker run --rm -it -v $PWD/config:/fluent-bit/config:ro -v $PWD/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro fluent/fluent-bit:2.2.2
Fluent Bit v2.2.2
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

____________________
< Fluent Bit v2.2.2 >
 -------------------
          \
           \
            \          __---__
                    _-       /--______
               __--( /     \ )XXXXXXXXXXX\v.
             .-XXX(   O   O  )XXXXXXXXXXXXXXX-
            /XXX(       U     )        XXXXXXX\
          /XXXXX(              )--_  XXXXXXXXXXX\
         /XXXXX/ (      O     )   XXXXXX   \XXXXX\
         XXXXX/   /            XXXXXX   \__ \XXXXX
         XXXXXX__/          XXXXXX         \__---->
 ---___  XXX__/          XXXXXX      \__         /
   \-  --__/   ___/\  XXXXXX            /  ___--/=
    \-\    ___/    XXXXXX              '--- XXXXXX
       \-\/XXX\ XXXXXX                      /XXXXX
         \XXXXXXXXX   \                    /XXXXX/
          \XXXXXX      >                 _/XXXXX/
            \XXXXX--__/              __-- XXXX/
             -XXXXXXXX---------------  XXXXXX-
                \XXXXXXXXXXXXXXXXXXXXXXXXXX/
                  ""VXXXXXXXXXXXXXXXXXXV""

[2024/01/15 10:48:48] [ info] [fluent bit] version=2.2.2, commit=eeea396e88, pid=1
[2024/01/15 10:48:48] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/01/15 10:48:48] [ info] [cmetrics] version=0.6.6
[2024/01/15 10:48:48] [ info] [ctraces ] version=0.4.0
[2024/01/15 10:48:48] [ info] [input:cpu:cpu.0] initializing
[2024/01/15 10:48:48] [ info] [input:cpu:cpu.0] storage_strategy='memory' (memory only)
[2024/01/15 10:48:48] [ info] [sp] stream processor started

I saw the same behaviour with 2.2.1 and 2.2.2 so it is not something resolved by 2.2.2.

Please can you provide more details on how to reproduce it locally? I would also check the actual mounted filesystem to make sure you have not ended up with some symlinks or other cycles in the config directory.

@patrick-stephens You are correct my mistake regarding the INPUT stdout i wrote it incorrectly in the description. I tried using it as you mentioned, this is my fluent-bit.conf file:

[SERVICE]
    Flush         5
    Daemon        off
    HTTP_Server   On
    HTTP_Listen   0.0.0.0
    HTTP_Port     2020
    Health_Check On
    Hot_Reload   On
    HC_Errors_Count 5
    HC_Retry_Failure_Count 5
    HC_Period 5
    Log_Level     info
    Parsers_File  parsers.conf

@INCLUDE /fluent-bit/config/*.conf

[INPUT]
    Name                tail
    Alias               input-kube-pods-logs
    Path                /var/log/containers/*.log
    Path_Key            path
    Offset_Key          logs.offset
    multiline.parser    cri
    Refresh_Interval    5
    tag                 kubernetes.*
    db                  /etc/fluentbit/db/kubernetes-pods.db
    threaded            on

[INPUT]
    Name                tail
    Alias               input-kube-node-logs
    Path                /var/log/*.log, /var/log/syslog, /var/log/messages
    Path_Key            path
    Parser              syslog-rfc3164
    Refresh_Interval    5
    tag                 kubernetes-nodes.*
    db                  /etc/fluentbit/db/kubernetes-nodes.db
    threaded            on

[FILTER]
    Name                kubernetes
    Alias               filter-kubernetes
    Match               kubernetes.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
    Kube_Tag_Prefix     kubernetes.var.log.containers.
    Merge_Log           Off
    K8S-Logging.Parser  On
    K8S-Logging.Exclude On
    Kube_Meta_Cache_TTL 10
    Buffer_Size         100k

[FILTER]
    Name        parser
    Alias       parse-json
    Match       *
    Key_Name    log
    Parser      json
    Reserve_Data On

 [OUTPUT]
    Name        forward
    Alias       forward-to-fluentd
    Match       *
    Upstream    /fluent-bit/etc/upstream.conf
    Port        24224
    Retry_Limit False

and this is what i put in the /fluent-bit/config/temp.conf file:

[INPUT]
  name cpu
  tag test

And i receive the following error:

Fluent Bit v2.2.1
* Copyright (C) 2015-2023 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
       \ || /
       = ** =
        /__\ 
       /____\
      /______\
     /________\
    /__________\
   /____________\
  /__Fluent Bit__\
 /__|__v2.2.1_|___\
/___\_________/____\
        |||
        |||
       =====

[2024/01/15 10:59:37] [ warn] [config] Read twice. path=/fluent-bit/config/temp.conf
[2024/01/15 10:59:37] [error] configuration file contains errors, aborting.

Also this are my volume mounts:

  - name: tempdir
    emptyDir: {}
  - name: config
    projected:
      defaultMode: 0644
      configMaps:
      - name: $RELEASE-config
        items:
        - key: fluent-bit.conf.tpl
          path: fluent-bit.conf
  - name: parsers
    projected:
      defaultMode: 0644
      configMaps:
      - name: $RELEASE-config
        items:
        - key: parsers.conf.tpl
          path: parsers.conf
  - name: upstream
    projected:
      defaultMode: 0644
      configMaps:
      - name: $RELEASE-config
        items:
        - key: upstream.conf.tpl
          path: upstream.conf
  - name: temp
    projected:
      defaultMode: 0644
      configMaps:
      - name: $RELEASE-config
        items:
        - key: temp.conf.tpl
          path: temp.conf
      volumeMounts:
      - name: tempdir
        mountPath: /fluent-bit/config
      - name: temp
        mountPath: /fluent-bit/config/temp.conf
        subPath: temp.conf
      - name: config
        mountPath: /fluent-bit/etc/fluent-bit.conf
        subPath: fluent-bit.conf
      - name: parsers
        mountPath: /fluent-bit/etc/parsers.conf
        subPath: parsers.conf
      - name: upstream
        mountPath: /fluent-bit/etc/upstream.conf
        subPath: upstream.conf
      - name: varlog
        mountPath: /var/log
        readOnly: true

I do see temp.conf file with the correct configuration being mounted to the pod and without @INCLUDE the fluent-bit.conf configuration works correctly as well.

I just need a temporary config file in a new directory so i will be able to upload additional configs there and use there with @INCLUDE functionality, this way worked in the version before v2.

I am having the same problem. If I remove my @INCLUDE statement it works on version 2.2. If I revert to 1.99 it works with @iNCLUDE

I am using a helm chart to deploy fluentbit and mounting the configmaps.

I have also tried adding a command to specify the config and working directory which did not help.

Fluent Bit v2.2.2

Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
https://fluentbit.io

< Fluent Bit v2.2.2 >

      \
       \
        \          __---__
                _-       /--______
           __--( /     \ )XXXXXXXXXXX\v.
         .-XXX(   O   O  )XXXXXXXXXXXXXXX-
        /XXX(       U     )        XXXXXXX\
      /XXXXX(              )--_  XXXXXXXXXXX\
     /XXXXX/ (      O     )   XXXXXX   \XXXXX\
     XXXXX/   /            XXXXXX   \__ \XXXXX
     XXXXXX__/          XXXXXX         \__---->

---_ XXX/ XXXXXX \ / - --_/ /\ XXXXXX / --/= -\ / XXXXXX '--- XXXXXX -\/XXX\ XXXXXX /XXXXX \XXXXXXXXX \ /XXXXX/ \XXXXXX > _/XXXXX/ \XXXXX--/ -- XXXX/ -XXXXXXXX--------------- XXXXXX- \XXXXXXXXXXXXXXXXXXXXXXXXXX/ ""VXXXXXXXXXXXXXXXXXXV""

[2024/01/16 20:15:31] [ warn] [config] Read twice. path=/fluent-bit/config.d/spot-log.conf [2024/01/16 20:15:31] [error] configuration file contains errors, aborting.

[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    debug
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020
    Parsers_File /fluent-bit/etc/parser.conf

@INCLUDE /fluent-bit/config.d/spot-log.conf

spot-log.conf:
----
[INPUT]
    Name             tail
    Path             /var/log/containers/spot-log*.log
    Parser           docker
    Tag              kube.spot-log*
    Mem_Buf_Limit    100MB
    Docker_Mode      On
    Skip_Long_Lines  On
    Alias            spot-log_input
[FILTER]
    Name             kubernetes
    Match            kube.spot-log*
    Kube_URL         https://kubernetes.default.svc.cluster.local:443
    Kube_CA_File     /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File  /var/run/secrets/kubernetes.io/serviceaccount/token
    Kube_Tag_Prefix  kube.var.log.containers.
    Merge_Log        On
    Merge_Log_Trim   On
    Alias            spot-log_k8_filter
[OUTPUT]
    Name             datadog
    Alias            spot-log_output
    Match            kube.spot-log*
    Host             http-intake.logs.datadoghq.com
    TLS              on
    compress         gzip
    apikey           <retracted>
    dd_service       spot-log
    dd_source        fbit-eks
    dd_message_key   log

$ k describe ds fluentbit| grep -i volume -A 5
      /fluent-bit/config.d/ from configd-volume (rw)
      /fluent-bit/etc/ from config (rw)
      /var/lib/docker/containers from varlibdockercontainers (ro)
      /var/log from varlog (rw)
  Volumes:
   config:
    Type:               Projected (a volume that contains injected data from multiple sources)
    ConfigMapName:      fluentbit-main-conf
    ConfigMapOptional:  <nil>
    ConfigMapName:      fluentbit-parser
    ConfigMapOptional:  <nil>
   configd-volume:
    Type:               Projected (a volume that contains injected data from multiple sources)
    ConfigMapName:      fluent-bit-spot-logs

@LeonD9 I am dynamically adding INCLUDE statements to my main fluentbit config using a function in helm called 'range'. It will iterate over a list of config names to add. Thus I don't need to do a *

Changing the configuration to yaml configuration from the classic configuration helped fix the include issue for me:

includes:
    - ../config/*.yaml

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

This issue was closed because it has been stalled for 5 days with no activity.

fluent / fluent-bit

Dynamically include configuration files #8375

< Fluent Bit v2.2.2 >