Environment Variables are not evaluated before @INCLUDE

[ahollist]

Bug Report

Describe the bug When setting the main configuration file to use a different output configuration file based on environment variables, the @ INCLUDE directive does not parse the value of environment variables and instead just directly uses the written line in the configuration file.

To Reproduce $ export LOG_LEVEL=DEBUG main.conf:

[SERVICE]
...
[INPUT]
...

@INCLUDE output_match_${LOG_LEVEL}.conf

output_match_DEBUG.conf:

[OUTPUT]
    Name    stdout
    Match   *

Terminal output: $ /opt/td-agent-bit/bin/td-agent-bit -c /etc/td-agent-bit/main.conf

Fluent Bit v1.3.10
Copyright (C) Treasure Data

[2020/03/10 01:45:10] [Warning] [config] I cannot open /etc/td-agent-bit//output_match_${LOG_LEVEL}.conf file
Error: Configuration file contains errors. Aborting

Expected behavior The environment variable should expand to its value and load output_match_DEBUG.conf

Your Environment

• Version used: v1.3.10 td-agent-bit
• Environment name and version: td-agent-bit via fluent bit apt servers
• Server type and version: Raspberry Pi3
• Operating System and version: Raspbian Debian Buster
• Filters and plugins: no filters or outputs are relevant here

Additional context

I have several different matches and outputs I would like to use when various environment variables are set, and I am unable to figure out a way how, though if there is a preferred different way of doing this, I am happy to learn how.

I noticed that there is an "inline config" cli input option for Fluentd, but as far as I can tell there is no equivalent for fluent bit.

Note I am calling this directly using the td-agent-bit binary, and I experienced the same issue when attempting to launch through a service unit file using the EnvironmentFile option as well.

[acarsercan]

Maybe I misunderstood this, but are you suggesting your ENV are not picked up by td-agent-bit? If so, I'm having the exact same problem with 1.4.2

[ahollist]

Environment Variables used in filter/parser/input/output sections are used correctly, but if I try to use a variable in an @include directive it is not evaluated to its value.

[edsiper]

thanks, we will fix that shortly

[adeelamin15]

@edsiper Can you assign this to me?

[koalalam]

I got similar issue in version 1.4.2 - is environment variable supported in Parser File?

[marcosflobo]

Good catch on this @ahollist .

Is there any vision about which version this enhancement will come?

[NachoRaik]

Hi @edsiper , was there any news around this issue?

[frosiere]

This issue would definitly help handling conditional outputs. Can we expect it in the next release of Fluent-bit? Any workaround? Thanks for your help.

[garciarodrigor]

Hi @edsiper , was there any news around this issue? We have the same issue, any workaround? Thanks for your help.

[Clausewitz45]

Hi, I created a feature request to be able to use Environment variables in Tag also - maybe if this will be fixed, 5240 can be addressed also.

[stevenolen]

I'm also going to have to do some awkward file management since this is missing. any update?

[VariLilley]

Any update on this issue? Such a simple and powerful feature hasn't been implemented for nearly 3 years.

[ahollist]

For anyone who might come across this issue at some point (and it seems like some of you have, throughout the years 😉 ), I wanted to share what our usecase was, and how we worked around the limitations of fluentbit's environment variable evaluation (whether or not there was a better solution available at the time, or now, is irrelevant--we got things working as we intended them):

The Problem:

We have a remote device feeding us logs, but we want to reduce network usage as much as possible while still allowing a way to get more verbose logs during debugging sessions. We can remotely send a new log level to the device through MQTT, which gets applied as an environment variable to our td-agent-bit.service via the etc/system/systemd/td-agent-bit.d/logging-environment.conf file.

Using @INCLUDE to set an output match file determined by an environment variable didn't work in the version of fluentbit we were using 1.3.10

The Workaround:

Each of our logs getting parsed by td-agent-bit has a level field, so we can tag each log as it comes through with a defined level as appropriate (certain logs have already been tagged with app. in our configuration):

[FILTER]
    Name rewrite_tag
    Match app.*
    Rule $level ^(warn|error|panic|fatal)$ default.$TAG false

We can then filter the output, in our case to Stackdriver, with the appropriate tag level:

# LOG_LEVEL is set by an EnvironmentFile override provided in /etc/systemd/system/td-agent-bit.d/
# LOG_LEVEL might be debug ("") or "default", so the Match is * or default*
[OUTPUT]
    Name  stackdriver
    Match ${LOG_LEVEL}*
    google_service_credentials ${SERVICE_ACCOUNT_CREDENTIALS_PATH}
    resource generic_node
    location <X>
    namespace <X>
    node_id <X>

So, as this is currently configured fluentbit will now prepend a default. tag to the existing tag, for entries with level warn|error|panic|fatal, meaning that if the LOG_LEVEL environment variable is set to default, the output will only receive entries with those applicable severity level tags.

etc/system/systemd/td-agent-bit.d/logging-environment.conf

[Service]
# If the LOG_LEVEL does not exist, the filter is configured to pass all records to the output
EnvironmentFile=-/run/fluentbit/LOG_LEVEL

We also have to restart the service when the LOG_LEVEL file is changed, so we have this service and accompanying .path file:

etc/system/systemd/log-watcher.service

[Unit]
Description=restart td-agent-bit log services when Log Level or metadata changes

[Service]
Type=oneshot
ExecStart=/bin/systemctl restart td-agent-bit.service

[Install]
WantedBy=multi-user.target

etc/system/systemd/log-watcher.path

[Unit]
Description=Watcher for Logging Level or metadata changes

[Path]
PathModified=/plantiga/share/LOG_LEVEL
...

So what we're left with is a system that has customizable log level filtering (based on tagging entries as they come through the input/parser/filter levels of fluentbit), still uses an environment variable/file to set/change the log level, and manages to work around the environment variable expansion in @INCLUDE by just using it in a match configuration instead.