Invalid "Format" values in "file" output module are silently ignored

[ppar]

Bug Report

Describe the bug

When, in the file output module, Format is set to an unknown value, fluent-bit assumes out_file instead of reporting an error.

To Reproduce

Create a configuration (fluent-bit.test.001.conf) with:

[SERVICE]
    Flush 1
    Daemon Off
    Log_Level debug
    HTTP_Server On
    HTTP_Listen 0.0.0.0
    HTTP_Port 8123

[INPUT]
    Name tail
    Path test.input.tail.*.log
    Tag input_tail

[OUTPUT]
    Name file
    Match *
    Format out_file
    Path test.output.file.out_file

[OUTPUT]
    Name file
    Match *
    Format random_junk
    Path test.output.file.random_junk

[OUTPUT]
    Name file
    Match *
    Path test.output.file.plain
    Format plain

[OUTPUT]
    Name file
    Match *
    Format csv
    Path test.output.file.csv

• Formats out_file, plain, csv are found in https://docs.fluentbit.io/manual/pipeline/outputs/file#out_file-format

Create directories:

mkdir test.output.file.out_file test.output.file.random_junk test.output.file.plain test.output.file.csv

Install fluent-bit 1.7.7 from homebrew on macOS 10.15.7 (19H1030) (Darwin 19.6.0)

brew install fluent-bit

Run fluent-bit in a separate terminal:

fluent-bit -c fluent-bit.test.001.conf

Create input data:

echo "$(date) barbar" >> test.input.tail.bar.log

Look at the output:

$ grep . test.output.file.*/*
test.output.file.csv/input_tail:1622245524.634801000,"Sat May 29 02:44:44 EEST 2021 barbar"
test.output.file.csv/input_tail:1622245524.634804000,"Sat May 29 02:44:48 EEST 2021 barbar"
test.output.file.out_file/input_tail:input_tail: [1622245524.634801000, {"log":"Sat May 29 02:44:44 EEST 2021 barbar"}]
test.output.file.out_file/input_tail:input_tail: [1622245524.634804000, {"log":"Sat May 29 02:44:48 EEST 2021 barbar"}]
test.output.file.plain/input_tail:{"log":"Sat May 29 02:44:44 EEST 2021 barbar"}
test.output.file.plain/input_tail:{"log":"Sat May 29 02:44:48 EEST 2021 barbar"}
test.output.file.random_junk/input_tail:input_tail: [1622245524.634801000, {"log":"Sat May 29 02:44:44 EEST 2021 barbar"}]
test.output.file.random_junk/input_tail:input_tail: [1622245524.634804000, {"log":"Sat May 29 02:44:48 EEST 2021 barbar"}]

Output from the "random_junk" format matches that from "out_file":

$ md5 -r test.output.file.*/*  | sort
5990e37913dec02b9223fa9d73195eec test.output.file.csv/input_tail
75e0a8ddd279ca4741e7622771d65087 test.output.file.plain/input_tail
ee428eb77c97bb19b70434e7b319db63 test.output.file.out_file/input_tail
ee428eb77c97bb19b70434e7b319db63 test.output.file.random_junk/input_tail

Expected behavior

Fluent-bit should have refused to start because the configuration contains an invalid format:

    Format random_junk

Your Environment

• Version used: 1.7.7
• Configuration: See above
• Operating System and version: homebrew on macOS 10.15.7 (19H1030) (Darwin 19.6.0)
• Filters and plugins: See below

$ fluent-bit -h
Usage: fluent-bit [OPTION]

Available Options
  -b  --storage_path=PATH specify a storage buffering path
  -c  --config=FILE       specify an optional configuration file
  -d, --daemon            run Fluent Bit in background mode
  -D, --dry-run           dry run
  -f, --flush=SECONDS     flush timeout in seconds (default: 5)
  -F  --filter=FILTER     set a filter
  -i, --input=INPUT       set an input
  -m, --match=MATCH       set plugin match, same as '-p match=abc'
  -o, --output=OUTPUT     set an output
  -p, --prop="A=B"        set plugin configuration property
  -R, --parser=FILE       specify a parser configuration file
  -e, --plugin=FILE       load an external plugin (shared lib)
  -l, --log_file=FILE     write log info to a file
  -t, --tag=TAG           set plugin tag, same as '-p tag=abc'
  -T, --sp-task=SQL       define a stream processor task
  -v, --verbose           increase logging verbosity (default: info)
  -w, --workdir           set the working directory
  -H, --http              enable monitoring HTTP server
  -P, --port              set HTTP server TCP port (default: 2020)
  -s, --coro_stack_size   set coroutines stack size in bytes (default: 12288)
  -q, --quiet             quiet mode
  -S, --sosreport         support report for Enterprise customers
  -V, --version           show version number
  -h, --help              print this help

Inputs
  tail                    Tail files
  dummy                   Generate dummy data
  head                    Head Input
  health                  Check TCP server health
  http                    HTTP
  collectd                collectd input plugin
  statsd                  StatsD input plugin
  serial                  Serial input
  stdin                   Standard Input
  syslog                  Syslog
  exec                    Exec Input
  tcp                     TCP
  mqtt                    MQTT, listen for Publish messages
  forward                 Fluentd in-forward
  random                  Random

Filters
  alter_size              Alter incoming chunk size
  aws                     Add AWS Metadata
  record_modifier         modify record
  throttle                Throttle messages using sliding window algorithm
  kubernetes              Filter to append Kubernetes metadata
  modify                  modify records by applying rules
  nest                    nest events by specified field values
  parser                  Parse events
  expect                  Validate expected keys and values
  grep                    grep events by specified field values
  rewrite_tag             Rewrite records tags
  lua                     Lua Scripting Filter
  stdout                  Filter events to STDOUT
  geoip2                  add geoip information to records

Outputs
  azure                   Send events to Azure HTTP Event Collector
  azure_blob              Azure Blob Storage
  bigquery                Send events to BigQuery via streaming insert
  counter                 Records counter
  datadog                 Send events to DataDog HTTP Event Collector
  es                      Elasticsearch
  exit                    Exit after a number of flushes (test purposes)
  file                    Generate log file
  forward                 Forward (Fluentd protocol)
  http                    HTTP Output
  influxdb                InfluxDB Time Series
  logdna                  LogDNA
  loki                    Loki
  kafka-rest              Kafka REST Proxy
  nats                    NATS Server
  nrlogs                  New Relic
  null                    Throws away events
  plot                    Generate data file for GNU Plot
  slack                   Send events to a Slack channel
  splunk                  Send events to Splunk HTTP Event Collector
  stackdriver             Send events to Google Stackdriver Logging
  stdout                  Prints events to STDOUT
  syslog                  Syslog
  tcp                     TCP Output
  td                      Treasure Data
  flowcounter             FlowCounter
  gelf                    GELF Output
  websocket               Websocket
  cloudwatch_logs         Send logs to Amazon CloudWatch
  kinesis_firehose        Send logs to Amazon Kinesis Firehose
  kinesis_streams         Send logs to Amazon Kinesis Streams
  s3                      Send to S3

Internal
 Event Loop  = kqueue
 Build Flags = FLB_HAVE_PARSER FLB_HAVE_RECORD_ACCESSOR FLB_HAVE_STREAM_PROCESSOR JSMN_PARENT_LINKS JSMN_STRICT FLB_HAVE_TLS FLB_HAVE_AWS FLB_HAVE_SIGNV4 FLB_HAVE_SQLDB FLB_HAVE_METRICS FLB_HAVE_HTTP_SERVER FLB_HAVE_FORK FLB_HAVE_TIMESPEC_GET FLB_HAVE_GMTOFF FLB_HAVE_CLOCK_GET_TIME FLB_HAVE_UNIX_SOCKET FLB_HAVE_PROXY_GO FLB_HAVE_LIBBACKTRACE FLB_HAVE_REGEX FLB_HAVE_UTF8_ENCODER FLB_HAVE_LUAJIT

Additional context How has this issue affected you?

• Silently ignoring invalid configuration can lead to missed configuration errors, hard to diagnose problems, data loss, etc.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.