search

fluent / fluentd

Fluentd: Unified Logging Layer (project under CNCF)
https://www.fluentd.org
Apache License 2.0
12.92k stars 1.34k forks source link

Quoted string parser reports end of file when encountering an empty line #4691

Open Alainx277 opened 3 weeks ago

Alainx277 commented 3 weeks ago

Describe the bug

When quoting a string it is possible to have text that goes on multiple lines. If the multiline string contains an empty line however, the parser fails even if the string is properly closed.

To Reproduce

Create a fluentd config like this:

<filter test>
  @type record_transformer
  <record>
    hello "world

"
  </record>
</filter>

Run fluentd in dry-run mode:

fluentd --dry-run -c /tmp/fluent.conf
Traceback (most recent call last):
    18: from /Users/me/.rbenv/versions/2.5.1/bin/fluentd:23:in `<main>'
    17: from /Users/me/.rbenv/versions/2.5.1/bin/fluentd:23:in `load'
    16: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/bin/fluentd:15:in `<top (required)>'
    15: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:85:in `require'
    14: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:85:in `require'
    13: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/command/fluentd.rb:351:in `<top (required)>'
    12: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/supervisor.rb:624:in `configure'
    11: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/supervisor.rb:682:in `setup_global_logger'
    10: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config.rb:52:in `build'
     9: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config.rb:71:in `parse'
     8: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:33:in `parse'
     7: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:44:in `parse!'
     6: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:96:in `parse_element'
     5: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:96:in `parse_element'
     4: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:131:in `parse_element'
     3: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/literal_parser.rb:75:in `parse_literal'
     2: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/literal_parser.rb:82:in `scan_string'
     1: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/literal_parser.rb:114:in `scan_double_quoted_string'
/Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/basic_parser.rb:92:in `parse_error!': unexpected end of file in a double quoted string at fluent.conf line 6,17 (Fluent::ConfigParseError)
  5:   <record>
  6:     hello "world

     -----------------^
  7:

Expected behavior

The string should be parsed like any other multiline string. In this case I'd expect hello to be set to "world\n\n".

Your Environment

- Fluentd version: 1.16.6

Your Configuration

<filter test>
  @type record_transformer
  <record>
    hello "world

"
  </record>
</filter>

Your Error Log

Traceback (most recent call last):
    18: from /Users/me/.rbenv/versions/2.5.1/bin/fluentd:23:in `<main>'
    17: from /Users/me/.rbenv/versions/2.5.1/bin/fluentd:23:in `load'
    16: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/bin/fluentd:15:in `<top (required)>'
    15: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:85:in `require'
    14: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:85:in `require'
    13: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/command/fluentd.rb:351:in `<top (required)>'
    12: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/supervisor.rb:624:in `configure'
    11: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/supervisor.rb:682:in `setup_global_logger'
    10: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config.rb:52:in `build'
     9: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config.rb:71:in `parse'
     8: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:33:in `parse'
     7: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:44:in `parse!'
     6: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:96:in `parse_element'
     5: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:96:in `parse_element'
     4: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/v1_parser.rb:131:in `parse_element'
     3: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/literal_parser.rb:75:in `parse_literal'
     2: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/literal_parser.rb:82:in `scan_string'
     1: from /Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/literal_parser.rb:114:in `scan_double_quoted_string'
/Users/me/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/fluentd-1.16.6/lib/fluent/config/basic_parser.rb:92:in `parse_error!': unexpected end of file in a double quoted string at fluent.conf line 6,17 (Fluent::ConfigParseError)
  5:   <record>
  6:     hello "world

     -----------------^
  7:

Additional context

No response

Alainx277 commented 3 weeks ago

This also happens if a line is terminated with a literal backslash:

<filter test>
  @type record_transformer
  <record>
    hello "world \\
test"
  </record>
</filter>
kenhys commented 3 weeks ago

It seems that the limitation of current implementation. As a workaround, use "\n" explicitly.

daipom commented 3 weeks ago

Thanks for your report. It appears this is a bug of Fluentd.

This works.

<filter test.**>
  @type record_transformer
  <record>
    hello "world
"
  </record>
</filter>

However, the following don't work.

<filter test.**>
  @type record_transformer
  <record>
    hello "world

"
  </record>
</filter>
<filter test.**>
  @type record_transformer
  <record>
    hello 'world
'
  </record>
</filter>
<filter test.**>
  @type record_transformer
  <record>
    hello 'world

'
  </record>
</filter>
daipom commented 3 weeks ago

It looks like single-quoted strings don't support multiline. https://docs.fluentd.org/configuration/config-file#multiline-support-for-quoted-string-array-and-hash-values

So, we need to check double quoted strings behavior, as @Alainx277 says.