Can I use this with multiline format?

[blackxored]

Can I use this plugin's output with multiline format for logging Rails requests?

[tagomoris]

Probably. But logs to be parsed should be just a event (record), including newlines.

[johannesfritsch]

I tried but without luck using this regular expression:

https://regex101.com/r/kJ2zG2/4

Maybe someone has an idea, why this is not working?

[bongnv]

I have the same question. I'm using fluentd logging driver for docker. Each line will be sent in an event and the plugin cannot parse multiple lines in multiple events. Sadly, docker doesnot support send multiple lines in 1 event. So can we support it?

[tagomoris]

Joining multi records into a record is very hard problem... There're many problems like these:

• how to specify marks of start/end of records?
• how/when to abort merging multiline if end-mark record is missing?
• how to handle disordering of records?
• how to handle mixed records from multi data sources?

Considering about Docker, I think it's better to create a new plugin to get logs from a container and join it into a records. It's not a feature of this plugin.

[okkez]

@thaohien1812 How about okkez/fluent-plugin-concat ?

[freemanh]

@okkez, the concat plugin solve multiline issue perfectly.

[jwerak]

@okkez how did concat solve it for you? I am trying to concat and parse record, but still strugling with that. I will give an example here in hope that someone will tell me I am doing something obviously wrong...

I have input of:

WARN  [2017-09-14T18:38:34.472Z] class: com.appuri.mapper.endpoint.EventSinkEndpoint mapping_id: 38 feed_id: 3d4b7c99-1cb1-4b1e-acdd-e3ae3f238e38 app_id: acd8b094-fa04-451c-9741-111b3184cb54 org_id: liquidplanner app_name: LiquidPlanner 3 message: Record could not be parsed into event format for validation.
! java.lang.IllegalArgumentException: Invalid format: "2017-05-25 18:02:46.748388" is malformed at " 18:02:46.748388"
! at org.joda.time.format.DateTimeFormatter.parseDateTime(DateTimeFormatter.java:945)
! at com.fasterxml.jackson.datatype.joda.deser.DateTimeDeserializer.deserialize(DateTimeDeserializer.java:90)
! at com.fasterxml.jackson.datatype.joda.deser.DateTimeDeserializer.deserialize(DateTimeDeserializer.java:22)
! at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:490)
! at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:465)
! ... 10 common frames omitted
! Causing: com.fasterxml.jackson.databind.JsonMappingException: Invalid format: "2017-05-25 18:02:46.748388" is malformed at " 18:02:46.748388" (through reference chain: com.appuri.mapper.event.Event["ts"])
! at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:378)
! at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:338)
! at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1510)
! at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:467)
! at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:380)
! at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1123)
! at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:298)
! at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:133)
! at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3779)
! at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2050)
! at com.fasterxml.jackson.databind.ObjectMapper.treeToValue(ObjectMapper.java:2547)
! at com.appuri.mapper.endpoint.EventSinkEndpoint.push(EventSinkEndpoint.java:56)
! at com.appuri.mapper.feed.MappedFeedProcessor.pushRecords(MappedFeedProcessor.java:183)
! at com.appuri.mapper.feed.MappedFeedProcessor.run(MappedFeedProcessor.java:92)
WARN  [2017-09-14T18:38:34.472Z] class: com.appuri.mapper.endpoint.EventSinkEndpoint mapping_id: 38 feed_id: 3d4b7c99-1cb1-4b1e-acdd-e3ae3f238e38 app_id: acd8b094-fa04-451c-9741-111b3184cb54 org_id: liquidplanner app_name: LiquidPlanner 3 message: Record could not be parsed into event format for validation.
! java.lang.IllegalArgumentException: Invalid format: "2017-05-25 18:02:46.556926" is malformed at " 18:02:46.556926"
! at org.joda.time.format.DateTimeFormatter.parseDateTime(DateTimeFormatter.java:945)
! at com.fasterxml.jackson.datatype.joda.deser.DateTimeDeserializer.deserialize(DateTimeDeserializer.java:90)
! at com.fasterxml.jackson.datatype.joda.deser.DateTimeDeserializer.deserialize(DateTimeDeserializer.java:22)
! at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:490)
! at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:465)
! ... 10 common frames omitted

I will run it through concat filter

<filter kube.mapper>
  @type concat
  key message
  multiline_start_regexp /[A-Z]*\s*\[\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z\]\sclass/
</filter>

which gives me records like

2017-09-14 22:46:56.327679460 +0000 kube.mapper: {"message":"WARN  [2017-09-14T18:38:34.472Z] class: com.appuri.mapper.endpoint.EventSinkEndpoint mapping_id: 38 feed_id: 3d4b7c99-1cb1-4b1e-acdd-e3ae3f238e38 app_id: acd8b094-fa04-451c-9741-111b3184cb54 org_id: liquidplanner app_name: LiquidPlanner 3 message: Record could not be parsed into event format for validation.\n! java.lang.IllegalArgumentException: Invalid format: \"2017-05-25 18:02:46.298237\" is malformed at \" 18:02:46.298237\"\n! at org.joda.time.format.DateTimeFormatter.parseDateTime(DateTimeFormatter.java:945)\n! at com.fasterxml.jackson.datatype.joda.deser.DateTimeDeserializer.deserialize(DateTimeDeserializer.java:90)\n! at com.fasterxml.jackson.datatype.joda.deser.DateTimeDeserializer.deserialize(DateTimeDeserializer.java:22)\n! at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:490)\n! at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:465)\n! ... 10 common frames omitted\n! Causing: com.fasterxml.jackson.databind.JsonMappingException: Invalid format: \"2017-05-25 18:02:46.298237\" is malformed at \" 18:02:46.298237\" (through reference chain: com.appuri.mapper.event.Event[\"ts\"])\n! at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:378)\n! at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:338)\n! at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1510)\n! at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:467)\n! at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:380)\n! at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1123)\n! at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:298)\n! at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:133)\n! at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3779)\n! at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2050)\n! at com.fasterxml.jackson.databind.ObjectMapper.treeToValue(ObjectMapper.java:2547)\n! at com.appuri.mapper.endpoint.EventSinkEndpoint.push(EventSinkEndpoint.java:56)\n! at com.appuri.mapper.feed.MappedFeedProcessor.pushRecords(MappedFeedProcessor.java:183)\n! at com.appuri.mapper.feed.MappedFeedProcessor.run(MappedFeedProcessor.java:92)\n\n"}
2017-09-14 22:46:56.327722513 +0000 kube.mapper: {"message":"WARN  [2017-09-14T18:38:34.472Z] class: com.appuri.mapper.endpoint.EventSinkEndpoint mapping_id: 38 feed_id: 3d4b7c99-1cb1-4b1e-acdd-e3ae3f238e38 app_id: acd8b094-fa04-451c-9741-111b3184cb54 org_id: liquidplanner app_name: LiquidPlanner 3 message: Record could not be parsed into event format for validation.\n! java.lang.IllegalArgumentException: Invalid format: \"2017-05-25 18:02:46.748388\" is malformed at \" 18:02:46.748388\"\n! at org.joda.time.format.DateTimeFormatter.parseDateTime(DateTimeFormatter.java:945)\n! at com.fasterxml.jackson.datatype.joda.deser.DateTimeDeserializer.deserialize(DateTimeDeserializer.java:90)\n! at com.fasterxml.jackson.datatype.joda.deser.DateTimeDeserializer.deserialize(DateTimeDeserializer.java:22)\n! at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:490)\n! at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:465)\n! ... 10 common frames omitted\n! Causing: com.fasterxml.jackson.databind.JsonMappingException: Invalid format: \"2017-05-25 18:02:46.748388\" is malformed at \" 18:02:46.748388\" (through reference chain: com.appuri.mapper.event.Event[\"ts\"])\n! at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:378)\n! at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:338)\n! at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1510)\n! at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:467)\n! at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:380)\n! at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1123)\n! at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:298)\n! at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:133)\n! at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3779)\n! at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2050)\n! at com.fasterxml.jackson.databind.ObjectMapper.treeToValue(ObjectMapper.java:2547)\n! at com.appuri.mapper.endpoint.EventSinkEndpoint.push(EventSinkEndpoint.java:56)\n! at com.appuri.mapper.feed.MappedFeedProcessor.pushRecords(MappedFeedProcessor.java:183)\n! at com.appuri.mapper.feed.MappedFeedProcessor.run(MappedFeedProcessor.java:92)"}

but if I add parser filter I will always be cut with \n

<filter kube.mapper>
  @type parser
  key_name message
  format /(?<severity>[A-Z]*)\s*\[(?<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z)\]\sclass:\s*(?<class>[^\ ]*)\smapping_id:\s*(?<other>.*)/
</filter>

and results in

2017-09-14 18:38:34.472000000 +0000 kube.mapper: {"severity":"WARN","class":"com.appuri.mapper.endpoint.EventSinkEndpoint","other":"38 feed_id: 3d4b7c99-1cb1-4b1e-acdd-e3ae3f238e38 app_id: acd8b094-fa04-451c-9741-111b3184cb54 org_id: liquidplanner app_name: LiquidPlanner 3 message: Record could not be parsed into event format for validation."}
2017-09-14 18:38:34.472000000 +0000 kube.mapper: {"severity":"WARN","class":"com.appuri.mapper.endpoint.EventSinkEndpoint","other":"38 feed_id: 3d4b7c99-1cb1-4b1e-acdd-e3ae3f238e38 app_id: acd8b094-fa04-451c-9741-111b3184cb54 org_id: liquidplanner app_name: LiquidPlanner 3 message: Record could not be parsed into event format for validation."}

I can change separator which "solves" the issue of not capturing whole log record, but looks hidious in kibana...

<filter kube.mapper>
  @type concat
  key message
  multiline_start_regexp /[A-Z]*\s*\[\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z\]\sclass/
  separator "NEWLINE"
</filter>

Is there a way to not ignore \n in this parser?

[okkez]

@jwerak Could you describeexpected result? full configuration? Which version of Fluentd do you use? v0.12.x or v0.14.x?

If you use Fluentd v0.14.x, you can use built-in filter_parser plugin which supports multiline option.

[jwerak]

I use

# td-agent --version
td-agent 0.14.21

expected result would be keeping whole log, not just the first line. Concat joins them, but parser filter throws away everything after \n even if in single record.

My full config for this test is

<match fluent.**>
  @type null
</match>

<source>
  @type tail
  path /tmp/mapper*
  tag kube.mapper
  format none
  read_from_head true
</source>

<filter kube.mapper>
  @type concat
  key message
  multiline_start_regexp /[A-Z]*\s*\[\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z\]\sclass/
</filter>

<filter kube.mapper>
  @type parser
  key_name message
  format /(?<severity>[A-Z]*)\s*\[(?<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z)\]\sclass:\s*(?<class>[^\ ]*)\smapping_id:\s*(?<other>.*)/
</filter>

<match **>
   @type stdout
</match>

I am also trying to setup <parse> section, but I am still finding myself in this new format, this is what I have:

<match fluent.**>
  @type null
</match>

<source>
  @type tail
  path /tmp/mapper*
  tag kube.mapper
  format none
  read_from_head true
</source>

<filter kube.mapper>
  @type parser
  key_name message
  format multiline
  <parse>
    @type multiline
    format_firstline /[A-Z]*\s*\[\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z\]\sclass/
    format1 /(?<severity>[A-Z]*)\s*\[(?<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z)\]\sclass:\s*(?<class>[^\ ]*)\smapping_id:\s*(?<other>.*)/
    time_format %d/%b/%Y:%H:%M:%S %z
  </parse>
</filter>

<match **>
   @type stdout
</match>

but it throws

2017-09-15 10:10:36 +0000 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2017-09-15 10:10:36 +0000 [error]: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Invalid regexp '': No named captures"

[okkez]

How about following configuration? I don't change your regular expression. I use regexp parser's multiline option

<source>
  @type tail
  path /tmp/mapper*
  tag kube.mapper
  format none
  read_from_head true
</source>

<filter kube.mapper>
  @type concat
  key message
  multiline_start_regexp /[A-Z]*\s*\[\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z\]\sclass/
  continuous_line_regexp /^!.+/
</filter>

<filter kube.mapper>
  @type parser
  key_name message
  <parse>
    @type regexp
    multiline true
    expression /(?<severity>[A-Z]*)\s*\[(?<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z)\]\sclass:\s*(?<class>[^\ ]*)\smapping_id:\s*(?<other>.*)/
  </parse>
</filter>

<match kube.mapper>
   @type stdout
</match>

I got following result.

2017-09-15 03:38:34.472000000 +0900 kube.mapper: {"severity":"WARN","class":"com.appuri.mapper.endpoint.EventSinkEndpoint","other":"38 feed_id: 3d4b7c99-1cb1-4b1e-acdd-e3ae3f238e38 app_id: acd8b094-fa04-451c-9741-111b3184cb54 org_id: liquidplanner app_name: LiquidPlanner 3 message: Record could not be parsed into event format for validation.\n! java.lang.IllegalArgumentException: Invalid format: \"2017-05-25 18:02:46.748388\" is malformed at \" 18:02:46.748388\"\n! at org.joda.time.format.DateTimeFormatter.parseDateTime(DateTimeFormatter.java:945)\n! at com.fasterxml.jackson.datatype.joda.deser.DateTimeDeserializer.deserialize(DateTimeDeserializer.java:90)\n! at com.fasterxml.jackson.datatype.joda.deser.DateTimeDeserializer.deserialize(DateTimeDeserializer.java:22)\n! at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:490)\n! at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:465)\n! ... 10 common frames omitted\n! Causing: com.fasterxml.jackson.databind.JsonMappingException: Invalid format: \"2017-05-25 18:02:46.748388\" is malformed at \" 18:02:46.748388\" (through reference chain: com.appuri.mapper.event.Event[\"ts\"])\n! at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:378)\n! at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:338)\n! at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1510)\n! at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:467)\n! at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:380)\n! at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1123)\n! at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:298)\n! at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:133)\n! at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3779)\n! at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2050)\n! at com.fasterxml.jackson.databind.ObjectMapper.treeToValue(ObjectMapper.java:2547)\n! at com.appuri.mapper.endpoint.EventSinkEndpoint.push(EventSinkEndpoint.java:56)\n! at com.appuri.mapper.feed.MappedFeedProcessor.pushRecords(MappedFeedProcessor.java:183)\n! at com.appuri.mapper.feed.MappedFeedProcessor.run(MappedFeedProcessor.java:92)"}
2017-09-15 03:38:34.472000000 +0900 kube.mapper: {"severity":"WARN","class":"com.appuri.mapper.endpoint.EventSinkEndpoint","other":"38 feed_id: 3d4b7c99-1cb1-4b1e-acdd-e3ae3f238e38 app_id: acd8b094-fa04-451c-9741-111b3184cb54 org_id: liquidplanner app_name: LiquidPlanner 3 message: Record could not be parsed into event format for validation.\n! java.lang.IllegalArgumentException: Invalid format: \"2017-05-25 18:02:46.556926\" is malformed at \" 18:02:46.556926\"\n! at org.joda.time.format.DateTimeFormatter.parseDateTime(DateTimeFormatter.java:945)\n! at com.fasterxml.jackson.datatype.joda.deser.DateTimeDeserializer.deserialize(DateTimeDeserializer.java:90)\n! at com.fasterxml.jackson.datatype.joda.deser.DateTimeDeserializer.deserialize(DateTimeDeserializer.java:22)\n! at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:490)\n! at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:465)\n! ... 10 common frames omitted"}

See also

• https://docs.fluentd.org/v0.14/articles/filter_parser
• https://docs.fluentd.org/v0.14/articles/parser_regexp

[jwerak]

Thanks for your time @okkez

I am running with same configuration and I get following error:

root@fa24ade8ec36:/# td-agent --version
td-agent 0.14.21
root@fa24ade8ec36:/# td-agent
2017-09-25 10:35:15 +0000 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2017-09-25 10:35:15 +0000 [error]: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="'format' parameter is required"

It works if I add format none (see below) but it doesn't parametrize the log.

<filter kube.mapper>
  @type parser
  key_name message
  format none
  <parse>
    @type regexp
    multiline true
    expression /(?<severity>[A-Z]*)\s*\[(?<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z)\]\sclass:\s*(?<class>[^\ ]*)\smapping_id:\s*(?<other>.*)/
  </parse>
</filter>

It looks like I don't have proper version, but I install it via https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent3.sh script on Ubuntu 16.04 (in Docker container)

Any idea what could be wrong with my installation?

[okkez]

I tried using configuration in https://github.com/tagomoris/fluent-plugin-parser/issues/30#issuecomment-331765844 on my ubuntu container. It works well.

format parameter is required by old version of filter_parser. So you are using old version of filter_parser, I think.

Could you check your environment and your installation? Could you show me full log on boot like following?

# td-agent -c /etc/td-agent/td-agent.conf
2017-09-26 01:18:00 +0000 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2017-09-26 01:18:00 +0000 [info]: starting fluentd-0.14.16 pid=3872
2017-09-26 01:18:00 +0000 [info]: spawn command to main:  cmdline=["/opt/td-agent/embedded/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/sbin/td-agent", "-c", "/etc/td-agent/td-agent.conf", "--under-supervisor"]
2017-09-26 01:18:01 +0000 [info]: gem 'fluent-plugin-concat' version '2.1.0'
2017-09-26 01:18:01 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '1.9.5'
2017-09-26 01:18:01 +0000 [info]: gem 'fluent-plugin-kafka' version '0.5.5'
2017-09-26 01:18:01 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.5'
2017-09-26 01:18:01 +0000 [info]: gem 'fluent-plugin-s3' version '1.0.0.rc3'
2017-09-26 01:18:01 +0000 [info]: gem 'fluent-plugin-td' version '1.0.0.rc1'
2017-09-26 01:18:01 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2017-09-26 01:18:01 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.1.1'
2017-09-26 01:18:01 +0000 [info]: gem 'fluentd' version '0.14.16'
2017-09-26 01:18:01 +0000 [info]: adding filter pattern="kube.mapper" type="concat"
2017-09-26 01:18:01 +0000 [info]: adding filter pattern="kube.mapper" type="parser"
2017-09-26 01:18:01 +0000 [info]: adding match pattern="kube.mapper" type="stdout"
2017-09-26 01:18:01 +0000 [info]: adding source type="tail"
2017-09-26 01:18:01 +0000 [warn]: #0 'pos_file PATH' parameter is not set to a 'tail' source.
2017-09-26 01:18:01 +0000 [warn]: #0 this parameter is highly recommended to save the position to resume tailing.
2017-09-26 01:18:01 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type tail
    path "/tmp/mapper*"
    tag "kube.mapper"
    format none
    read_from_head true
    <parse>
      @type none
    </parse>
  </source>
  <filter kube.mapper>
    @type concat
    key "message"
    multiline_start_regexp "/[A-Z]*\\s*\\[\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\.\\d{3}Z\\]\\sclass/"
    continuous_line_regexp "/^!.+/"
  </filter>
  <filter kube.mapper>
    @type parser
    key_name "message"
    <parse>
      @type "regexp"
      multiline true
      expression "/(?<severity>[A-Z]*)\\s*\\[(?<time>\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\.\\d{3}Z)\\]\\sclass:\\s*(?<class>[^\\ ]*)\\smapping_id:\\s*(?<other>.*)/"
    </parse>
  </filter>
  <match kube.mapper>
    @type stdout
  </match>
</ROOT>

[jwerak]

ok, that is the problem, I have parser for fluentd 0.12...

root@f82654d64f1a:/# td-agent -c /etc/td-agent/td-agent.conf
2017-09-26 08:31:21 +0000 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2017-09-26 08:31:21 +0000 [warn]: 'pos_file PATH' parameter is not set to a 'tail' source.
2017-09-26 08:31:21 +0000 [warn]: this parameter is highly recommended to save the position to resume tailing.
2017-09-26 08:31:21 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type tail
    path "/tmp/mapper*"
    tag "kube.mapper"
    format none
    read_from_head true
    <parse>
      @type none
    </parse>
  </source>
  <filter kube.mapper>
    @type concat
    key "message"
    multiline_start_regexp "/[A-Z]*\\s*\\[\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\.\\d{3}Z\\]\\sclass/"
    continuous_line_regexp "/^!.+/"
  </filter>
  <filter kube.mapper>
    @type parser
    format none
    key_name "message"
    <parse>
      @type regexp
      multiline true
      expression /(?<severity>[A-Z]*)\s*\[(?<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z)\]\sclass:\s*(?<class>[^\ ]*)\smapping_id:\s*(?<other>.*)/
    </parse>
    <format>
      @type none
    </format>
  </filter>
  <match kube.mapper>
    @type stdout
  </match>
</ROOT>
2017-09-26 08:31:21 +0000 [info]: starting fluentd-0.14.21 pid=67
2017-09-26 08:31:21 +0000 [info]: spawn command to main:  cmdline=["/opt/td-agent/embedded/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/sbin/td-agent", "-c", "/etc/td-agent/td-agent.conf", "--under-supervisor"]
2017-09-26 08:31:22 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2017-09-26 08:31:22 +0000 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2017-09-26 08:31:22 +0000 [info]: gem 'fluent-plugin-concat' version '2.1.0'
2017-09-26 08:31:22 +0000 [info]: gem 'fluent-plugin-flatten-hash' version '0.5.0'
2017-09-26 08:31:22 +0000 [info]: gem 'fluent-plugin-kafka' version '0.5.5'
2017-09-26 08:31:22 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '0.29.0'
2017-09-26 08:31:22 +0000 [info]: gem 'fluent-plugin-mongo' version '0.8.0'
2017-09-26 08:31:22 +0000 [info]: gem 'fluent-plugin-parser' version '0.6.1'
2017-09-26 08:31:22 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.5'
2017-09-26 08:31:22 +0000 [info]: gem 'fluent-plugin-s3' version '0.8.2'
2017-09-26 08:31:22 +0000 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2017-09-26 08:31:22 +0000 [info]: gem 'fluent-plugin-systemd' version '0.3.0'
2017-09-26 08:31:22 +0000 [info]: gem 'fluent-plugin-td' version '0.10.29'
2017-09-26 08:31:22 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2017-09-26 08:31:22 +0000 [info]: gem 'fluent-plugin-webhdfs' version '0.4.2'
2017-09-26 08:31:22 +0000 [info]: gem 'fluentd' version '0.14.21'
2017-09-26 08:31:22 +0000 [info]: gem 'fluentd' version '0.12.35'
2017-09-26 08:31:22 +0000 [info]: adding filter pattern="kube.mapper" type="concat"
2017-09-26 08:31:22 +0000 [info]: adding filter pattern="kube.mapper" type="parser"
2017-09-26 08:31:22 +0000 [info]: adding match pattern="kube.mapper" type="stdout"
2017-09-26 08:31:22 +0000 [info]: adding source type="tail"
2017-09-26 08:31:22 +0000 [warn]: #0 'pos_file PATH' parameter is not set to a 'tail' source.
2017-09-26 08:31:22 +0000 [warn]: #0 this parameter is highly recommended to save the position to resume tailing.
2017-09-26 08:31:22 +0000 [warn]: section <parse> is not used in <filter kube.mapper> of none plugin
2017-09-26 08:31:22 +0000 [warn]: section <parse> is not used in <filter kube.mapper> of none plugin
2017-09-26 08:31:22 +0000 [warn]: section <parse> is not used in <filter kube.mapper> of none plugin
2017-09-26 08:31:22 +0000 [warn]: section <format> is not used in <filter kube.mapper> of none plugin
2017-09-26 08:31:22 +0000 [info]: #0 starting fluentd worker pid=71 ppid=67 worker=0
2017-09-26 08:31:22 +0000 [info]: #0 following tail of /tmp/mapper-test
2017-09-26 08:31:22 +0000 [info]: #0 disable filter chain optimization because [Fluent::Plugin::ConcatFilter, Fluent::ParserFilter] uses `#filter_stream` method.

My installation should be standard, I am using package provided by treasuredata: https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent3.sh

What installation process is preferable or which one do you use @okkez ?

And again thanks a lot for your help, I can see the light on the end of tunnel :)

[okkez]

Have you install td-agent2? If yes, you must uninstall td-agent2 before installing td-agent3, I think.

Root cause of error="'format' parameter is required is gem 'fluent-plugin-parser' version '0.6.1' and gem 'fluentd' version '0.12.35'. Multiple versions of Fluentd gem is trouble maker... In this case, I think that filter_parser in fluent-plugin-parser overwrites Fluentd v0.14.21 built-in filter_parser.

You can avoid above case using Gemfile with --gemfile option. See https://docs.fluentd.org/v0.14/articles/plugin-management#ldquondashgemfilerdquo-option

Another way, uninstall unused plugins.

• fluent-mixin-config-placeholders
• fluent-mixin-plaintextformatter
• fluent-plugin-parser
• fluent-plugin-scribe
• fluentd v0.12.35
• and, etc.

What installation process is preferable or which one do you use @okkez ?

I use clean Ubuntu container with install-ubuntu-xenial-td-agent3.sh to investigate your problem. https://docs.fluentd.org/v0.14/categories/installation

If you want to use Fluentd with Docker container, you can use Fluentd official docker image https://hub.docker.com/r/fluent/fluentd/ .

I can say that clean installation is very important.