Introduce parse section to parse CloudWatchLogs' log with arbitrary format

rsingh2411 commented 4 years ago

Problem

I am unable to fetch non json logs from cloud watch, as per the fix stated in https://github.com/fluent-plugins-nursery/fluent-plugin-cloudwatch-logs/issues/156, used the latest gem, during event read the exception is thrown.

Following are error logs [error]: #0 fluent/log.rb:362:error: Invalid JSON encountered while parsing event.message 2019-12-20 01:42:53 +0000 [warn]: #0 fluent/log.rb:342:warn: dump an error event: error_class=JSON::ParserError error="767: unexpected token at 'I1219 02:38:18.287898 1 flags.go:33] FLAG: --address=\"0.0.0.0\"'" location="/usr/local/lib/ruby/2.6.0/json/common.rb:156:in `parse'" tag="cloudwatch.in" time=1576723098 record={:message=>"I1219 02:38:18.287898 1 flags.go:33] FLAG: --address=\"0.0.0.0\""}

...

Steps to replicate

Provide example config and message

 <source>
     @type cloudwatch_logs
     tag cloudwatch.in
     log_group_name /aws/eks/rohit-rds74/cluster
     log_stream_name kube-scheduler-579eb45e4a3af719c368f47497c0b033
     state_file /tmp/fluent-plugin-cloudwatch-example.state
     #use_log_stream_name_prefix true
     #endpoint http://localhost:5000/
     json_handler json
     use_aws_timestamp true
</source>
<match cloudwatch.in>
     @type stdout
</match>

Expected Behavior or What you need to ask

Is it expected, non json logs wont be supported and in case they are present, it will handle exception without plugin to crash and fail? Please let us know ...

Using Fluentd and CloudWatchLogs plugin versions

fluentd --version fluentd 1.7.4

Running on docker with latest gem for aws cloudwatch installed

Gem list addressable (2.7.0) aws-eventstream (1.0.3) aws-partitions (1.255.0) aws-sdk-cloudwatchlogs (1.27.0) aws-sdk-core (3.86.0) aws-sigv4 (1.1.0) bigdecimal (default: 1.4.1) bundler (default: 1.17.2, 1.16.2) cmath (default: 1.0.0) concurrent-ruby (1.1.5) cool.io (1.5.4) csv (default: 3.0.9) date (default: 2.0.0) dbm (default: 1.0.0) dig_rb (1.0.1) domain_name (0.5.20190701) e2mmap (default: 0.1.0) elasticsearch (7.4.0) elasticsearch-api (7.4.0) elasticsearch-transport (7.4.0) elasticsearch-xpack (7.4.0) etc (default: 1.0.1) excon (0.71.1) faraday (0.17.1) fcntl (default: 1.0.0) ffi (1.11.3) ffi-compiler (1.0.1) fiddle (default: 1.0.0) fileutils (default: 1.1.0) fluent-config-regexp-type (1.0.0) fluent-plugin-cloudwatch-logs (0.7.6) fluent-plugin-concat (2.4.0) fluent-plugin-detect-exceptions (0.0.13) fluent-plugin-elasticsearch (3.7.1) fluent-plugin-grok-parser (2.6.1) fluent-plugin-json-in-json-2 (1.0.2) fluent-plugin-kubernetes_metadata_filter (2.3.0) fluent-plugin-multi-format-parser (1.0.0) fluent-plugin-prometheus (1.6.1) fluent-plugin-record-modifier (2.0.1) fluent-plugin-rewrite-tag-filter (2.2.0) fluent-plugin-systemd (1.0.2) fluentd (1.7.4) forwardable (default: 1.2.0) gdbm (default: 2.0.0) http (4.2.0) http-accept (1.7.0) http-cookie (1.0.3) http-form_data (2.1.1) http-parser (1.2.1) http_parser.rb (0.6.0) io-console (default: 0.4.7) ipaddr (default: 1.2.2) irb (default: 1.0.0) jmespath (1.4.0) json (default: 2.1.0) kubeclient (4.5.0) logger (default: 1.3.0) lru_redux (1.1.0) matrix (default: 0.1.0) mime-types (3.3) mime-types-data (3.2019.1009) msgpack (1.3.1) multi_json (1.14.1) multipart-post (2.1.1) mutex_m (default: 0.1.0) netrc (0.11.0) oj (3.8.1) openssl (default: 2.1.2) ostruct (default: 0.1.0) prime (default: 0.1.0) prometheus-client (0.9.0) psych (default: 3.1.0) public_suffix (4.0.1) quantile (0.2.1) rake (13.0.1) rdoc (default: 6.1.2) recursive-open-struct (1.1.0) rest-client (2.1.0) rexml (default: 3.1.9) rss (default: 0.2.7) scanf (default: 1.0.0) sdbm (default: 1.0.0) serverengine (2.2.0) shell (default: 0.7) sigdump (0.2.4) stringio (default: 0.0.2) strptime (0.2.3) strscan (default: 1.0.0) sync (default: 0.5.0) systemd-journal (1.3.3) thwait (default: 0.1.0) tracer (default: 0.1.0) tzinfo (2.0.0) tzinfo-data (1.2019.3) unf (0.1.4) unf_ext (0.0.7.6) webrick (default: 1.4.2) yajl-ruby (1.4.1) zlib (default: 1.0.0)

iShiBin commented 4 years ago

It will still emit an error event according to the fix: router.emit_error_event(@tag, time, { message: event.message }, error)

I'd like to enhance it by introduce the parser section so that you can define a regex to format the cloudwatch message.

cosmo0920 commented 4 years ago

in_cloudwatch_logs plugin already implements <parse> section to parse arbitrary format. But it is not described in README. I'll write down it.

fluent-plugins-nursery / fluent-plugin-cloudwatch-logs