repeatedly / fluent-plugin-multi-format-parser

Multi format parser for Fluentd
108 stars 16 forks source link

Can multi-format parse special json format? #15

Closed ZhangSIming-blyq closed 4 years ago

ZhangSIming-blyq commented 4 years ago

Traefik log is like:

{"log":"{\"BackendAddr\":\"172.62.187.135:16686\",\"BackendName\":\"tracing-test.shannonai.com\",\"BackendURL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"172.62.187.135:16686\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"ClientAddr\":\"10.20.3.252:14432\",\"ClientHost\":\"10.20.11.156\",\"ClientPort\":\"14432\",\"ClientUsername\":\"-\",\"DownstreamContentSize\":530,\"DownstreamStatus\":200,\"DownstreamStatusLine\":\"200 OK\",\"Duration\":118040406,\"FrontendName\":\"tracing-test.shannonai.com\",\"OriginContentSize\":530,\"OriginDuration\":117940134,\"OriginStatus\":200,\"OriginStatusLine\":\"200 OK\",\"Overhead\":100272,\"RequestAddr\":\"tracing-test.shannonai.com\",\"RequestContentSize\":0,\"RequestCount\":174683,\"RequestHost\":\"tracing-test.shannonai.com\",\"RequestLine\":\"GET /api/traces?end=1576485011758000\\u0026limit=20\\u0026lookback=1h\\u0026maxDuration\\u0026minDuration\\u0026service=osprey\\u0026start=1576481411758000 HTTP/1.1\",\"RequestMethod\":\"GET\",\"RequestPath\":\"/api/traces?end=1576485011758000\\u0026limit=20\\u0026lookback=1h\\u0026maxDuration\\u0026minDuration\\u0026service=osprey\\u0026start=1576481411758000\",\"RequestPort\":\"-\",\"RequestProtocol\":\"HTTP/1.1\",\"RetryAttempts\":0,\"StartLocal\":\"2019-12-16T16:30:11.875918571+08:00\",\"level\":\"info\",\"msg\":\"\",\"time\":\"2019-12-16T16:30:11+08:00\"}\n","stream":"stdout","time":"2019-12-16T08:30:11.994250321Z"}

Notice that in "log"'s value, it have a subfield called "BackendURL". And it's value is also a json dictionary format.

I use below config, it cannot return expected result, but ignore the whole log

   <filter kubernetes.**>
      @id filter_parser
      @type parser
      key_name log
      reserve_data true
      remove_key_name_field true
      <parse>
        @type multi_format
        <pattern>
          format json
        </pattern>
        <pattern>
          format none
        </pattern>
      </parse>
    </filter>

But when I use another configuration(below), it can return a field called "message", which contain all the value of "log" "json" and "none"‘s order is different than before

   <filter kubernetes.**>
      @id filter_parser
      @type parser
      key_name log
      reserve_data true
      remove_key_name_field true
      <parse>
        @type multi_format
        <pattern>
          format none
        </pattern>
        <pattern>
          format json
        </pattern>
      </parse>
    </filter>

result like:

"message": "{\"BackendAddr\":\"172.62.187.135:16686\",\"BackendName\":\"tracing-test.shannonai.com\",\"BackendURL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"172.62.187.135:16686\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"ClientAddr\":\"10.20.3.252:14432\",\"ClientHost\":\"10.20.11.156\",\"ClientPort\":\"14432\",\"ClientUsername\":\"-\",\"DownstreamContentSize\":530,\"DownstreamStatus\":200,\"DownstreamStatusLine\":\"200 OK\",\"Duration\":118040406,\"FrontendName\":\"tracing-test.shannonai.com\",\"OriginContentSize\":530,\"OriginDuration\":117940134,\"OriginStatus\":200,\"OriginStatusLine\":\"200 OK\",\"Overhead\":100272,\"RequestAddr\":\"tracing-test.shannonai.com\",\"RequestContentSize\":0,\"RequestCount\":174683,\"RequestHost\":\"tracing-test.shannonai.com\",\"RequestLine\":\"GET /api/traces?end=1576485011758000\\u0026limit=20\\u0026lookback=1h\\u0026maxDuration\\u0026minDuration\\u0026service=osprey\\u0026start=1576481411758000 HTTP/1.1\",\"RequestMethod\":\"GET\",\"RequestPath\":\"/api/traces?end=1576485011758000\\u0026limit=20\\u0026lookback=1h\\u0026maxDuration\\u0026minDuration\\u0026service=osprey\\u0026start=1576481411758000\",\"RequestPort\":\"-\",\"RequestProtocol\":\"HTTP/1.1\",\"RetryAttempts\":0,\"StartLocal\":\"2019-12-16T16:30:11.875918571+08:00\",\"level\":\"info\",\"msg\":\"\",\"time\":\"2019-12-16T16:30:11+08:00\"}\n"

My QUESTION : Can json parse json in my certain log? Or what should I do to config a right parse operation?

devstein commented 4 years ago

@ZhangSIming-blyq What was the solution to this?