Decoding nested JSON objects

[luckyb56]

When using Packetbeat to send messages directly to Graylog2, the nested JSON objects won't be decoded and would be seen as '[object Object],[object Object]' under search UI. Example input JSON message:

{
  "_index" : "graylog_5",
  "_type" : "message",
  "_id" : "572b9193-16df-11e6-8a3b-000c2942c251",
  "_version" : 1,
  "found" : true,
  "_source" : {
    "packetbeat_bytes_in" : 32,
    "packetbeat_method" : "QUERY",
    "packetbeat_type" : "dns",
    "packetbeat_responsetime" : 140,
    "packetbeat_query" : "class IN, type A, conn.skype.com",
    "gl2_remote_ip" : "172.16.220.1",
    "packetbeat_dns_question_name" : "conn.skype.com",
    "gl2_remote_port" : 65532,
    "packetbeat_dns_additionals_count" : 0,
    "packetbeat_dns_answers_count" : 2,
    "source" : "abs-MacBook-Pro.local",
    "type" : "dns",
    "gl2_source_input" : "572a39d0cdf3830902a406df",
    "packetbeat_dns_response_code" : "NOERROR",
    "packetbeat_direction" : "out",
    "packetbeat_client_ip" : "192.168.0.3",
    "packetbeat_dns_flags_recursion_allowed" : true,
    "packetbeat_dns_flags_truncated_response" : false,
    "packetbeat_dns_question_class" : "IN",
    "gl2_source_node" : "b6d4add1-2cfc-4fd1-b18d-0ad0478e00a8",
    "packetbeat_dns_flags_authoritative" : false,
    "packetbeat_status" : "OK",
    "packetbeat_client_port" : 60426,
    "timestamp" : "2016-05-10 18:45:16.558",
    "packetbeat_ip" : "192.168.0.1",
    "packetbeat_dns_op_code" : "QUERY",
    "packetbeat_bytes_out" : 83,
    "packetbeat_dns_flags_recursion_desired" : true,
    "packetbeat_transport" : "udp",
    "packetbeat_dns_authorities_count" : 0,
    "packetbeat_resource" : "conn.skype.com",
    "streams" : [ "572ae5c9cdf3830902a4bb7f" ],
    "packetbeat_dns_answers" : [ {
      "class" : "IN",
      "data" : "conn.skype.akadns.net",
      "name" : "conn.skype.com",
      "ttl" : 464,
      "type" : "CNAME"
    }, {
      "class" : "IN",
      "data" : "91.190.216.81",
      "name" : "conn.skype.akadns.net",
      "ttl" : 300,
      "type" : "A"
    } ],
    "message" : "-",
    "packetbeat_dns_question_type" : "A",
    "packetbeat_count" : 1,
    "name" : "MacBook-Pro.local",
    "packetbeat_dns_id" : 62527,
    "facility" : "packetbeat",
    "packetbeat_port" : 53
  }
}

packetbeat_dns_answers structure won't be decoded in this example.

[pramodanarase]

@luckyb56 what you want ? decode in single message or multiple message? if decode into multiple message on nested field(in your case packetbeat_dns_answers) i think we need to implement feature like logstash split filter using "MultiMessageCodec".

[luckyb56]

@pramodanarase A single message would be great as it will keep the context and do basic level of stats.

[rompic]

+1

[kroepke]

This has been partially addressed in https://github.com/Graylog2/graylog2-server/pull/3106 and #29