search

logstash-plugins / logstash-codec-json

Apache License 2.0
23 stars 29 forks source link

Logstash Codec JSON Plugin Target Option Not Working #40

Closed chelseaRickman closed 2 years ago

chelseaRickman commented 2 years ago

Logstash information:

Please include the following information:

  1. Logstash version (e.g. bin/logstash --version)

logstash 8.0.0

  1. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker)

Docker image, docker.elastic.co/logstash/logstash-oss:8.0.0

  1. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes): 

    docker run -it -p 8080:8080 -v {path to logstash.conf}:/usr/share/logstash/pipeline/logstash.conf docker.elastic.co/logstash/logstash-oss:8.0.0

  2. How was the Logstash Plugin installed

JVM (e.g. java -version): 17.0.2

If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:

  1. JVM version (java -version)
  2. JVM installation source (e.g. from the Operating System's package manager, from source, etc).
  3. Value of the JAVA_HOME environment variable if set.

OS version (uname -a if on a Unix-like system):

macOS Catalina Version 10.15.7

Description of the problem including expected versus actual behavior:

The target configuration option for the JSON codec plugin does not store the parsed JSON data at the specified target field (expected behavior). The parsed data is stored at the root of the event (actual behavior).

Steps to reproduce:

Created a logstash.conf file using the provided example provided from elastic: https://www.elastic.co/guide/en/logstash/8.0/plugins-codecs-json.html#plugins-codecs-json-target

The logstash.conf file consisted of:

input {
    http {
        codec => json {
            target => "[document]"
        }
    }
}

output {
    stdout {
        codec => "rubydebug"
    }
}

Ran the logstash Docker image, mounting this logstash.conf into the container:

docker run -it -p 8080:8080 -v {path to my logstash.conf file}:/usr/share/logstash/pipeline/logstash.conf docker.elastic.co/logstash/logstash-oss:8.0.0

Using Postman, sent a POST request to 0.0.0.0/8080 with request body of: { "test": "testing", "test2": "second test" }

The output to stdout within the container:

{
          "test" => "testing",
          "host" => {
        "ip" => "ip"
    },
      "@version" => "1",
         "event" => {
        "original" => "{\n    \"test\": \"testing\",\n    \"test2\": \"second test\"\n}"
    },
          "http" => {
        "version" => "HTTP/1.1",
        "request" => {
                 "body" => {
                "bytes" => "69"
            },
            "mime_type" => "application/json"
        },
         "method" => "POST"
    },
    "@timestamp" => 2022-02-15T22:55:14.575445Z,
         "test2" => "second test",
           "url" => {
          "port" => 8080,
          "path" => "/",
        "domain" => "0.0.0.0"
    },
    "user_agent" => {
        "original" => "PostmanRuntime/7.29.0"
    }
}

I expected the body of my request to be placed in a "document" field, but there is no "document" field present and the body of my request was placed at the root of the event.

Please include a minimal but complete recreation of the problem, including (e.g.) pipeline definition(s), settings, locale, etc. The easier you make for us to reproduce it, the more likely that somebody will take the time to look at it.

  1. Used default /config/pipelines.yml
  2. Used the above logstash.conf as the only logstash pipeline config within /usr/share/logstash/pipeline

Provide logs (if relevant):

docker run -it -p 8080:8080  -v ${PWD}/pipeline/test.conf:/usr/share/logstash/pipeline/logstash.conf --name logstash-body-test docker.elastic.co/logstash/logstash-oss:8.0.0

Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2022-02-15T22:55:03,705][INFO ][logstash.runner          ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
[2022-02-15T22:55:03,722][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.0.0", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[2022-02-15T22:55:03,724][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/]
[2022-02-15T22:55:03,779][INFO ][logstash.settings        ] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[2022-02-15T22:55:03,798][INFO ][logstash.settings        ] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[2022-02-15T22:55:04,297][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"ddf03ecc-2875-4829-98e1-e4fbe2928b46", :path=>"/usr/share/logstash/data/uuid"}
[2022-02-15T22:55:05,379][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-02-15T22:55:06,434][INFO ][org.reflections.Reflections] Reflections took 82 ms to scan 1 urls, producing 120 keys and 417 values 
[2022-02-15T22:55:07,577][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2022-02-15T22:55:07,759][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>6, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>750, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x49871e28 run>"}
[2022-02-15T22:55:08,733][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.97}
[2022-02-15T22:55:08,833][INFO ][logstash.codecs.json     ][main] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2022-02-15T22:55:08,986][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-02-15T22:55:09,004][INFO ][logstash.inputs.http     ][main][43dc907c5a4f362379eb81441028a35b81941f8ea27e7403ded38d8dc77f777f] Starting http input listener {:address=>"0.0.0.0:8080", :ssl=>"false"}
[2022-02-15T22:55:09,128][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2022-02-15T22:55:14,442][INFO ][logstash.codecs.json     ][main][43dc907c5a4f362379eb81441028a35b81941f8ea27e7403ded38d8dc77f777f] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
{
          "test" => "testing",
          "host" => {
        "ip" => "ip"
    },
      "@version" => "1",
         "event" => {
        "original" => "{\n    \"test\": \"testing\",\n    \"test2\": \"second test\"\n}"
    },
          "http" => {
        "version" => "HTTP/1.1",
        "request" => {
                 "body" => {
                "bytes" => "69"
            },
            "mime_type" => "application/json"
        },
         "method" => "POST"
    },
    "@timestamp" => 2022-02-15T22:55:14.575445Z,
         "test2" => "second test",
           "url" => {
          "port" => 8080,
          "path" => "/",
        "domain" => "0.0.0.0"
    },
    "user_agent" => {
        "original" => "PostmanRuntime/7.29.0"
    }
}
yaauie commented 2 years ago

The issue here is that the HTTP input automatically creates a content-type-to-codec mapping for application/json to a self-instantiated codec that does not include your target directive (see: HTTP Input: additional_codecs), and will prefer the codecs found in the content type mapping to the "fallback" codec for the plugin itself.

To avoid this, you will need to provide an empty mapping to the HTTP input:

input {
  http {
   additional_codecs => {}
  }
}