magnusbaeck / logstash-filter-verifier

Apache License 2.0
191 stars 27 forks source link

type fields not added by input #166

Closed jgough closed 2 years ago

jgough commented 2 years ago

In Logstash input configs the type parameter can be used to add a type field to every event.

With the following input config:

input {
    stdin {
        id => "input"
        type => "test_event"
    }
}

The field type should be set to test_event

input_plugin: "input"
ignore:
  - "@timestamp"
testcases:
  - input:
      - Test
    expected:
      - message: Test
        type: test_event

However LFV2 beta 2 does not seem to set the type from this config and it gets ignored.

--- /tmp/067579569/test.yml/1/expected  2021-11-26 13:31:28.944385569 +0000
+++ /tmp/067579569/test.yml/1/actual    2021-11-26 13:31:28.944385569 +0000
@@ -1,4 +1,3 @@
 {
-  "message": "Test",
-  "type": "test_event"
+  "message": "Test"
 }

Full repro Dockerfile:

# syntax=docker/dockerfile:1.3-labs
FROM docker.elastic.co/logstash/logstash:7.10.2

ENV LOGSTASH_FILTER_VERIFIER_VERSION v2.0.0-beta.2

USER root
RUN yum clean expire-cache && yum update -y && yum install curl && yum clean all
ADD https://github.com/magnusbaeck/logstash-filter-verifier/releases/download/${LOGSTASH_FILTER_VERIFIER_VERSION}/logstash-filter-verifier_${LOGSTASH_FILTER_VERIFIER_VERSION}_linux_386.tar.gz /opt/
RUN tar xvzf /opt/logstash-filter-verifier_${LOGSTASH_FILTER_VERIFIER_VERSION}_linux_386.tar.gz -C /opt \
    && mv /opt/logstash-filter-verifier /usr/bin/

USER logstash
RUN <<EOF
mkdir tests
mkdir pipeline/pipeline1
mkdir pipeline/pipeline2

cat <<EOT > /usr/share/logstash/config/pipelines.yml
- pipeline.id: pipeline
  path.config: "pipeline/pipeline.conf"
EOT

cat <<EOT > /usr/share/logstash/tests/test.yml
input_plugin: "input"
ignore:
  - "@timestamp"
testcases:
  - input:
      - Test
    expected:
      - message: Test
        type: test_event
EOT

cat <<EOT > /usr/share/logstash/pipeline/pipeline.conf
input {
    stdin {
        id => "input"
        type => "test_event"
    }
}

output {
    stdout { }
}
EOT

cat <<EOT > /usr/share/logstash/run_tests.sh
echo "Starting daemon..."
logstash-filter-verifier daemon start &
sleep 5
logstash-filter-verifier daemon run --pipeline /usr/share/logstash/config/pipelines.yml --pipeline-base /usr/share/logstash/ --testcase-dir /usr/share/logstash/tests/test.yml --add-missing-id
EOT

chmod a+x run_tests.sh
EOF

CMD ["/bin/bash", "/usr/share/logstash/run_tests.sh"]

Build with

DOCKER_BUILDKIT=1 docker build --tag test .
docker run --rm test

It looks like add_field is persisted and works correctly, as well as tags, but type does not currently work.