Parsing errors when using --plugin-mock

jgough commented 2 years ago

Since all of the wonderful fixes since Beta 1 I thought I'd try building from master to test a few of these.

I am having difficulty with parser errors though. I don't know if I'm building it incorrectly, but the following pipeline produces an error when I use --plugin-mock

input {
  # Comment with a quote ' mark
  stdin {
    id => "input"
  }
  # And another quote ' mark
}

output { stdout {} }

Results in

logstash-filter-verifier: error: pipeline/pipeline1/01-input.conf:3:3 (18): no match found, expected: "#", "{" or [ \t\r\n]

I believe it is related to the ' character as when I remove these it works fine. If I replace with " it also fails. Additionally when I do not use --plugin-mock it works just fine. I didn't have this issue with Beta 1.

As I say, maybe I am building it incorrectly - in which case could you give me some pointers?

Once again a Dockerfile to repro! (let me know if these are helpful):

# syntax=docker/dockerfile:1.3-labs
FROM golang:1.17-buster as lfv_builder

RUN git clone https://github.com/magnusbaeck/logstash-filter-verifier.git
RUN apt update && apt install -y protobuf-compiler
# LFV tests do not work if run as root due to permissions so need to run as alternative user
RUN useradd -m -s /bin/bash go && chown -R go /go

USER go
WORKDIR /go/logstash-filter-verifier
RUN make && make check test

#------------------------------------------------------#

FROM docker.elastic.co/logstash/logstash:7.10.2

COPY --from=lfv_builder /go/logstash-filter-verifier /usr/bin

RUN <<EOF
mkdir tests
mkdir pipeline/pipeline1

cat <<EOT > /usr/share/logstash/config/pipelines.yml
- pipeline.id: pipeline1
  path.config: "pipeline/pipeline1/*.conf"
EOT

cat <<EOT > /usr/share/logstash/tests/test1.yml
input_plugin: "input"
ignore:
  - "@timestamp"
testcases:
  - input:
    - foo
    expected:
      - message: foo
EOT

cat <<EOT > /usr/share/logstash/pipeline/pipeline1/01-input.conf
input {
  # Comment with a quote ' mark
  stdin {
    id => "input"
  }
  # And another quote ' mark
}

output { stdout {} }
EOT

cat <<EOT > /usr/share/logstash/tests/plugin-mock.yml
- id: test
EOT

cat <<EOT > /usr/share/logstash/run_tests.sh
logstash-filter-verifier daemon start --no-cleanup &
sleep 5
logstash-filter-verifier daemon run --pipeline /usr/share/logstash/config/pipelines.yml --pipeline-base /usr/share/logstash/ --testcase-dir /usr/share/logstash/tests/test1.yml --add-missing-id --plugin-mock /usr/share/logstash/tests/plugin-mock.yml
EOT

EOF

CMD ["/bin/bash", "/usr/share/logstash/run_tests.sh"]

Run with

DOCKER_BUILDKIT=1 docker build --tag test .
docker run --rm -it test

Once again thanks for the hard work you guys are putting into this!

jgough commented 2 years ago

That changeset seems to test comments that are prefixed with // but logstash only supports comments prefixed with # ?

breml commented 2 years ago

Is this related to this change? breml/logstash-config@238da34#diff-bdb49894b46c8f4d3fd4a420659e75b04a009ee30ae86c7f384975f289f42085R15

Good catch, I must have been pretty tired, when I did this change.

breml commented 2 years ago

@jgough If you like, you can checkout #135 , which contains the fixed version of github.com/breml/logstash-config, which should resolve this issue.

Be aware, that a recent version of the Go compiler / tool chain is needed to build this, because of the updated dependencies.

jgough commented 2 years ago

I've grabbed and built that, seems to be working a lot happier now! Thanks.

magnusbaeck / logstash-filter-verifier

Parsing errors when using --plugin-mock #141