Missing ID generation producing duplicate IDs causing Logstash failure

[jgough]

I've noticed that with a build from trunk I'm getting the following in the sut folder when my pipeline is generated:

filter {
  grok {
    match => [
      "[key]",
      "dog"
    ]
    id => "id_missing_0001"
  }
}
output {
  pipeline {
    send_to => [
      nextPipeline
    ]
    id => "id_missing_0001"
  }
}

Which contains duplicate IDs so logstash exits.

These two plugins are in separate files in my input config, so I don't know if something is going wrong with the id generation here with the ID counter being reset between files? I can try to produce a minimal test case to repro but may take me a couple of days to find time to do this.

[jgough]

Managed to repro this with the following minimal Dockerfile

# syntax=docker/dockerfile:1.3-labs
FROM golang:1.17-buster as lfv_builder

RUN git clone https://github.com/magnusbaeck/logstash-filter-verifier
RUN apt update && apt install -y protobuf-compiler
RUN useradd -m -s /bin/bash go && chown -R go /go

USER go
WORKDIR /go/logstash-filter-verifier
RUN make && make check test

FROM docker.elastic.co/logstash/logstash-oss:7.16.2

COPY --from=lfv_builder /go/logstash-filter-verifier /usr/bin

RUN <<EOF
mkdir tests
mkdir pipeline/pipeline1
mkdir pipeline/pipeline2

cat <<EOT > /usr/share/logstash/config/pipelines.yml
- pipeline.id: pipeline1
  path.config: "pipeline/pipeline1/*.conf"
- pipeline.id: pipeline2
  path.config: "pipeline/pipeline2.conf"
EOT

cat <<EOT > /usr/share/logstash/tests/test.yml
input_plugin: "input"
ignore:
  - "@timestamp"
testcases:
  - input:
      - >
        Test
    expected:
      - message: Test
EOT

echo 'input { stdin { id => "input" } }' > /usr/share/logstash/pipeline/pipeline1/1.conf
echo 'filter { mutate { add_field => { "blah" => "foo" } } }' > /usr/share/logstash/pipeline/pipeline1/2.conf
echo 'output { pipeline { send_to => [pipeline2] } }' > /usr/share/logstash/pipeline/pipeline1/3.conf

cat <<EOT > /usr/share/logstash/pipeline/pipeline2.conf
input {
  pipeline {
    address => pipeline2
  }
}
output {
  stdout {}
}
EOT

cat <<EOT > /usr/share/logstash/run_tests.sh
echo "Starting daemon..."
logstash-filter-verifier daemon start --loglevel debug --no-cleanup &
sleep 5
logstash-filter-verifier daemon run --pipeline /usr/share/logstash/config/pipelines.yml --pipeline-base /usr/share/logstash/ --testcase-dir /usr/share/logstash/tests/test.yml --add-missing-id

find /tmp/lfv-*/session/*/sut/pipeline/pipeline1/ -type f -exec cat {} +
grep ERROR /tmp/lfv-*/logstash-instance/*/logstash.log
EOT

chmod a+x run_tests.sh
EOF

Build and run with

DOCKER_BUILDKIT=1 docker build --tag test .
docker run --rm test

With this config two pulgins are given the same ID id_missing_0001 and logstash terminates with the error Config has duplicate Ids: : id_missing_0001