multiple output{} result in duplicate events

[shartge]

I noticed I see every event 4 times in the dashboards and wondered why that is. And then I noticed that every config file in /etc/logstash/conf.d includes an output to elasticsearch.

4 outputs -> 4 events

After removing the duplicate elasticsearch outputs (leaving the riemann one alone) and putting only one elasticsearch output into a separate es-output.conf file and restarting logstash the duplicate events are (of course) gone and the event counters for specific events like snapshot creation etc. look way more sensible now.

[shartge]

Suggestion:

Separate the outputs from the inputs in logstash/conf.d/, have one elasticsearch-output.conf and one riemann-output.conf and the other files only contain inputs and filters.

This way it is clear what data goes where.

[sexibytes]

Indeed we forgot to filter the ouputs! Thank you so much for reporting this :)

I think we're going to add the same conditions than the filter for the next release and re-think the conf files model as you suggested.

[shartge]

I don't see the point in having the same filters from the input on the output{}. Just creates needless overhead. After logstash has "done its thing" with an event it just ends up in the same ES instance anyway.

I just removed any output { elasticsearch ...} from any file and added a new es-output.conf with one elasticsearch output for all.

[rschitz]

you're right, that's the simple way to do it. thanks again.