search

mrkschan / nginxbeat

Superseded by https://github.com/elastic/beats/tree/master/metricbeat
Apache License 2.0
58 stars 7 forks source link

Publish multiple document types #10

Closed mrkschan closed 8 years ago

mrkschan commented 8 years ago

Resolve #7

mrkschan commented 8 years ago

yea, @urso, I just took a look at the libbeat/publisher/output.go and find that PublishEvents() would batch events so that network round trip can be reduced. Just wonder why don't PublishEvent() reuse PublishEvents() to make the library simplified.

urso commented 8 years ago

@mrkschan not sure if this kind of discussion would be better somewhere else instead of burying it in here (e.g. some separate github usse or discuss or stackoverflow).

Main reasoning PublishEvent is not reusing PublishEvents is that operators are allowed to disable usage of bulk inserts. In this case the output plugins PublishEvent function must be used to guarantee no bulk inserts are used (e.g. see elasticsearch outputer for example).

Also see async publisher not generating bulk workers (collecting multiple events into batches) if bulk insert is disabled. Some of bulk API yes or no config (and some other) can and should be pushed much closer (or right into) to the output plugin (we're just not there yet).

These are the results of some quite extensive refactoring to get sync/async in and I'm pretty sure we will see more refinements when starting on event filtering support.

If you want to help to continue refactoring the publisher feel free to discuss with me or start a PR.