search

bpaquet / node-logstash

Simple logstash implmentation in nodejs : file log collection, sent with zeromq
Other
517 stars 141 forks source link

Elasticsearch bulk update #80

Closed fujifish closed 9 years ago

fujifish commented 9 years ago

All changes are for the ElasticSearch output plugin:

  1. Support elastic search bulk updates. Bulk updates enables sending multiple messages to be indexed in a single REST API call. The output plugin buffers messages in memory and sends them over to the elasticsearch server in bulk updates. This should dramatically reduce the load on the agent, on the network and on the server.
  2. In addition, I added the ability to specify a different index name other than the default logstash- index name through the 'index_type' option.
  3. Finally, I changes how the is computed for the index path. Instead of computing it for every message, compute it only once every minute and cache the result. This too should improve CPU load on the agent.
fujifish commented 9 years ago

Hi Bertrand, Any updates on this? our production environment really benefits from bulk updates.

bpaquet commented 9 years ago

If you really need perf, you should consider the ZeroMQ River plugin which already implements bulk insert, and save the cost of the HTTP stack on the both sides.

Anyway, I will try to merge this feature ASAP.