search

pinojs / pino-elasticsearch

🌲 load pino logs into Elasticsearch
MIT License
179 stars 67 forks source link

Use the client bulk indexer #52

Closed delvedor closed 4 years ago

delvedor commented 4 years ago

The latest version of the client offers a bulk indexer, which automatically handles the retries and lets you pick the flush bytes threshold. You can find the bulk indexer documentation here.

The mechanics are slightly different, you can only know if a document has failed, so the insert event, which as far as I know it is only used for testing, will not be emitted more than once. The test now will fail because I didn't update the monkey patch yet, but I wanted to have an early feedback before continuing, especially around the stream handling.

This pr makes useless the bulk-size option, as the client's bulk indexer needs a value in bytes. This because you can't know in advance the size of a log, and could risk sending a big number of very big logs or very few small logs. The ability to set the flush size in bytes solves this problem.

A nice side effect of using the bulk indexer is that we gain the automatic retry of failed documents for free ;)

mcollina commented 4 years ago

This is pretty amazing. Huge +1.

delvedor commented 4 years ago

Oks, we should be ready! I've deprecated bulk-size, added flush-bytes and moved the test to async await :)