search

pinojs / pino-elasticsearch

🌲 load pino logs into Elasticsearch
MIT License
179 stars 67 forks source link

Indexing rate is very low #7

Closed megastef closed 8 years ago

megastef commented 8 years ago

As far my tests show the index rate is very slow, probably because of the little size of bulk requests.

Example, the test.log inludes 100 pino logs from restify: 

time cat test.log | ./pino-elasticsearch.js 
real    0m17.488s
user    0m0.526s
sys 0m0.059s

For 1000 logs 

time cat test.log | ./pino-elasticsearch.js 
real    1m32.682s
user    0m0.849s
sys 0m0.112s

pino-elasticsearch generates very small bulk requests with only 15 objects in each http-request (generated in each write from "pump" with only 15 chunks). Tested with several elasticsearch version on a MacBook Pro. When I use logagent-js, reading from a file 2 bulks with 1000 docs are generated and inserted in a second.

megastef commented 8 years ago

In my PR the bulk size is 500 and the times are much better :) 

time tail -n 1000 test.log | ./pino-elasticsearch.js

real    0m4.111s
user    0m0.639s
sys 0m0.070s