search

driskell / log-courier

The Log Courier Suite is a set of lightweight tools created to ship and process log files speedily and securely, with low resource usage, to Elasticsearch or Logstash instances.
Other
419 stars 107 forks source link

Logs lost in log-courier - > logstash #270

Closed rhoml closed 8 years ago

rhoml commented 8 years ago

More than an issue this ticket is to ask if someone has noticed some missing logs in kibana comparing to the logfiles.

At the moment our setup is similar to:

log-courier -> logstash(filter) -> AWS Kinesis -> logstash(indexer) -> Elasticsearch

From a couple of months ago we've noticed that some logs get lost and we see that on our log-courier box the logfile has 260+ entries with some "pattern" and when I search for the same pattern in kibana I only see roughly 60 entries. So I was wondering if someone has seem the same issues. I don't see any parsing errs in our logstash filters/indexers and neither in elasticsearch which makes me suspect of AWS kinesis or log-courier.

thanks for the help

driskell commented 8 years ago

Hi rhoml,

If you can reproduce in isolation, one good test is to use lc-admin to connect to Log Courier to see how many events published, and make sure it matches lines in the logs. Usually though this tends to be caused by problems further down the chain, for me I've seen it during indexing where Elasticsearch bulk output failed too many times and it discarded the events (it doesn't guarantee I don't think.)

driskell commented 8 years ago

HI rhoml, I will close this now. If you do continue to have issues let me know and I can try advise where else to look.