search

lukewaite / logstash-input-cloudwatch-logs

Input plugin for Logstash to stream events from CloudWatch Logs
Other
142 stars 74 forks source link

Logs from time window #25

Closed chris-west-travelex closed 7 years ago

chris-west-travelex commented 7 years ago

adding tom's changes

chris-west-travelex commented 7 years ago

sorry; meant to merge this to the Travelex fork! happy to discuss bringing this upstream though

tedder commented 7 years ago

To confirm, are you saying "don't bother getting logs any farther back than 1hr before container starts up"?

chris-west-travelex commented 7 years ago

@tedder that's exactly what this was for. we run logstash in a stateless container and can't (or rather, don't want to) rely on a persistent sincedb or similar. instead we start reading from X hours ago and rely on ElasticSearch's UPSERT behaviour to do away with any overlapping events.