We are using logstash s3 input plugin to parse data of an S3 bucket that has millions of records. All the logfiles are prefixed with timestamp (eg:- yyyy-mm-dd-) i am unable to find a way to specify the pattern to match just the logs for the day. Any help is much appreciated.
The problem here is it has to cycle through all the files to pick the ones for today. i can use hard coded prefix (2019-11-) to match this month but multiple updates to this pipeline is not a feasible solution.
We are using logstash s3 input plugin to parse data of an S3 bucket that has millions of records. All the logfiles are prefixed with timestamp (eg:- yyyy-mm-dd-) i am unable to find a way to specify the pattern to match just the logs for the day. Any help is much appreciated.
The problem here is it has to cycle through all the files to pick the ones for today. i can use hard coded prefix (2019-11-) to match this month but multiple updates to this pipeline is not a feasible solution.