search

lukewaite / logstash-input-cloudwatch-logs

Input plugin for Logstash to stream events from CloudWatch Logs
Other
142 stars 74 forks source link

Missing logs from cloudwatch #90

Closed SaharHudad closed 3 years ago

SaharHudad commented 3 years ago

Hi,

I'm using this plugin to transfer ecs containers logs from cloudwatch to logstash to kibana. it appears I'm missing some logs, I mentioned the entire logroup which has around 50-60 steams of logs but some logs are missing at the kibana dashboard. I do see all of the logs in the cloudwatch dashboard. Not seeing any errors just that some logs are missing.

We use the following setup: logstash 7.9.1 using logstash-input-cloudwatch_logs (1.0.3) Elasticsearch version 7.4.

Someone familiar with it?

srolskyi commented 3 years ago

@SaharHudad Yeah, we had the same situation. Looks like the current project isn't supported. Please take a look at this: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-awscloudwatch.html

Logic will be the same, but plus one additional tool Filebeat.

Filebeat grabs logs from cloudwatch, then transfer them into logstash.

SaharHudad commented 3 years ago

Thanks, Such a simple solution and works perfect.!