search

s12v / awsbeats

AWS Kinesis plugin for Filebeat
Apache License 2.0
61 stars 26 forks source link

Firehose events don't have separator #39

Closed Bashton-MattBuckland closed 6 years ago

Bashton-MattBuckland commented 6 years ago

Hello,

I've been trying to use your firehose output to send to firehose -> s3 -> logstash, but I noticed some missing events. On closer inspection I noticed the contents of the s3 objects have lines with multiple json objects without any delimiter. Logstash doesn't complain about this; it processes the first event, then silently drops the rest of the events on any given line. The aws kinesis agent produces newline-delimited logs, so I'd expect awsbeats to do the same.

I've had a quick look through the code and I noticed this for the streams output: https://github.com/s12v/awsbeats/blob/master/streams/client.go#L138

But there is no corresponding manipulation for firehose/client.go. I tried an ignorant copy-paste of that line into what looked like the correct place in firehose/client.go but I ended up with broken json for some reason.

Am I missing something, or is that a bug?

Cheers,

Matt

Bashton-MattBuckland commented 6 years ago

Got it working. Have sent a PR (#40)... but it's failing tests. I'm a bit short on time at the moment but hopefully I can get back to this later and make it pass whatever it's failing.