search

Graylog2 / graylog-plugin-integrations

A collection of open source Graylog integrations that will be released together.
Other
14 stars 14 forks source link

Add AWS SQS and SNS inputs #285

Open danotorrey opened 4 years ago

danotorrey commented 4 years ago

Description

Add support for reading messages directly from SQS and SNS. We indirectly support this wit the the CloudTrail input, but we have received at least one request from a customer asking to be able to read messages from an SQS queue directly.

References

See https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-how-it-works.html https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/welcome.html

Also see the indirect code that reads from an SQS queue for the CloudTrail input: https://github.com/Graylog2/graylog-plugin-aws/blob/dc36a579fb047772e1c14a46c4144f10b4f3521b/src/main/java/org/graylog/aws/inputs/cloudtrail/notifications/CloudtrailSQSClient.java#L40

superdave commented 4 years ago

I'd like to see this as well. I was looking for ways to get my CloudFront access logs into Graylog, and it looks like the current answer is to basically write a lambda to dump it into a Kinesis stream, which is... not great. It would be great if gzip'ed logs could be supported. Let me know if a PR would be helpful.

danotorrey commented 4 years ago

@superdave We recently added the ability to read arbitrary logs from within files in AWS S3 buckets with https://github.com/Graylog2/graylog-s3-lambda

This has not been specifically tested with CloudFront logs. However, if the log files are written to S3 with one line per log message, it could work. Gzip compression is also supported.

When each new log file is written to S3, each line in the file is read and sent to Graylog by the Lambda function. This method avoids the need for Kinesis.

superdave commented 4 years ago

Oh, hey, I didn't see that plugin called out anywhere in the documentation. That sounds fine in principle, but my Graylog instance isn't visible to the Internet, so it needs to be able to pull. Expanding the SQS option would be excellent for this (not uncommon, I think) use case.

I'm happy to contribute code when I have time, even if it does mean dirtying my hands with Java. :-)

superdave commented 4 years ago

I guess what gets me down is that I did set up the CloudTrail SQS reader in hopes that it might parse the CloudFront logs anyway. It doesn't, but I can see that it is consuming the SQS messages (and then ignoring them because they're for the wrong type), so my hope is it's a trivial extension. Maybe I'll try to bang something up this weekend.