Setup Role for Fluentd and Graylog2 Stack - Centralized Log Management/Monitoring

[JJediny]

This is the master issue for log management. Most work should be performed in the other issues, in fact we may close this once those are all created, and associate them with the milestone.

Setup a Master Log Aggregator (Fluentd), Storage/Search (AWS Elasticsearch), and Dashboard/Event Management (Graylog2) for centralized logging and monitoring.

• [ ] Setup inventory of td-agents/plugins for each Ansible --tag
  • [x] Nginx
  • [x] Apache2
  • [x] Tomcat7
  • [ ] Python interpreter
  • [x] PHP interpreter
• [ ] Setup Role for main Graylog2 server
• [ ] Setup AWS Elasticsearch Service
• [x] Config AWS CloudWatch IAM and Plugin

[JJediny]

After some testing it has become clear that the ELK/EFK approach is great for analytics but is lacking in the event management to trigger notifications based on logging data. So after testing out Graylog2 we will be setting up a role for an instance of Graylog2 within the VPC using it to consolidate logs from Fluentd agents while also giving it a limited IAM role to also tie in AWS Cloudtrail and Flowlogs using https://github.com/Graylog2/graylog-plugin-aws

[neilhunt1]

Do we need this in separate VPC?

[neilhunt1]

@JJediny Just to clarify, in your diagram you show fluentd, elastic search, kibana in a separate VPC. Is that what you’d like to see (in the new case, Fluentd and Graylog2)? Do you want any automation I use to set these up to include the provisioning of the VPC? Or should we assume that VPC will be setup separately for now and I can start creating playbooks for FluentD and Graylog inside a hard-coded VPC?

[neilhunt1]

I am marking this as duplicate and closing. Let me know if we need to reopen. All the elements of this issue have been divided into their own issues and added to the Log Analysis milestone.