A BOSH release to help you ingest, parse, and visualize your AWS logs.
We can currently handle these log types...
We rely a process flow which looks like...
s3-notification
job waits for messages on that SQS topicWe like this setup because...
Before getting started, you must set up the following in AWS...
Fetch and install our logsearch configuration (details)...
$ wget -O- https://logsearch-for-aws-boshrelease.s3.amazonaws.com/release/latest/logsearch-config.tgz | tar -xz
$ open \
logsearch-config/logs/elasticsearch-mappings.json \
logsearch-config/logs/logstash-filters.conf
...snip...
Upload the release to your director...
$ bosh upload release https://logsearch-for-aws-boshrelease.s3.amazonaws.com/release/latest/tarball.tgz
Update your deployment to add the release, templates, and properties...
releases:
...snip...
- name: "logsearch-for-aws"
version: "latest"
jobs:
- name: "l4aws"
templates:
...snip...
- release: "logsearch-for-aws"
name: "s3-notification"
properties:
...snip...
l4aws:
access_key_id: "...snip..."
secret_access_key: "...snip..."
s3_notification:
queues:
# [ queue region , queue name , log format type ]
- [ "us-east-1" , "l4aws-billing" , "billing" ]
- [ "us-east-1" , "l4aws-cloudtrail" , "cloudtrail" ]
- [ "us-east-1" , "l4aws-s3" , "s3" ]
If you want to add parsing for a new log type... here are the things you should keep in mind...
s3-notification
job templates...
config/logstash.conf.erb
- add it to the list used to create file
inputsbin/main_ctl
- add it to the list used to mkdir
directorieslogsearch/logs.yml
- add a dummy entry to ensure the logsearch-config reference is activesrc/scripts/transform-{aws-log-format-name}
which will convert the raw S3 log file format into a single event per line.src/logsearch-config/logs/{aws-log-format-name}
...
{aws-log-format-name}
to name
expected.testdata
logstash-filters.conf
elasticsearch-mapping.json
src/aws-helper/s3-sns-sqs/generator/regenerate.sh
(and then execute) to add the new log type.src/aws-helper/iam/sample-policy.json
to add the new sample directory (if applicable).README.md
to add the log file format to our list of supported logs.Once updated, run ./bin/logsearch-config
to test your log parsing filters and generate new configuration in ./logsearch-config
. Use those configuration files in your test environment and verify your new log runs through the whole process.
Share your work with a Pull Request :)