A logstash plugin that allows to send logs to AWS CloudWatch Logs service.
To get started, you'll need JRuby with the Bundler gem installed.
Clone the repository.
Install dependencies
bundle install
bundle install
bundle exec rspec
Gemfile
and add the local plugin path, for example:
gem "logstash-output-cloudwatchlogs", :path => "/your/local/logstash-output-cloudwatchlogs"
bin/plugin install --no-verify
At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
You can use the same 2.1 method to run your plugin in an installed Logstash by editing its Gemfile
and pointing the :path
to your local plugin development directory or you can build the gem and install it using:
gem build logstash-output-cloudwatchlogs.gemspec
bin/plugin install /your/local/plugin/logstash-output-cloudwatchlogs.gem
Below sample configuration reads 2 log4j logs and sends them to 2 log streams respectively.
input {
file {
path => "/path/to/app1.log"
start_position => beginning
tags => ["app1"]
}
file {
path => "/path/to/app2.log"
start_position => beginning
tags => ["app2"]
}
}
filter {
multiline {
pattern => "^%{MONTHDAY} %{MONTH} %{YEAR} %{TIME}"
negate => true
what => "previous"
}
grok {
match => { "message" => "(?<timestamp>%{MONTHDAY} %{MONTH} %{YEAR} %{TIME})" }
}
date {
match => [ "timestamp", "dd MMM yyyy HH:mm:ss,SSS" ]
target => "@timestamp"
}
}
output {
if "app1" in [tags] {
cloudwatchlogs {
"log_group_name" => "app1"
"log_stream_name" => "host1"
}
}
if "app2" in [tags] {
cloudwatchlogs {
"log_group_name" => "app2"
"log_stream_name" => "host1"
}
}
}
Here are all the supported options:
In addition to configuring the AWS credential in the configuration file, credentials can also be loaded automatically from the following locations:
cloudwatchlogs {
"log_group_name" => "lg2"
"log_stream_name" => "ls1"
"batch_count" => 1000
"batch_size" => 1048576
"buffer_duration" => 5000
"queue_size" => 10
"dry_run" => false
}
git checkout -b my-new-feature
)git commit -am 'Added some feature'
)git push origin my-new-feature
)