Logstash Cloud Foundry Filter

[ahahtyler]

Hello everyone,

I've been working on a logstash filter that interfaces with the PaaS platform Cloud Foundry and I would like to have it published in the logstash plugin repo.

Application logs generated in this environment contain a non-static GUID that relates back to an application. This makes it very difficult to determine a logs source when multiple applications are funneling to the same logging endpoint. This filter aims to solve this issue by using the GUID in each log to determine which application it came from.

I've put together a suite of rspec tests and a README at the github link below to better detail out the filter fields and features. As per the logstash filter creation guide the first step was to create an issue here. Any feed back, suggestions, or criticisms are welcomed.

You can find the filter here: https://github.com/ahahtyler/logstash-filter-cloudfoundry

[ahahtyler]

Figured I’d put together a short guide on how to set up Cloud Foundry, push and application, and forward logs to logsatsh.

Download the CF CLI

1. Go to https://github.com/cloudfoundry/cli/releases
2. Download and install the rpm, dpk, or exe for the OS where logstash is running.

Get access to Cloud Foundry

1. Go to https://run.pivotal.io/
2. Click “Sign Up For Free”.
3. Click “Create Account” and follow the instructions.
4. Once you’ve created your org you’re good to go.

Push your first application

1. The easiest app to install in my opinion is: https://github.com/cloudfoundry-samples/cf-php-demo
2. Download the application: git clone https://github.com/cloudfoundry-samples/cf-php-demo
3. In the manifest.yml remove lines 6 (host) and 9 (buildpack).
4. Log into Cloud Foundry through the CLI: cf login -a api.run.pivotal.io -u username -p password
5. From the command prompt navigate to the cf-php-demo directory
6. Push the application: cf push --random-route -b php_buildpack

Forward application logs to logstash

1. Create a syslog drain with the IP of your logstash instance and the port of a syslog input in your logstash config. cf cups logstash-drain -l syslog://IP:PORT
2. Bind it to your application: cf bs cf-php-demo logstash-drain
3. Restage app: cf restage cf-php-demo

Set up your logstash config

1. Set up the logstash filter plugin with the following info

    cf_api      => "https://api.run.pivotal.io"
    cf_user     => "username"
    cf_password => "password"
    cf_org      => "your org"
    cf_space    => "your space"

[ahahtyler]

Doubling back on this, just want to make sure there isn't any thing else I forgot to do to get this plugin reviewed.

[suyograo]

@ahahtyler I created https://github.com/logstash-plugins/logstash-filter-cloudfoundry. Can you do an initial PR?

[ahahtyler]

@suyograo Done and done. Thanks!

[ahahtyler]

I definitely dropped the ball on this one. I've submitted the initial pull request and realized I didn't put any reviewers on it: https://github.com/logstash-plugins/logstash-filter-cloudfoundry/pull/1

I'm not sure if anything else needs to be done. I see the subsequent Next round of new Logstash plugins has been quite for quite some time.