MyGovBC-Splunk-Forwarder

A NodeJS based Splunk forwarder for MyGovBC-MSP user interfaces.

Features:

1. Receive events on a given port from a variety of sources.
2. Log events locally in a rotating log structure in the Gluster PV attached.
3. Forwards it to HIBC Splunk server.
4. Allow remote access to local logs via a simple REST lookup protected by username and password.

Developer Prerequisites

• Node.js 4.2.5 or later.
• Splunk Enterprise 6.3.0 or later, or Splunk Cloud.
• An HTTP Event Collector token from your Splunk Enterprise server.

First, update npm to the latest version by running:

sudo npm install npm -g

Then run:

npm start server

To test:

• curl -XPOST -H "Authorization: Splunk XXX" -H "Content-Type: application/json" -d '{"body": "xyz"}' localhost:5504/log

  where XXX is the SERVICE_AUTH_TOKEN environment variable passed to the service.

Splunk Server Configuration

Splunk Server version 3.2.1 or greater is required.

1. Turn on HTTP Event Collector (HEC) by enabling its endpoint as it is not enabled by default.
2. Generate an HEC token.
3. Configure the HEC token in the configuration environment variable of the Splunk Forwarder.

For more information, pleast consult the Event Collector component of Splunk.

Splunk Forwarder Configuration

All configuration is done via a user's shell environment variable and read in NodeJS via process.env.

These are:

The max storage size used for the entire splunk-forwarder will be MAX_FILES * MAX_BYTES_PER_FILE. It's default storage size is 750mb.

A note on the two different AUTH tokens. The idea is to not expose the remote Splunk Server's authorization token to the service clients of the Splunk Forwarder. These use SERVICE_AUTH_TOKEN, while the Splunk Forwarder itself uses SPLUNK_AUTH_TOKEN generated by HEC.

To view and download local log files created by the Splunk Forwarder, go to URL of the route of the Splunk Forwarder followed by "/monitor". For example: https://gcpe-mygovbc-splunk-forwader-dev.pathfinder.gov.bc.ca/monitor This is often useful when the remote Splunk Service is unavailable.

The username and password should match the environment variables configured.

Splunk Forwarder Client

An application client wishing to use the Splunk-Forwarder, will make an HTTP POST request to the host/port that the Splunk Forwarder listens on. Typically, the following environment values are used in the client:

The client creates an http POST request. An example of a function that posts a message string in javascript:

function logSplunkError (message) {

var body = JSON.stringify({
   message: message
})

var options = {
  hostname: process.env.LOGGER_HOST,
  port: process.env.LOGGER_PORT,
  path: '/log',
  method: 'POST',
  headers: {
     'Content-Type': 'application/json',
     'Authorization': 'Splunk ' + process.env.SPLUNK_AUTH_TOKEN,
     'Content-Length': Buffer.byteLength(body),
     'logsource': process.env.HOSTNAME,
     'timestamp': moment().format('DD-MMM-YYYY'),
     'program': 'name of the client application',
     'serverity': 'error'
   }
};

var req = http.request(options, function (res) {
   res.setEncoding('utf8');
   res.on('data', function (chunk) {
      console.log("Body chunk: " + JSON.stringify(chunk));
   });
   res.on('end', function () {
      console.log('End of chunks');
   });
});

req.on('error', function (e) {
   console.error("error sending to splunk-forwarder: " + e.message);
});

// write data to request body
req.write(body);
req.end();
}

Notice that all POSTs are made to the path "/log" of the splunk-forwarder.

Also notice the format of the Authorization header as required by Splunk HEC.

Production Setup

See Deploy to OpenShift docs.