CloudWatch Loki Logging Example

Simple demo project to demonstrate how CloudWatch logs can be shipped to Grafanas Loki logging backend.

Check out the related blog post here.

Idea

The idea is to create a lambda function which registers a trigger for certain CloudWatch log groups. The function is trigger for all new log entries and forwards the entries to Loki. To enabled fine grained filter functionality in Loki, log groups are enriched with AWS tags. The AWS tags are transparently forwarded to Loki as labels.

Architecture

Prerequisites

• AWS Account
• AWC CLI User with the following permissions:
  • Lambda full Access
  • IAM read access
  • CloudWatch logs full access
  • CloudWatch events full access
• A IAM Role with name "loki-shipper" for the loki-shipper lambda function with the following permissions
  • CloudWatch logs full access
• Install virtualenv. Link
• Ensure aws credentials are in place. The Demo CLI utilizes boto3 and thus relies on the credentials. link

Install

• Clone this repository

• Install Loki and Grafana on a host which is reachable from the internet.

  • Copy docker-compose file to host, and run it.

    docker-compose up

  • Open Grafana UI http://:3000
  • Configure Loki datasource in Grafana() link
  • If deployed with docker-compose, the Loki endpoint is http://loki:3100
  • Ensure ports 3000 and 3100 are reachable from the internet

• Setup the virtual environment on your box

  source setup.rc

Attach Loki shipper to existing Application Logs

• Start Loki Shipper

  # Start the shipper lambda function
  (venv)$ python demo/cli.py -p <aws.profile> -r <aws-region> shipper -l http://<host-ip>:3100 start
  Created Lambda: loki-shipper

• Attach log group to loki-shipper

  (venv)$ python demo/cli.py -p <aws.profile> -r <aws-region>  attach --tags "tag1=tag1value" -t "tag2=tag2value" <log-group-to-be-shipped-to-loki>
  Created/Updated LogGroup: <log-group-to-be-shipped-to-loki>
  Created log subscription for <log-group-to-be-shipped-to-loki>

• Stop Loki Shipper

  # Start the shipper lambda function
  (venv)$ python demo/cli.py -p <aws.profile> -r <aws-region> shipper stop
  Stopped Lambda Function: loki-shipper

Full Demo

For demonstration purposes we'll deploy two continuously invoked demo lambdas to produce log entries in CloudWatch. CloudWatch events are used to schedule the lambdas.

In addition a loki-shipper lambda which subscribes to the demo lambdas log groups. In the following it is demonstrated how the demo can be installed and started.

The demo is started with the provided python CLI.

• Start Demo

  (venv)$ python demo/cli.py -p <aws.profile> -r <aws-region>  demo -l http://<host-ip>:3100  start   
  Starting lambda: loki-shipper
  Created Lambda: loki-shipper
  Created/Updated LogGroup: /aws/lambda/demofunc-func1
  Starting lambda: demofunc-func1
  Updated Lambda: demofunc-func1
  Created log subscription for /aws/lambda/demofunc-func1
  Created/Updated LogGroup: /aws/lambda/demofunc-func2
  Starting lambda: demofunc-func2
  Updated Lambda: demofunc-func2
  Created log subscription for /aws/lambda/demofunc-func2
  Demo start

• Stop Demo

  (venv)$ cd scripts
  (venv)$ python cli.py demo stop
  Demo stop

Grafana Loki UI

Now visit the Explore tab at http://:3000 and you should see something similar to: