Centralized Logging on AWS Solution

Centralized Logging on AWS has been superseded by the Centralized Logging with OpenSearch Centralized Logging with OpenSearch solution. All existing deployments will continue to work but the solution will no longer be supported and maintained.

Table of content

• Solution Overview
• Architecture
• Installation
• Customization
  • Setup
  • Changes
  • Unit Test
  • Build
  • Deploy
• Sample Scenario
• File Structure
• License

Solution Overview

Centralized Logging on AWS is a reference implementation that provides a foundation for logging to a centralized account. Customers can leverage the solution to index CloudTrail Logs, CW Logs, VPC Flow Logs on an Amazon OpenSearch Service domain. The logs can then be searched on different fields.

This solution gives you a turnkey environment to begin logging and analyzing your AWS environment and applications. Additionally, if you are looking to

• collect logs from multiple AWS accounts and organizations
• collect logs from multiple regions
• a single pane view for log analysis and visualization

then you can get all this with this 1-click deployment solution.

This solution uses Amazon OpenSearch Service (successor to Amazon Elasticsearch Service) and Kibana, an analytics and visualization platform that is integrated with Amazon OpenSearch Service, that results in a unified view of all the log events.

Architecture

The Centralized Logging on AWS solution contains the following components: log ingestion, log indexing, and visualization. You must deploy the AWS CloudFormation template in the AWS account where you intend to store your log data.

Customization

• Prerequisite: Node.js>=16 | npm >= 8

Setup

Clone the repository and run the following commands to install dependencies, format and lint as per the project standards

npm ci
npm run prettier-format
npm run lint

Changes

You may make any needed change as per your requirement. If you want to customize the Centralized Logging on AWS opinionated defaults, you can modify the solution manifest file. You can also control sending solution usage metrics to aws-solutions, from the manifest file.

"solutionVersion": "%%VERSION%%", #provide a valid value eg. v1.0
"sendMetric": "Yes",

Unit Test

You can run unit tests with the following command from the root of the project

 npm run test

Build

You can build lambda binaries with the following command from the root of the project

 npm run build

Deploy

Run the following command from the root of the project. Deploys all the primary solution components needed for Centralized Logging on AWS. Deploy in Primary Account

cd source/resources
npm ci

npm run cdk-bootstrap -- --profile <PROFILE_NAME>
npm run cdk-synth
npm run cdk-deploy -- CL-PrimaryStack --parameters AdminEmail=<EMAIL> --parameters SpokeAccounts=<ACCOUNT-ID-1,ACCOUNT-ID-2...> --parameters JumpboxKey=<EC2_KEY_PAIR> --parameters JumpboxDeploy='Yes' --profile <PROFILE_NAME>

Note: for PROFILE_NAME, substitute the name of an AWS CLI profile that contains appropriate credentials for deploying in your preferred region.

Sample Scenario (Enabling CloudWatch logging on Elasticsearch domain)

The default deployment uses opinionated values as setup in solution manifest file. In this scenario let's say we want to enable CloudWatch logging for ES domain.

You would need to update the ESDomain resource in cl-primary-stack.ts as below:

 logging: {
        slowSearchLogEnabled: true,
        appLogEnabled: true,
        slowIndexLogEnabled: true,
      },

File structure

Centralized Logging on AWS solution consists of:

• cdk constructs to generate needed resources
• helper for bootstrapping purposes like creating CloudWatch Logs Destinations
• transformer to translate kinesis data stream records into Elasticsearch documents

|-config_files                    [ config files for prettier, eslint etc. ]
|-architecture.png                [ solution architecture diagram ]
|-source/
  |dashboard.ndjson               [ sample dashboard for demo ]  
  |run-unit-test.sh               [ script to run unit tests ]
  |-resources
    |-bin/
      |-app.ts                    [ entry point for CDK app ]
    |-__tests__/                  [ unit tests for CDK constructs ] 
    |-lib/
      |-cl-demo-ec2-construct.ts  [ CDK construct for demo web server resource ]
      |-cl-demo-stack.ts          [ CDK construct for demo stack]
      |-cl-jumpbox-construct.ts   [ CDK construct for windows jumpbox resource ]  
      |-cl-primary-stack.ts       [ CDK construct for primary stack and related resources ]
      |-utils.ts                  [ utilities for generic functionalities across CDK constructs ]   
      |-manifest.json             [ manifest file for CDK resources ]
    |-config_files                [ tsconfig, jest.config.js, package.json etc. ]
  |-services/
    |-@aws-solutions/utils/       [ library with generic utility functions for microservice ]
    |-helper/                     [ lambda backed helper custom resource to help with solution launch/update/delete ]
    |-transformer/                [ microservice to translate kinesis records into es documents ]

License

See license here

Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.

Licensed under the Apache License Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at

http://www.apache.org/licenses/LICENSE-2.0

or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions and limitations under the License.