search

amiracle / grand_central

Grand Central logging for Cloud Services to Splunk
MIT License
36 stars 5 forks source link

readme

Grand Central App for Splunk

Manage and Monitor your Cloud Data Providers in Splunk from one centralized data platform.

This Splunk based app relies on the work done by Project Trumpet and the AWS Organizational model.

Grand Central User's Guide :

  • Version 3.0.1

    Getting Started

Requirements Grand Central works with the AWS Organizations framework and does not require either Landing Zone or Control Tower to work. By having the organization setup with multiple accounts, Grand Central will be able to discover the accounts and add into management within Splunk. Please refer to the Amazon Web Services documentation on how to get started with Organizations.

Architecture

architecture

Before Deploying Grand Central

You will need to be able to create an IAM User in the Master Account and the sub accounts that will be added into management under Splunk. By default there will be two IAM policies created, one to list all the accounts in the Organization and the second will be a deployment policy.

IAM Role Creation Shortcut (Simplified)

  1. Log into your Master AWS Account AWS Account: aws_console

  2. Click on Grand Central Create iam_create_cf

  3. Copy the Access Key / Secret Key from CloudFormation Output : copy_ak_sk

  4. Download grand_central_300.spl file from S3 bucket and install on your Splunk instance. Splunk Cloud customers request to have the app installed. install_gc

Cloudformation Template for IAM User Creation

Here is the Cloudformation template used for this IAM Role Creation : https://github.com/amiracle/grand_central/blob/master/aws_artifacts/cf_templates/GC_StackSet_UserCreate_CFTemplateRev2.json

Here is the policy which it will deploy for reference:

IAM Policy - GCPolicy_Rev2.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "CFTemplateSSInstance",
            "Effect": "Allow",
            "Action": [
                "cloudformation:CreateStackInstances",
                "cloudformation:DeleteStackInstances"
            ],
            "Resource": [
                "arn:aws:cloudformation:*:*:stackset-target/grandcentral*",
                "arn:aws:cloudformation:*:*:stackset/grandcentral*",
                "arn:aws:cloudformation:*::type/resource/AWS-IAM-Role",
                "arn:aws:cloudformation:*::type/resource/AWS-S3-Bucket",
                "arn:aws:cloudformation:*::type/resource/AWS-Lambda-Function",
                "arn:aws:cloudformation:*::type/resource/AWS-KinesisFirehose-DeliveryStream",
                "arn:aws:cloudformation:*::type/resource/AWS-IAM-Policy",
                "arn:aws:cloudformation:*::type/resource/AWS-Events-Rule"
            ]
        },
        {
            "Sid": "CFTemplateCreateStackSet",
            "Effect": "Allow",
            "Action": [
                "cloudformation:CreateStackSet",
                "cloudformation:DeleteStackSet"
            ],
            "Resource": [
                "arn:aws:cloudformation:*:*:stackset/grandcentral*",
                "arn:aws:cloudformation:*::type/resource/AWS-IAM-Role",
                "arn:aws:cloudformation:*::type/resource/AWS-S3-Bucket",
                "arn:aws:cloudformation:*::type/resource/AWS-Lambda-Function",
                "arn:aws:cloudformation:*::type/resource/AWS-KinesisFirehose-DeliveryStream",
                "arn:aws:cloudformation:*::type/resource/AWS-IAM-Policy",
                "arn:aws:cloudformation:*::type/resource/AWS-Events-Rule"
            ]
        },
        {
            "Sid": "OrganizationList",
            "Effect": "Allow",
            "Action": [
                "organizations:List*",
                "organizations:DescribeOrganization",
                "organizations:DescribeOrganizationalUnit",
                "organizations:EnableAWSServiceAccess"
            ],
            "Resource": "*",
            "Condition": {
                "ForAllValues:StringLike": {
                    "organizations:ServicePrincipal": [
                        "cloudformation.*.amazonaws.com",
                        "cloudformation-fips.*.amazonaws.com",
                        "cloudtrail.*.amazonaws.com",
                        "cloudtrail-fips.*.amazonaws.com",
                        "config.*.amazonaws.com",
                        "config-fips.*.amazonaws.com",
                        "events.*.amazonaws.com",
                        "events-fips.*.amazonaws.com",
                        "logs.*.amazonaws.com",
                        "logs-fips.*.amazonaws.com",
                        "s3.amazonaws.com",
                        "s3.*.amazonaws.com",
                        "organizations.*.amazonaws.com"
                    ]
                }
            }
        }
    ]
}

IAM Policy - Grand_Central_IAM_Policy.json Grand Central Policy for Master Account :

Individual Account IAM Policy (Optional)

If you are going to use individual accounts and policies in each account, use this IAM policy. Grand Central Policy for Individual AWS Accounts : Grand_Central_IAM_Policy.json

Setting up Grand Central

Adding Master Account Log into the Grand Central App and navigate to the Accounts Section (Configure Data Sources -> Amazon Web Services -> Grand Central Accounts).

https://yoursplunk.com:8000/en-US/app/grand_central/grand_central_accounts

Click on the “New Organization Master Account” button:

master_account

Add the Master Account ID (must be a number) access key / secret key to Splunk:

master_account

The Master Account should now show up in Splunk:

master_account

Once the Master Account has been added, now you should be able to view the accounts in the organization. Under Actions, navigate to the List All Accounts in the dropdown: master_account

List of all the accounts: master_account

In order to add the discovered accounts into Splunk, select "Add Accounts in Organization to Grand Central" in the Actions dropdown: master_account

Click on the Add Button:

master_account

All the accounts should now show up under management in Splunk: master_account

Splunk Endpoints Now, add the destination where you will be sending your data. This is typically a Firehose endpoint on your Splunk Cloud Deployment. Here is an example of how you should fill out the fields: Note that if you are using Splunk Cloud the URL for your firehose endpoint should look like this:

https://http-inputs-firehose-mystack.splunkcloud.com:443

master_account

BYOL Cloud Deployments

https://mystack.com:8088

master_account

AWS Stacksets Define your AWS Stackset here in Splunk. What will happen is as new accounts are vended into Organizational Units (OU's), they will automatically have these configurations sent to the newly AWS Accounts. Addiontally, if an existing account is moved into an OU, it will be configured with these settings.

Example : Account 987654321 is in the Core OU after being provisioned, then gets moved into the DevOps OU, the AWS Stackset "grandcentralCloudTrail" will deploy in that account and setup the CloudTrail data collection.

Deploy an AWS Stackset

aws_stack_set

Your Splunk Accounts should look like this when you're done: master_account

Pro Tip : For now, create a Splunk HEC token for each sourcetype. E.g. aws:cloudtrail for CloudTrail, aws:config for Config, and aws:cloudwatchlogs:vpcflow for VPCFlow Logs

The legacy Bulk Data Deployment system still works if you do not want to use AWS Stacksets and would rather use Access Key / Secret Keys for each of your accounts. Just follow the steps for the Bulk Credential Upload or set up each account individually. Now you can deploy data collection to all these accounts, click on the Bulk Deployment button and select your accounts, regions, destination and data source: master_account Click Deploy. Splunk will communicate with AWS and begin deploying the CloudFormation templates in all the accounts and regions you've selected: In the Observation Deck dashboard you will see the successfully deployed Accounts and Regions: master_account

Binary File Declaration

## Terraform
- Binary and Checksum: https://releases.hashicorp.com/terraform/0.12.19/
- Source Code: https://github.com/hashicorp/terraform

Alternative deployments

Under the hood, Grand Central is simply storing account configurations and deploying Stacksets/CloudFormation templates to those accounts, however the state and persistence of the Stackset/Cloudformation stack itself is stored within the AWS services. It is therefore possible to deploy the same architecture deployed by Grand Central using the native Stacket/Cloudformation consoles and CLI tools to better match any existing IaaS workflows you may already be using.

This section will briefly describe how to manually deploy the same architecture as Grand Central. This is especially relevant for users with concerns about long term storage of account credentials. Note that this section will be describing how to manually deploy to an AWS Organization using Stacksets, rather than individual accounts using Cloudformation. To deploy customized data collection Cloudformation templates to individual AWS accounts without Grand Central, please see Project Trumpet.

There are two main options for deploying the data collection infrastructure set up by Grand Central without relying on the Grand Central app for management of the stacks.