aws_cloudtrail: Create Trail from CloudTrail delegated admin

RubenFr commented 1 year ago

Describe the bug

I want to create an organization trail from a CloudTrail delegated administration account. The trail is supposed to ship all the logs to a S3 bucket in a centralized logging account. When I try to create the trail from the management console, everything works. However when I create it with CDK, I get the following error:

Error: The stack named my-stack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Resource handler returned message: "Resource of type 'AWS::CloudTrail::Trail' with identifier 'my-trail' was not found."

From what I saw in the logs, the trail is created in the management account and when CFN calls the StartLogging API, it looks for the train in the delegated admin account.

This is the CDK code:

from constructs import Construct
from aws_cdk import (
    Stack,
    aws_s3 as s3,
    aws_cloudtrail as cloudtrail,
)

class CloudTrailManagementStack(Stack):
    def __init__(
            self,
            scope: Construct,
            id: str,
            trail_name: str,
            target_bucket: s3.Bucket,
            **kwargs
    ) -> None:

        super().__init__(scope, id, **kwargs)

        trail = cloudtrail.Trail(
            self,
            "MyTrail",
            bucket=target_bucket,
            enable_file_validation=True,
            is_multi_region_trail=True,
            is_organization_trail=True,
            management_events=cloudtrail.ReadWriteType.ALL,
            trail_name=trail_name,
        )

N.B: When running this stack in the management account, everything worked.

Another weir behavior, when I look at the logs, I can see the API StartLogging is called before CreateTrail.

Expected Behavior

I would expect from the Trail to be created normally from the delegated admin account.

Current Behavior

I get the following issue:

Error: The stack named my-stack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: Resource handler returned message: "Resource of type 'AWS::CloudTrail::Trail' with identifier 'my-trail' was not found."

Reproduction Steps

Run the stack. Verify the bucket has the correct policy.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.90.0 (build 8c535e4)

Framework Version

No response

Node.js Version

v18.12.1

OS

amazon linux

Language

Python

Language Version

No response

Other information

No response

peterwoodworth commented 1 year ago

I don't think I understand this bug report, which resource getting created threw this error? I also suspect there is more code to go along with this that I'm not aware of, plus this snippet isn't copy+pastable since I have to guess at what some values are. Could you please give a stack with no variables that reproduces this error?

RubenFr commented 1 year ago

To fully reproduce the error:

Management account: 111111111111
Delegated Admin account: 222222222222
Centralized logging account: 333333333333

First Step

Add the 222222222222 as CloudTrail delegated Admin

Second Step

In the Centralized logging account deploy the following stack

#!/usr/bin/env python3

from aws_cdk import (
    App,
    Stack,
    Environment,
    aws_s3 as s3,
    aws_iam as iam,
    aws_s3 as s3,
)

app = App()
env = Environment(account='333333333333', region='us-west-2')
stack = Stack(app, 'trail-bucket-stack', env=env)
bucket_name = f'trail-bucket-{stack.account}-{stack.region}'

trail_bucket = s3.Bucket(
    stack,
    id='Bucket',
    bucket_name=bucket_name,
    access_control=s3.BucketAccessControl.PRIVATE,
    block_public_access=s3.BlockPublicAccess(
        block_public_acls=True,
        block_public_policy=True,
        ignore_public_acls=True,
        restrict_public_buckets=True),
    encryption=s3.BucketEncryption.S3_MANAGED,
    enforce_ssl=True,
    versioned=True
)

trail_bucket.add_to_resource_policy(

    # Allow CloudTrail to Get the Bucket ACL
    iam.PolicyStatement(
        sid="AWSCloudTrailAclCheck",
        effect=iam.Effect.ALLOW,
        principals=[
            iam.ServicePrincipal("cloudtrail.amazonaws.com")
        ],
        actions=["s3:GetBucketAcl"],
        resources=[f"arn:aws:s3:::{bucket_name}"],
        conditions={
            "StringEquals": {
                "AWS:SourceArn": f"arn:aws:cloudtrail:{stack.region}:111111111111:trail/org-trail"
            }
        }
    )
)

trail_bucket.add_to_resource_policy(

    # Allow CloudTrail to Put Logs in the Bucket
    iam.PolicyStatement(
        sid="AWSCloudTrailWrite",
        effect=iam.Effect.ALLOW,
        principals=[
            iam.ServicePrincipal("cloudtrail.amazonaws.com")
        ],
        actions=["s3:PutObject"],
        resources=[
            f"arn:aws:s3:::{bucket_name}/AWSLogs/*",
        ],
        conditions={
            "StringEquals": {
                "AWS:SourceArn": f"arn:aws:cloudtrail:{stack.region}:111111111111:trail/org-trail",
                "s3:x-amz-acl": "bucket-owner-full-control"
            }
        }
    )
)

app.synth()

Note: Although the trail will be created in the 222222222222 account, I found the only way for it to work is to but the management account as owner of the trail in the bucket policy.

Third Step

Create the Trail. Run this stack in the Delegated Admin account (222222222222)

#!/usr/bin/env python3

from aws_cdk import (
    App,
    Stack,
    Environment,
    aws_s3 as s3,
    aws_s3 as s3,
    aws_cloudtrail as cloudtrail
)

app = App()
env = Environment(account='222222222222', region='us-west-2')
stack = Stack(app, 'org-trail-stack', env=env)
bucket_name = f'trail-bucket-333333333333-{stack.region}'

trail_bucket = s3.Bucket.from_bucket_arn(
    stack,
    id="TrailBucket",
    bucket_arn=f"arn:aws:s3:::{bucket_name}"
)

cloudtrail.Trail(
    stack,
    "OrgTrail",
    bucket=trail_bucket,
    enable_file_validation=True,
    is_multi_region_trail=True,
    is_organization_trail=True,
    management_events=cloudtrail.ReadWriteType.ALL,
    trail_name='org-trail',
)

app.synth()

What doesn't work

I ran these two stacks and got the same error as before:

Error: The stack named org-trail-stack failed creation, it may need to be manually deleted from the AWS console: ROLL BACK_COMPLETE: Resource handler returned message: "Resource of type 'AWS::CloudTrail::Trail' with identifier 'org-trail' was not found."

I can see that the trail has been created with success but is Off. From the logs I can see two things:

CreateTrail (called with success)

"userIdentity": {
"type": "AssumedRole",
"principalId": "XXXXXXXXXXXXXXX:AWSCloudFormation",
"arn": "arn:aws:sts::222222222222:assumed-role/AWSCloudFormationExecutionRole/AWSCloudFormation",
"accountId": "222222222222",
"accessKeyId": "XXXXXXXXXXXXXXX",
"sessionContext": {
    "sessionIssuer": {
        "type": "Role",
        "principalId": "XXXXXXXXXXXXXXX",
        "arn": "arn:aws:iam::222222222222:role/AWSCloudFormationExecutionRole",
        "accountId": "222222222222",
        "userName": "AWSCloudFormationExecutionRole"
    },
    "attributes": {
        "creationDate": "2023-08-23T07:39:21Z",
        "mfaAuthenticated": "false"
    }
},
"invokedBy": "cloudformation.amazonaws.com"
},
"requestParameters": {
"name": "org-trail",
"s3BucketName": "trail-bucket-333333333333-us-west-2",
"s3KeyPrefix": "",
"snsTopicName": "",
"includeGlobalServiceEvents": true,
"isMultiRegionTrail": true,
"enableLogFileValidation": true,
"kmsKeyId": "",
"isOrganizationTrail": true
},
"responseElements": {
"name": "org-trail",
"s3BucketName": "trail-bucket-333333333333-us-west-2",
"s3KeyPrefix": "",
"includeGlobalServiceEvents": true,
"isMultiRegionTrail": true,
"trailARN": "arn:aws:cloudtrail:us-west-2:111111111111:trail/org-trail",
"logFileValidationEnabled": true,
"isOrganizationTrail": true
}

Note: You can see in the response that the trail has been created in the management account, although the command is called in the 222222222222 account.

StartLogging --> Failed with error: Unknown trail: arn:aws:cloudtrail:us-west-2:222222222222:trail/org-trail for the user: 222222222222

"userIdentity": {
    "type": "AssumedRole",
    "principalId": "XXXXXXXXXXXXXXX:AWSCloudFormation",
    "arn": "arn:aws:sts::222222222222:assumed-role/AWSCloudFormationExecutionRole/AWSCloudFormation",
    "accountId": "222222222222",
    "accessKeyId": "XXXXXXXXXXXXXXX",
    "sessionContext": {
        "sessionIssuer": {
            "type": "Role",
            "principalId": "XXXXXXXXXXXXXXX",
            "arn": "arn:aws:iam::222222222222:role/AWSCloudFormationExecutionRole",
            "accountId": "222222222222",
            "userName": "AWSCloudFormationExecutionRole"
        },
        "attributes": {
            "creationDate": "2023-08-23T07:39:21Z",
            "mfaAuthenticated": "false"
        }
    },
    "invokedBy": "cloudformation.amazonaws.com"
},
"eventTime": "2023-08-23T07:39:22Z",
"eventSource": "cloudtrail.amazonaws.com",
"eventName": "StartLogging",
"awsRegion": "us-west-2",
"sourceIPAddress": "cloudformation.amazonaws.com",
"userAgent": "cloudformation.amazonaws.com",
"errorCode": "TrailNotFoundException",
"errorMessage": "Unknown trail: arn:aws:cloudtrail:us-west-2:222222222222:trail/org-trail for the user: 222222222222",
"requestParameters": {
    "name": "org-trail"
},
"responseElements": null,

Note: You can see in the command is looking for the trail in the 222222222222 account, which we saw before that was created in the 111111111111 account... An interesting thing is that if I go to the management console and StartLogging the trail manually, it works...

@peterwoodworth - If you need anything else, fell free to ask

FilipPyrek commented 1 year ago

I'm also getting this error. It's very strange.

I also had to switch to CfnTrail L1 construct because I needed to pass Topic ARN instead of topic name, since it's multi account Trail. https://github.com/aws/aws-cdk/issues/11387

Policies for both KMS encryption key, S3 bucket and SNS topic should be okay. Because I spent quite some time with debugging those and also contacted AWS support for some help.

But now I'm stuck at this strange error. 🤯

mhmdio commented 1 year ago

same here, you can see how much workaround done here https://github.com/awslabs/landing-zone-accelerator-on-aws

FilipPyrek commented 1 year ago

I just got message from AWS support on this:

After searching internally, I was able to find that this is a known issue and that creating an Organization trail using a CloudFormation template is not supported yet. There is already an open feature request for this.

However, It is possible to create a CloudTrail trail using CloudFormation and then update that trail using the AWS CLI [3] to enable the trail as an Organization wide Multi-region trail [4].

I have added your voice to the feature request. As Premium support has no visibility over the process, I cannot comment on an ETA for when the feature may get released. However, you may stay updated through our What's New [5] and Blog [6] pages on such feature release news.

So simply said, CloudFormation offers this option, but doesn't support it. That's crazy 😢

faultylee commented 1 year ago

I'm not using CDK but I ran into the same issue with deploying CloudTrail on the delegated admin account through CloudFormation. It worked once yesterday, then failed every since. When I get the error, the trail is already created. At this point I suspect this is due to repeated creation and deletion of CloudTrail with the same name and configuration. What I noticed is that the moment the S3 bucket is created, it was already populated with CloudTrail objects even though the new Trail is not enabled yet. This gave me the impression that the old trail of the same name and configuration still running in the background trying to upload S3 objects and waiting for timeout to happen. I'll wait a few days and see if this issue goes away.

faultylee commented 1 year ago

Update: I still experience the same issue. Someone shared https://github.com/hashicorp/terraform-provider-aws/issues/28440 with me and it appears to be the same issue. So the root issue is that when the CloudTrail is created, the returned arn contains the management account id instead of the delegated admin account id, which confused CloudFormation thinking the Trail failed to create.

beniusij commented 1 year ago

I am currently experiencing the same issue as @faultylee described though I use CDK (2.73.0). Creating an org trail through a delegated administrator account results in an error message saying that the trail with a given name is not found even though that trail does appear in the console across different accounts of the organization. Furthermore, the associated s3 bucket has no policies set. Finally, the trail arn contains the root account's ID rather than the delegated admin account's id, which might explain the error.

Are there any plans to remedy this issue any time soon or is there a suggested workaround? Currently, I am considering creating non-org trails in each account and setting them to use the s3 bucket created in the delegated account which is responsible for storing centralised cloudtrail logs. I think that should replicate the intended behaviour of using isOrganizationTrail flag, just lengthy, but I'd like to see if there are any other suggestions for solving this.

FilipPyrek commented 1 year ago

I am currently experiencing the same issue as @faultylee described though I use CDK (2.73.0). Creating an org trail through a delegated administrator account results in an error message saying that the trail with a given name is not found even though that trail does appear in the console across different accounts of the organization. Furthermore, the associated s3 bucket has no policies set. Finally, the trail arn contains the root account's ID rather than the delegated admin account's id, which might explain the error.

Are there any plans to remedy this issue any time soon or is there a suggested workaround? Currently, I am considering creating non-org trails in each account and setting them to use the s3 bucket created in the delegated account which is responsible for storing centralised cloudtrail logs. I think that should replicate the intended behaviour of using isOrganizationTrail flag, just lengthy, but I'd like to see if there are any other suggestions for solving this.

@beniusij I overcome it in a way that I setup everything in the delegated administrator account (S3 bucket, KMS encryption key, etc.) and then just created the organization trail manually in the root account. Not ideal, but doesn't hurt that much, because probably I wont touch the CloudTrail configuration anymore anyway.

faultylee commented 1 year ago

Currently, I am considering creating non-org trails in each account and setting them to use the s3 bucket created in the delegated account which is responsible for storing centralised cloudtrail logs.

I won't recommend this because the output path is different and it can't enforce CloudTrail on any new account. You can create CloudTrail manually via the delegated account if that's what you need.

jpSimkins commented 6 months ago

So it seems this is still the case. I am having this same issue. It worked when my CloudFormation template was deployed on the primary account but having the same issue with the delegated admin. Wasted 8 hours today on this...

aws / aws-cdk