awslabs / landing-zone-accelerator-on-aws

Deploy a multi-account cloud foundation to support highly-regulated workloads and complex compliance requirements.
https://aws.amazon.com/solutions/implementations/landing-zone-accelerator-on-aws/
Apache License 2.0
547 stars 436 forks source link

Issue with enable guardduty on security-config.yml #143

Closed quangchustudy closed 1 year ago

quangchustudy commented 1 year ago

*Describe the bug Issue with guardduty enable and it take more than one hour then cause to token be expired as below

IMG_5165

To Reproduce Modify security-config.yml as below

IMG_5166

Expected behavior Expect to finish without error

Please complete the following information about the solution:

Screenshots If applicable, add screenshots to help explain your problem (please DO NOT include sensitive information).

Additional context Add any other context about the problem here.

quangchustudy commented 1 year ago

Hi @awsclemj , any ideas?

erwaxler commented 1 year ago

Hi @quangchustudy , thank you for identifying this issue! Our team is currently investigating this behavior, I will provide an update here once we have more information.

thetechbender commented 1 year ago

@erwaxler @quangchustudy, I also see this error on a new LZA v1.4.0 deployment. Please let me know if I can provide additional info to help troubleshoot as this is a blocker for our current LZA deployment. Thanks!

rvanbutselaar commented 1 year ago

I have a similar issue when I try to enable GuardDuty. It looks like an issue with the AWSAccelerator-SecurityAu-CustomGuardDutyCreateMem Lambda function inside the Audit account.

The event which triggers the lambda is missing the ResponseURL, so it's unable to send back the status to CloudFormation:

{
    "RequestType": "Create",
    "ServiceToken": "arn:aws:lambda:eu-central-1:XXXXX:function:AWSAccelerator-SecurityAu-CustomGuardDutyCreateMem-6NoGk8ILZ2Vs",
    "ResponseURL": "...",
    "StackId": "arn:aws:cloudformation:eu-central-1:XXXXX:stack/AWSAccelerator-SecurityAuditStack-XXXX-eu-central-1/97be33b0-f4eb-11ed-a0ff-026e9f2381fc",
    "RequestId": "3a7cb1cb-a3c6-4e85-8e24-2c908ad5bd1a",
    "LogicalResourceId": "GuardDutyMembersD34CA003",
    "ResourceType": "Custom::GuardDutyCreateMembers",
    "ResourceProperties": {
        "ServiceToken": "arn:aws:lambda:eu-central-1:XXXXX:function:AWSAccelerator-SecurityAu-CustomGuardDutyCreateMem-6NoGk8ILZ2Vs",
        "partition": "aws",
        "enableS3Protection": "false",
        "region": "eu-central-1",
        "enableEksProtection": "false"
    }
}

The Lambda function is also hitting it's memory limit, and get's killed:


RequestId: ae70d767-8c1a-4a27-beea-7bbcb060bbdf Error: Runtime exited with error: signal: killed

REPORT RequestId: ae70d767-8c1a-4a27-beea-7bbcb060bbdf  Duration: 22198.77 ms   Billed Duration: 22199 ms   Memory Size: 128 MB Max Memory Used: 128 MB Init Duration: 156.17 ms
--
craigwilkins commented 1 year ago

I too encountered a memory issue, both Custom::GuardDutyUpdateDetector and Custom::GuardDutyCreateMembers used more than 128mb (the default in the constructs) eg. https://github.com/awslabs/landing-zone-accelerator-on-aws/blob/main/source/packages/%40aws-accelerator/constructs/lib/aws-guardduty/guardduty-members.ts#L56

Once increased, the deployment worked through to completion with no further issues.

GuardDutyCreateMembers

REPORT RequestId: cbfb79b8-d35a-40c8-8953-8472287efc37  Duration: 34516.16 ms   Billed Duration: 34517 ms   Memory Size: 512 MB Max Memory Used: 259 MB Init Duration: 173.13 ms    
erwaxler commented 1 year ago

@quangchustudy @thetechbender @rvanbutselaar This evening we released v1.4.1 which includes an increase to the memory of this Lambda function. I'll be closing this issue as resolved, please go ahead and update to the release/v1.4.1 branch to get past this. Thank you for your patience, and please continue to create issues if you experience unexpected behavior.