Logging stack fails upgrade from 1.6.3 to 1.7.0

richardkeit commented 3 months ago

Describe the bug Logging Stack fails with duplicate log name exists (screenshot below)

To Reproduce Move from 1.6.3 to 1.7.0 (using the TSE Codebase)

Expected behavior Upgrade works

Please complete the following information about the solution:

[x] Version: v1.7.0

To get the version of the solution, you can look at the description of the created AWS CloudFormation stack used to install the LZA (AWSAccelerator-InstallerStack). For example, "(SO0199) Landing Zone Accelerator on AWS. Version 1.5.1.". If the description does not contain the version information, you can look at the Parameters of the stack for the RepositoryBranchName as that should contain the version number.

[x] Region: ap-southeast-2
[ ] Was the solution modified from the version published on this repository?
[ ] If the answer to the previous question was yes, are the changes available on GitHub?
[ ] Have you checked your service quotas for the services this solution uses?
[ ] Were there any errors in the CloudWatch Logs?

Screenshots Existing stack version:

CloudTrail Entries:

[
  {
    "eventVersion": "1.09",
    "userIdentity": {
      "type": "AssumedRole",
      "principalId": "AROAUTHIL7XJVEBY366FN:AWSAccelerator-LoggingSta-AWSServiceRoleForAWSClou-8pHKlYXQbGjI",
      "arn": "arn:aws:sts::XXXXXXXXX:assumed-role/AWSAccelerator-LoggingSta-AWSServiceRoleForAWSCloud-I7cju9THeeyo/AWSAccelerator-LoggingSta-AWSServiceRoleForAWSClou-8pHKlYXQbGjI",
      "accountId": "XXXXXXXXX",
      "accessKeyId": "ASIAUTHIL7XJ4Y5AR3DZ",
      "sessionContext": "@{sessionIssuer=; attributes=}"
    },
    "eventTime": "2024-06-06T02:22:08Z",
    "eventSource": "logs.amazonaws.com",
    "eventName": "CreateLogGroup",
    "awsRegion": "ap-southeast-2",
    "sourceIPAddress": "13.54.85.221",
    "userAgent": "awslambda-worker/1.0 rusoto/0.48.0 rust/1.78.0 linux",
    "requestParameters": {
      "logGroupName": "/aws/lambda/AWSAccelerator-LoggingSta-AWSServiceRoleForAWSClou-8pHKlYXQbGjI"
    },
    "responseElements": null,
    "requestID": "c03e60b2-44db-4c58-aa08-3ef4da879999",
    "eventID": "63a79479-8cbc-4534-8c6b-ae98d46c6542",
    "readOnly": false,
    "eventType": "AwsApiCall",
    "apiVersion": "20140328",
    "managementEvent": true,
    "recipientAccountId": "XXXXXXXXX",
    "eventCategory": "Management",
    "tlsDetails": {
      "tlsVersion": "TLSv1.3",
      "cipherSuite": "TLS_AES_128_GCM_SHA256",
      "clientProvidedHostHeader": "logs.ap-southeast-2.amazonaws.com"
    }
  },
  {
    "eventVersion": "1.09",
    "userIdentity": {
      "type": "AssumedRole",
      "principalId": "AROAUTHIL7XJUV353OJC2:AWSAccelerator-LoggingSta-AWSServiceRoleForAWSClou-RwEwreN1DjfK",
      "arn": "arn:aws:sts::XXXXXXXXX:assumed-role/AWSAccelerator-LoggingSta-AWSServiceRoleForAWSCloud-tTk4hFbU4sYQ/AWSAccelerator-LoggingSta-AWSServiceRoleForAWSClou-RwEwreN1DjfK",
      "accountId": "XXXXXXXXX",
      "accessKeyId": "ASIAUTHIL7XJ4TC6JM6U",
      "sessionContext": "@{sessionIssuer=; attributes=}"
    },
    "eventTime": "2024-06-06T02:22:07Z",
    "eventSource": "logs.amazonaws.com",
    "eventName": "CreateLogGroup",
    "awsRegion": "ap-southeast-2",
    "sourceIPAddress": "3.107.0.129",
    "userAgent": "awslambda-worker/1.0 rusoto/0.48.0 rust/1.78.0 linux",
    "requestParameters": {
      "logGroupName": "/aws/lambda/AWSAccelerator-LoggingSta-AWSServiceRoleForAWSClou-RwEwreN1DjfK"
    },
    "responseElements": null,
    "requestID": "74ae3362-25cd-482d-823d-391028eb1650",
    "eventID": "aee253b5-8c75-4ad0-b2cd-e2f9654ad214",
    "readOnly": false,
    "eventType": "AwsApiCall",
    "apiVersion": "20140328",
    "managementEvent": true,
    "recipientAccountId": "XXXXXXXXX",
    "eventCategory": "Management",
    "tlsDetails": {
      "tlsVersion": "TLSv1.3",
      "cipherSuite": "TLS_AES_128_GCM_SHA256",
      "clientProvidedHostHeader": "logs.ap-southeast-2.amazonaws.com"
    }
  }
]

Additional context

richardkeit commented 3 months ago

Root cause identified - confirmed issue with code base logic

Custom Resource (Custom::CreateServiceLinkedRole) is invoked for serviceName cloud9.amazonaws.com. CreateLogStream request fails because log group doesn't exist
Internal process of Lambda execution creates LogGroup
Custom::CreateServiceLinkedRole CreateLogStream succeeds now that log group exists
Cloudformation attempts to describe the LogGroup prior to creation, fails as log group already exists

Suggested remediation: Hold the execution of the Custom::CreateServiceLinkedRole by adding an explicit DependsOn for the CloudwatchLogGroups created in CFN.

Below Cloudtrail exerts (in order per above): Failed CreateLogStream:

{
    "eventVersion": "1.09",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "YYYYYY:AWSAccelerator-LoggingSta-AWSServiceRoleForAWSClou-IFX56CyodGtV",
        "arn": "arn:aws:sts::XXXXXXXXX:assumed-role/AWSAccelerator-LoggingSta-AWSServiceRoleForAWSCloud-T5aZrEaGOYIq/AWSAccelerator-LoggingSta-AWSServiceRoleForAWSClou-IFX56CyodGtV",
        "accountId": "XXXXXXXXX",
        "accessKeyId": "ASIAUTHIL7XJTCC42DVP",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "YYYYYY",
                "arn": "arn:aws:iam::XXXXXXXXX:role/AWSAccelerator-LoggingSta-AWSServiceRoleForAWSCloud-T5aZrEaGOYIq",
                "accountId": "XXXXXXXXX",
                "userName": "AWSAccelerator-LoggingSta-AWSServiceRoleForAWSCloud-T5aZrEaGOYIq"
            },
            "attributes": {
                "creationDate": "2024-06-07T03:56:29Z",
                "mfaAuthenticated": "false"
            }
        }
    },
    "eventTime": "2024-06-07T03:56:33Z",
    "eventSource": "logs.amazonaws.com",
    "eventName": "CreateLogStream",
    "awsRegion": "ap-southeast-2",
    "sourceIPAddress": "13.239.5.175",
    "userAgent": "awslambda-worker/1.0 rusoto/0.48.0 rust/1.78.0 linux",
    "errorCode": "ResourceNotFoundException",
    "errorMessage": "The specified log group does not exist.",
    "requestParameters": {
        "logGroupName": "/aws/lambda/AWSAccelerator-LoggingSta-AWSServiceRoleForAWSClou-IFX56CyodGtV",
        "logStreamName": "2024/06/07/[$LATEST]a17db4c65498416ca816f49a1b779789"
    },
    "responseElements": null,
    "requestID": "3ed505a4-5a41-46eb-8c27-d560942c4614",
    "eventID": "d65b2bee-55f6-460f-b3f6-04e07aad3e61",
    "readOnly": false,
    "eventType": "AwsApiCall",
    "apiVersion": "20140328",
    "managementEvent": true,
    "recipientAccountId": "XXXXXXXXX",
    "eventCategory": "Management",
    "tlsDetails": {
        "tlsVersion": "TLSv1.3",
        "cipherSuite": "TLS_AES_128_GCM_SHA256",
        "clientProvidedHostHeader": "logs.ap-southeast-2.amazonaws.com"
    }
}

CreateLogGroup via Lambda function:

    "eventVersion": "1.09",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "YYYYYY:AWSAccelerator-LoggingSta-AWSServiceRoleForAWSClou-IFX56CyodGtV",
        "arn": "arn:aws:sts::XXXXXXXXX:assumed-role/AWSAccelerator-LoggingSta-AWSServiceRoleForAWSCloud-T5aZrEaGOYIq/AWSAccelerator-LoggingSta-AWSServiceRoleForAWSClou-IFX56CyodGtV",
        "accountId": "XXXXXXXXX",
        "accessKeyId": "ASIAUTHIL7XJTCC42DVP",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "YYYYYY",
                "arn": "arn:aws:iam::XXXXXXXXX:role/AWSAccelerator-LoggingSta-AWSServiceRoleForAWSCloud-T5aZrEaGOYIq",
                "accountId": "XXXXXXXXX",
                "userName": "AWSAccelerator-LoggingSta-AWSServiceRoleForAWSCloud-T5aZrEaGOYIq"
            },
            "attributes": {
                "creationDate": "2024-06-07T03:56:29Z",
                "mfaAuthenticated": "false"
            }
        }
    },
    "eventTime": "2024-06-07T03:56:33Z",
    "eventSource": "logs.amazonaws.com",
    "eventName": "CreateLogGroup",
    "awsRegion": "ap-southeast-2",
    "sourceIPAddress": "13.239.5.175",
    "userAgent": "awslambda-worker/1.0 rusoto/0.48.0 rust/1.78.0 linux",
    "requestParameters": {
        "logGroupName": "/aws/lambda/AWSAccelerator-LoggingSta-AWSServiceRoleForAWSClou-IFX56CyodGtV"
    },
    "responseElements": null,
    "requestID": "70d809d8-d749-4180-abdf-6659bffb8928",
    "eventID": "09190d4b-34d8-403d-b47f-15996e779bfb",
    "readOnly": false,
    "eventType": "AwsApiCall",
    "apiVersion": "20140328",
    "managementEvent": true,
    "recipientAccountId": "XXXXXXXXX",
    "eventCategory": "Management",
    "tlsDetails": {
        "tlsVersion": "TLSv1.3",
        "cipherSuite": "TLS_AES_128_GCM_SHA256",
        "clientProvidedHostHeader": "logs.ap-southeast-2.amazonaws.com"
    }
}

CreateLogStream succeeds via Lambda execution:

{
    "eventVersion": "1.09",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "YYYYYY:AWSAccelerator-LoggingSta-AWSServiceRoleForAWSClou-IFX56CyodGtV",
        "arn": "arn:aws:sts::XXXXXXXXX:assumed-role/AWSAccelerator-LoggingSta-AWSServiceRoleForAWSCloud-T5aZrEaGOYIq/AWSAccelerator-LoggingSta-AWSServiceRoleForAWSClou-IFX56CyodGtV",
        "accountId": "XXXXXXXXX",
        "accessKeyId": "ASIAUTHIL7XJTCC42DVP",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "YYYYYY",
                "arn": "arn:aws:iam::XXXXXXXXX:role/AWSAccelerator-LoggingSta-AWSServiceRoleForAWSCloud-T5aZrEaGOYIq",
                "accountId": "XXXXXXXXX",
                "userName": "AWSAccelerator-LoggingSta-AWSServiceRoleForAWSCloud-T5aZrEaGOYIq"
            },
            "attributes": {
                "creationDate": "2024-06-07T03:56:29Z",
                "mfaAuthenticated": "false"
            }
        }
    },
    "eventTime": "2024-06-07T03:56:33Z",
    "eventSource": "logs.amazonaws.com",
    "eventName": "CreateLogStream",
    "awsRegion": "ap-southeast-2",
    "sourceIPAddress": "13.239.5.175",
    "userAgent": "awslambda-worker/1.0 rusoto/0.48.0 rust/1.78.0 linux",
    "errorCode": "ResourceNotFoundException",
    "errorMessage": "The specified log group does not exist.",
    "requestParameters": {
        "logGroupName": "/aws/lambda/AWSAccelerator-LoggingSta-AWSServiceRoleForAWSClou-IFX56CyodGtV",
        "logStreamName": "2024/06/07/[$LATEST]a17db4c65498416ca816f49a1b779789"
    },
    "responseElements": null,
    "requestID": "3ed505a4-5a41-46eb-8c27-d560942c4614",
    "eventID": "d65b2bee-55f6-460f-b3f6-04e07aad3e61",
    "readOnly": false,
    "eventType": "AwsApiCall",
    "apiVersion": "20140328",
    "managementEvent": true,
    "recipientAccountId": "XXXXXXXXX",
    "eventCategory": "Management",
    "tlsDetails": {
        "tlsVersion": "TLSv1.3",
        "cipherSuite": "TLS_AES_128_GCM_SHA256",
        "clientProvidedHostHeader": "logs.ap-southeast-2.amazonaws.com"
    }
}

DescribeLogGroups fails due to Loggroup being created:

{
    "eventVersion": "1.09",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "AROAUTHIL7XJ7NJRQZAMT:AWSCloudFormation",
        "arn": "arn:aws:sts::XXXXXXXXX:assumed-role/AWSAccelerator-PipelineRole/AWSCloudFormation",
        "accountId": "XXXXXXXXX",
        "accessKeyId": "ASIAUTHIL7XJQAY5W356",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "AROAUTHIL7XJ7NJRQZAMT",
                "arn": "arn:aws:iam::XXXXXXXXX:role/AWSAccelerator-PipelineRole",
                "accountId": "XXXXXXXXX",
                "userName": "AWSAccelerator-PipelineRole"
            },
            "attributes": {
                "creationDate": "2024-06-07T03:57:17Z",
                "mfaAuthenticated": "false"
            }
        },
        "invokedBy": "cloudformation.amazonaws.com"
    },
    "eventTime": "2024-06-07T03:57:17Z",
    "eventSource": "logs.amazonaws.com",
    "eventName": "DescribeLogGroups",
    "awsRegion": "ap-southeast-2",
    "sourceIPAddress": "cloudformation.amazonaws.com",
    "userAgent": "cloudformation.amazonaws.com",
    "requestParameters": {
        "logGroupNamePrefix": "/aws/lambda/AWSAccelerator-LoggingSta-AWSServiceRoleForAWSClou-IFX56CyodGtV"
    },
    "responseElements": null,
    "requestID": "62190505-42e4-416f-8896-8e7354d3fac7",
    "eventID": "9293efc2-240d-4222-8c91-e0bcc9b12331",
    "readOnly": true,
    "resources": [
        {
            "accountId": "XXXXXXXXX",
            "type": "AWS::Logs::LogGroup",
            "ARN": "arn:aws:logs:ap-southeast-2:XXXXXXXXX:log-group:/aws/lambda/AWSAccelerator-LoggingSta-AWSServiceRoleForAWSClou-IFX56CyodGtV"
        }
    ],
    "eventType": "AwsApiCall",
    "apiVersion": "20140328",
    "managementEvent": true,
    "recipientAccountId": "XXXXXXXXX",
    "eventCategory": "Management"
}

erwaxler commented 2 months ago

Hey @richardkeit , thank you for reaching out to the Landing Zone Accelerator team! I've added some feedback to your pull request, thank you for implementing a fix. I plan to get your fix into our next release, I will keep this issue open until that becomes available.

erwaxler commented 2 months ago

Hi @richardkeit , your fix has been included in the v1.7.1 release so I'll go ahead and close this issue. Thank you again for contributing to the Landing Zone Accelerator!

Chandramouli15 commented 2 months ago

Hi Guys, trying to upgrade the lz from 1.4.3 to 1.7.0 version. Getting similar error " log group already exist" in log archive account, so manually deleted the log group but getting same error. As per the issue closing note issue fixed in 1.7.1 lz version , but getting same error in 1.7.1 version also. Please check and update. Thanks

erwaxler commented 2 months ago

Hi @Chandramouli15 , thanks for bringing this to our attention. Please respond here or create a new issue with the full error message to give us more visibility into the construct that is failing, from there I'll be able to investigate further.

Chandramouli15 commented 2 months ago

Thanks @erwaxler , new issue raised. please check "Logging stack fails upgrade from 1.4.3 to 1.7.1 #494".

awslabs / landing-zone-accelerator-on-aws

Logging stack fails upgrade from 1.6.3 to 1.7.0 #471