LZA Centralized Log Replication Not Working

[josh-romme]

Describe the bug global-config.yaml file should be set to have CloudWatch log groups replicated to Log Archive Centralized S3 Bucket. No subscription filter is applied to log groups and no logs are replicated.

Attempting to ensure this is working as designed by using a dynamic partitioning to log group named 'aws-client-vpn-lg', see log-filter.json file contents below:

[ { "logGroupPattern": "/AWSAccelerator-SecurityHub", "s3Prefix": "security-hub" }, { "logGroupPattern": "aws-client-vpn-lg", "s3Prefix": "aws-client-vpn"} ]

To Reproduce Use same global-config.yaml file and check for subscription filter applied to log groups.

Expected behavior Expect to see subscription filter applied to log groups for AWSAcceleratorCloudWatchToS3

Please complete the following information about the solution:

• [ ] Version: 1.3.2

• [ ] Region: us-east-1

• [ ] Was the solution modified from the version published on this repository? No

• [ ] If the answer to the previous question was yes, are the changes available on GitHub?

• [ ] Have you checked your service quotas for the sevices this solution uses? N/A

• [ ] Were there any errors in the CloudWatch Logs? No, pipeline runs successfully

logging:
  account: LogArchive
  centralizedLoggingRegion: us-east-1
  cloudtrail:
    enable: false
    organizationTrail: false
    organizationTrailSettings:
      multiRegionTrail: true
      globalServiceEvents: true
      managementEvents: true
      s3DataEvents: true
      lambdaDataEvents: true
      sendToCloudWatchLogs: true
      apiErrorRateInsight: false
      apiCallRateInsight: false
    accountTrails:
      - name: AccountTrail
        regions:
          - *HOME_REGION
        deploymentTargets:
          accounts: []
          organizationalUnits:
            - Root
        settings:
          multiRegionTrail: true
          globalServiceEvents: true
          managementEvents: true
          s3DataEvents: true
          lambdaDataEvents: true
          sendToCloudWatchLogs: true
          apiErrorRateInsight: false
          apiCallRateInsight: false
  sessionManager:
    sendToCloudWatchLogs: false
    sendToS3: true
    attachPolicyToIamRoles:
      - EC2-Default-SSM-AD-Role
  cloudwatchLogs:
    dynamicPartitioning: dynamic-partitioning/log-filters.json
    enable: true
    exclusions:
      - organizationalUnits:
        - cloud-accounts
        excludeAll: false
  centralLogBucket:
    lifecycleRules:
      - enabled: true
        id: CentralLifecycle
        abortIncompleteMultipartUpload: 14
        expiration: 2555
#        expiredObjectDeleteMarker: true
        noncurrentVersionExpiration: 2555
        noncurrentVersionTransitions:
          - storageClass: GLACIER
            transitionAfter: 365
        transitions:
          - storageClass: GLACIER
            transitionAfter: 365

Additional context Attempting to get log replication working, however the documentation is not clear on exactly which parameters need to be configured to ensure replication is occurring. Note that the exclusion is for non-LZA OU accounts, accounts that we want replication to be applied are not being excluded.

[awsclemj]

Hello @josh-romme, and thank you for reaching out to the LZA team!

Just to clarify the scope of the problem, are you not seeing any log groups being replicated to the central logs bucket, or is it just the aws-client-vpn-lg you mentioned in your dynamic partitioning configuration?

Documentation on LZA's centralized logging functionality can be found here. By default, there is no action you need to take in order for the subscription filter to be applied to log groups; this is applied automatically by a per-account EventBridge rule with the prefix AWSAccelerator-LoggingSta-NewCloudWatchLogsCreateE whenever a CreateLogGroup API call is made. This rule triggers a Lambda function with the same prefix value; you should be able to see if there are any failed invocations in the Monitoring tab of either of those resources. If you do happen to see any failures, could you provide us with the log stream from the failed invocation so that we may troubleshoot further?

Thanks, I look forward to hearing back from you!

[josh-romme]

Hi Jimmy,

We are not seeing any log groups get replicated. Does not appear that the subscription filter gets applied to any log groups. AWSAccelerator-LoggingSta-NewCloudWatchLogsCreateE EventBridge rule never shows any success or failed events...so it appears that the rule is never triggered at all when log groups are created. I have tested creating several log groups and then checked CloudTrail, the trail shows the event for log group created but EventBridge rule never gets triggered.

The only log group that gets replicated is the default /AWSAccelerator-SecurityHub from the best practices config file.

[awsclemj]

I have attempted replicating this issue in my environment (running v1.4.1) but I see the triggers and subscription are completing as expected. Additionally, this is not a known issue with v1.3.2.

@josh-romme, can you confirm whether or not the EventBridge rule is set to ENABLED and is targeting the appropriate Lambda function?

[josh-romme]

@awsclemj - I have confirmed that the EventBridge rule is ENABLED and targeting the appropriate Lamdba function.

[josh-romme]

Issue resolved