search

aws / aws-cli

Universal Command Line Interface for Amazon Web Services
Other
15.34k stars 4.08k forks source link

PutAccountPolicy doesn't recognize RoleArn #8823

Open karmaniverous opened 1 month ago

karmaniverous commented 1 month ago

Describe the bug

I am attempting to create an account-level CloudWatch subscription to direct logs to a Firehose delivery stream in another account. When I send the request to create the policy, I get an error indicating that RoleArn is missing, even though RoleArn is configured.

Expected Behavior

PutAccountPolicy should complete without error.

Current Behavior

I get:

InvalidParameterException: An error occurred (InvalidParameterException) when calling the
PutAccountPolicy operation: destinationArn for vendor logs cannot be used without roleArn

Reproduction Steps

Run this CLI command (I ran it in the cloud console with Admin permissions just to eliminate any platform issues). Replace REMOTE_ACCOUNT_ID with some other account id. Don't know if the delivery stream hast to exist in the target account for this command, didn't get that far. But it DOES exist in my environment, along with (I think) all necessary cross-account permissions.

aws logs put-account-policy \
--policy-name audit-logs \
--policy-document "{\"DestinationArn\":\"arn:aws:logs::REMOTE_ACCOUNT_ID:deliverystream/AuditLogs\",\"FilterPattern\":{\"$.level\":\"audit\"},\"RoleArn\":\"arn:aws:iam:::role/AuditLogger\"}" \
--policy-type SUBSCRIPTION_FILTER_POLICY

Possible Solution

I think the CLI command is just failing to parse the input correctly.

Note that I get the same error in the cloud and on my desktop.

Additional Information/Context

Here's the Terraform for the local role referenced above. Terraform throws the same error, but I was able to validate (see above) that the error is coming from AWS not TF:

locals {
  audit_logs_delivery_stream_arn = "arn:aws:logs::REMOTE_ACCOUNT_ID:deliverystream/AuditLogs"
}

resource "aws_iam_role" "audit_logger" {
  name = "AuditLogger"

  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect = "Allow",
        Principal = {
          Service = "logs.amazonaws.com"
        },
        Action = "sts:AssumeRole"
      }
    ]
  })
}

# AuditLogger role policy document.
data "aws_iam_policy_document" "audit_logger_role" {
  statement {
    effect = "Allow"
    actions = [
      "firehose:PutRecord",
      "firehose:PutRecordBatch"
    ]
    resources = [local.audit_logs_delivery_stream_arn]
  }
}

# Attach AuditLogger role policy document to role.
resource "aws_iam_role_policy" "audit_logger" {
  role   = aws_iam_role.audit_logger.id
  policy = data.aws_iam_policy_document.audit_logger_role.json
}

# CloudWatch account-level audit log subscription.
resource "aws_cloudwatch_log_account_policy" "audit_log" {
  policy_name = "audit-log"
  policy_type = "SUBSCRIPTION_FILTER_POLICY"
  policy_document = jsonencode(
    {
      DestinationArn = local.audit_logs_delivery_stream_arn
      RoleArn        = aws_iam_role.audit_logger.arn
      FilterPattern  = { "$.level" = "audit" }
    }
  )
}

CLI version used

aws-cli/2.17.14 Python/3.11.9 Linux/6.1.96-102.177.amzn2023.x86_64 exec-env/CloudShell exe/x86_64.amzn.2023

Environment details (OS name and version, etc.)

AWS Cloud

tim-finnigan commented 1 month ago

Thanks for reaching out. The put-account-policy command makes a request to the IAM PutAccountPolicy API. The error here is coming from the API, so it's not directly related to the CLI.

But I think there's an issue with the command you're running. I'll try to highlight using a separate JSON file:

aws logs put-account-policy \
--policy-name audit-logs \
--policy-document file://policy.json \
--policy-type SUBSCRIPTION_FILTER_POLICY

Note that the accountid was missing in the RoleArn

{
    "DestinationArn": "arn:aws:firehose:us-west-2:accoundid:deliverystream/streamname",
    "FilterPattern": "{\"$.level\":\"audit\"}",
    "RoleArn": "arn:aws:iam::accountid:role/rolename"
}

When running the above command I do not see the error you reported.

karmaniverous commented 1 month ago

Well I think we've clicked one step farther along here and thanks for that. I think at this point I am experiencing a permissions issue. Going to lay it out here in hopes you will help me troubleshoot further.

This is the relevant permission on my target S3 bucket in account B:

{
    "Sid": "FirehoseAuditLogger",
    "Effect": "Allow",
    "Principal": {
        "Service": "firehose.amazonaws.com"
    },
    "Action": [
        "s3:AbortMultipartUpload",
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:PutObject",
        "s3:PutObjectAcl"
    ],
    "Resource": [
        "arn:aws:s3:::audit-log-bucket",
        "arn:aws:s3:::audit-log-bucket/*"
    ]
}

When I run the demo data test from the Firehose console (the Firehose destination is also in account B), the test records do indeed land in the bucket.

The AuditLogger role in account A that is supposed to receive CloudWatch logs & pass them to Firehose looks like this:

Trust Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.us-east-1.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Permissions

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "firehose:*",
            "Effect": "Allow",
            "Resource": "arn:aws:firehose:us-east-1:ACCOUNT_B_ID:deliverystream/AuditLogs"
        }
    ]
}

Now if I run this command in the cloud console from Account A:

aws logs put-account-policy \
--policy-name audit-logs \
--policy-document "{\"DestinationArn\": \"arn:aws:firehose:us-east-1:ACCOUNT_B_ID:deliverystream/AuditLogs\",\"FilterPattern\": \"{\\\"$.level:\\\"audit\\\"}\",\"RoleArn\": \"arn:aws:iam:us-east-1:ACCOUNT_A_ID:role/AuditLogger\"}" \
--policy-type SUBSCRIPTION_FILTER_POLICY

... I get this error:

An error occurred (InvalidParameterException) when calling the PutAccountPolicy operation: Could not
deliver test message to specified Firehose stream. Check if the given Firehose stream is in ACTIVE
state.

The Firehose stream is definitely active, because it was able to deliver its own test messages and also it indicates an Active state.

Any ideas?

tim-finnigan commented 1 month ago

Hm, I'm not sure what is causing the issue here if your trust policies are valid and the stream is active. Could you share your debug logs (with any sensitive info redacted) by adding --debug to that command? That may help give us more insight into the underlying issue.

karmaniverous commented 1 month ago

I'm afraid this may not be super helpful...

[cloudshell-user@ip-10-136-61-149 ~]$ aws logs put-account-policy \
> --policy-name audit-logs \
> --policy-document "{\"DestinationArn\": \"arn:aws:firehose:us-east-1:ACCOUNT_B_ID:deliverystream/AuditLogs\",\"FilterPattern\": \"{\\\"$.level:\\\"audit\\\"}\",\"RoleArn\": \"arn:aws:iam:us-east-1:ACCOUNT_A_ID:role/AuditLogger\"}" \
> --policy-type SUBSCRIPTION_FILTER_POLICY \
> --debug
2024-08-07 11:34:57,195 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.17.20 Python/3.11.9 Linux/6.1.97-104.177.amzn2023.x86_64 exec-env/CloudShell exe/x86_64.amzn.2023
2024-08-07 11:34:57,195 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['logs', 'put-account-policy', '--policy-name', 'audit-logs', '--policy-document', '{"DestinationArn": "arn:aws:firehose:us-east-1:ACCOUNT_B_ID:deliverystream/AuditLogs","FilterPattern": "{\\"$.level:\\"audit\\"}","RoleArn": "arn:aws:iam:us-east-1:ACCOUNT_A_ID:role/AuditLogger"}', '--policy-type', 'SUBSCRIPTION_FILTER_POLICY', '--debug']
2024-08-07 11:34:57,213 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x7fe6103ecf40>
2024-08-07 11:34:57,213 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x7fe610618900>
2024-08-07 11:34:57,213 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2024-08-07 11:34:57,213 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7fe61058a660>
2024-08-07 11:34:57,213 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7fe61058bce0>
2024-08-07 11:34:57,213 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x7fe6103ef920>
2024-08-07 11:34:57,214 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x7fe6106631a0>
2024-08-07 11:34:57,214 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2024-08-07 11:34:57,214 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x7fe6103ef7e0>
2024-08-07 11:34:57,214 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7fe6102d89d0>>
2024-08-07 11:34:57,215 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.17.20/dist/awscli/data/cli.json
2024-08-07 11:34:57,217 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x7fe610530cc0>
2024-08-07 11:34:57,217 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x7fe610530fe0>
2024-08-07 11:34:57,217 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x7fe610530f40>
2024-08-07 11:34:57,217 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x7fe610531120>
2024-08-07 11:34:57,217 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x7fe610531080>
2024-08-07 11:34:57,218 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x7fe6102cdb80>
2024-08-07 11:34:57,218 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.17.20 Python/3.11.9 Linux/6.1.97-104.177.amzn2023.x86_64 exec-env/CloudShell exe/x86_64.amzn.2023
2024-08-07 11:34:57,219 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['logs', 'put-account-policy', '--policy-name', 'audit-logs', '--policy-document', '{"DestinationArn": "arn:aws:firehose:us-east-1:ACCOUNT_B_ID:deliverystream/AuditLogs","FilterPattern": "{\\"$.level:\\"audit\\"}","RoleArn": "arn:aws:iam:us-east-1:ACCOUNT_A_ID:role/AuditLogger"}', '--policy-type', 'SUBSCRIPTION_FILTER_POLICY', '--debug']
2024-08-07 11:34:57,219 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x7fe6103ed800>
2024-08-07 11:34:57,219 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7fe6109ebce0>
2024-08-07 11:34:57,219 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x7fe610298400>
2024-08-07 11:34:57,219 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x7fe610ade840>
2024-08-07 11:34:57,219 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7fe610a03600>
2024-08-07 11:34:57,226 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2024-08-07 11:34:57,228 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7fe61064f9c0>
2024-08-07 11:34:57,228 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x7fe610614b80>
2024-08-07 11:34:57,243 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.17.20/dist/awscli/botocore/data/logs/2014-03-28/service-2.json
2024-08-07 11:34:57,250 - MainThread - botocore.hooks - DEBUG - Event building-command-table.logs: calling handler <function inject_tail_command at 0x7fe6104079c0>
2024-08-07 11:34:57,251 - MainThread - botocore.hooks - DEBUG - Event building-command-table.logs: calling handler <function inject_start_live_tail_command at 0x7fe610405e40>
2024-08-07 11:34:57,251 - MainThread - botocore.hooks - DEBUG - Event building-command-table.logs: calling handler <function add_waiters at 0x7fe6103ef7e0>
2024-08-07 11:34:57,265 - MainThread - botocore.hooks - DEBUG - Event building-command-table.logs: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7fe6102d89d0>>
2024-08-07 11:34:57,266 - MainThread - awscli.clidriver - DEBUG - OrderedDict([('policy-name', <awscli.arguments.CLIArgument object at 0x7fe60f5f6650>), ('policy-document', <awscli.arguments.CLIArgument object at 0x7fe60f5f6890>), ('policy-type', <awscli.arguments.CLIArgument object at 0x7fe60f5f6a50>), ('scope', <awscli.arguments.CLIArgument object at 0x7fe60f5f6c10>), ('selection-criteria', <awscli.arguments.CLIArgument object at 0x7fe60f5f6dd0>)])
2024-08-07 11:34:57,266 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.logs.put-account-policy: calling handler <function add_streaming_output_arg at 0x7fe6103edd00>
2024-08-07 11:34:57,266 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.logs.put-account-policy: calling handler <function add_cli_input_json at 0x7fe610a03f60>
2024-08-07 11:34:57,266 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.logs.put-account-policy: calling handler <function add_cli_input_yaml at 0x7fe610a28040>
2024-08-07 11:34:57,267 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.logs.put-account-policy: calling handler <function unify_paging_params at 0x7fe610618f40>
2024-08-07 11:34:57,281 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.17.20/dist/awscli/botocore/data/logs/2014-03-28/paginators-1.json
2024-08-07 11:34:57,281 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.logs.put-account-policy: calling handler <function add_generate_skeleton at 0x7fe6105034c0>
2024-08-07 11:34:57,282 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.logs.put-account-policy: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7fe60f5f7290>>
2024-08-07 11:34:57,282 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.logs.put-account-policy: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7fe60f5f6690>>
2024-08-07 11:34:57,282 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.logs.put-account-policy: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7fe60f7b0990>>
2024-08-07 11:34:57,282 - MainThread - botocore.hooks - DEBUG - Event building-command-table.logs_put-account-policy: calling handler <function add_waiters at 0x7fe6103ef7e0>
2024-08-07 11:34:57,282 - MainThread - botocore.hooks - DEBUG - Event building-command-table.logs_put-account-policy: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7fe6102d89d0>>
2024-08-07 11:34:57,283 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.logs.put-account-policy.policy-name: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fe6102fc2d0>
2024-08-07 11:34:57,283 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.logs.put-account-policy: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fe610b23610>
2024-08-07 11:34:57,284 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'audit-logs' for parameter "policy_name": 'audit-logs'
2024-08-07 11:34:57,284 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.logs.put-account-policy.policy-document: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fe6102fc2d0>
2024-08-07 11:34:57,284 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.logs.put-account-policy: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fe610b23610>
2024-08-07 11:34:57,284 - MainThread - awscli.argprocess - DEBUG - Param policy_document looks like JSON, not considered for param shorthand.
2024-08-07 11:34:57,284 - MainThread - awscli.arguments - DEBUG - Unpacked value of '{"DestinationArn": "arn:aws:firehose:us-east-1:ACCOUNT_B_ID:deliverystream/AuditLogs","FilterPattern": "{\\"$.level:\\"audit\\"}","RoleArn": "arn:aws:iam:us-east-1:ACCOUNT_A_ID:role/AuditLogger"}' for parameter "policy_document": '{"DestinationArn": "arn:aws:firehose:us-east-1:ACCOUNT_B_ID:deliverystream/AuditLogs","FilterPattern": "{\\"$.level:\\"audit\\"}","RoleArn": "arn:aws:iam:us-east-1:ACCOUNT_A_ID:role/AuditLogger"}'
2024-08-07 11:34:57,284 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.logs.put-account-policy.policy-type: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fe6102fc2d0>
2024-08-07 11:34:57,284 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.logs.put-account-policy: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fe610b23610>
2024-08-07 11:34:57,284 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'SUBSCRIPTION_FILTER_POLICY' for parameter "policy_type": 'SUBSCRIPTION_FILTER_POLICY'
2024-08-07 11:34:57,284 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.logs.put-account-policy.scope: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fe6102fc2d0>
2024-08-07 11:34:57,285 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.logs.put-account-policy.selection-criteria: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fe6102fc2d0>
2024-08-07 11:34:57,285 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.logs.put-account-policy.cli-input-json: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fe6102fc2d0>
2024-08-07 11:34:57,285 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.logs.put-account-policy.cli-input-yaml: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fe6102fc2d0>
2024-08-07 11:34:57,285 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.logs.put-account-policy.generate-cli-skeleton: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fe6102fc2d0>
2024-08-07 11:34:57,285 - MainThread - botocore.hooks - DEBUG - Event calling-command.logs.put-account-policy: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7fe60f5f7290>>
2024-08-07 11:34:57,285 - MainThread - botocore.hooks - DEBUG - Event calling-command.logs.put-account-policy: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7fe60f5f6690>>
2024-08-07 11:34:57,285 - MainThread - botocore.hooks - DEBUG - Event calling-command.logs.put-account-policy: calling handler <bound method GenerateCliSkeletonArgument.generate_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7fe60f7b0990>>
2024-08-07 11:34:57,286 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2024-08-07 11:34:57,286 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2024-08-07 11:34:57,286 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2024-08-07 11:34:57,286 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2024-08-07 11:34:57,286 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2024-08-07 11:34:57,286 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: custom-process
2024-08-07 11:34:57,286 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: config-file
2024-08-07 11:34:57,287 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: ec2-credentials-file
2024-08-07 11:34:57,287 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: boto-config
2024-08-07 11:34:57,287 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: container-role
2024-08-07 11:34:57,288 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): localhost:1338
2024-08-07 11:34:57,290 - MainThread - urllib3.connectionpool - DEBUG - http://localhost:1338 "GET /latest/meta-data/container/security-credentials HTTP/1.1" 200 1404
2024-08-07 11:34:57,296 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.17.20/dist/awscli/botocore/data/endpoints.json
2024-08-07 11:34:57,311 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fe61283e660>
2024-08-07 11:34:57,340 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.17.20/dist/awscli/botocore/data/logs/2014-03-28/endpoint-rule-set-1.json
2024-08-07 11:34:57,341 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.17.20/dist/awscli/botocore/data/partitions.json
2024-08-07 11:34:57,345 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.cloudwatch-logs: calling handler <function add_generate_presigned_url at 0x7fe6127532e0>
2024-08-07 11:34:57,346 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for logs via: environment_service
2024-08-07 11:34:57,346 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for logs via: environment_global
2024-08-07 11:34:57,346 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for logs via: config_service
2024-08-07 11:34:57,346 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for logs via: config_global
2024-08-07 11:34:57,346 - MainThread - botocore.configprovider - DEBUG - No configured endpoint found.
2024-08-07 11:34:57,348 - MainThread - botocore.endpoint - DEBUG - Setting logs timeout as (60, 60)
2024-08-07 11:34:57,349 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.cloudwatch-logs.PutAccountPolicy: calling handler <function base64_decode_input_blobs at 0x7fe6102984a0>
2024-08-07 11:34:57,349 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.cloudwatch-logs.PutAccountPolicy: calling handler <function generate_idempotent_uuid at 0x7fe6124689a0>
2024-08-07 11:34:57,350 - MainThread - botocore.regions - DEBUG - Calling endpoint provider with parameters: {'Region': 'us-east-1', 'UseDualStack': False, 'UseFIPS': False}
2024-08-07 11:34:57,351 - MainThread - botocore.regions - DEBUG - Endpoint provider result: https://logs.us-east-1.amazonaws.com
2024-08-07 11:34:57,351 - MainThread - botocore.hooks - DEBUG - Event before-call.cloudwatch-logs.PutAccountPolicy: calling handler <function inject_api_version_header_if_needed at 0x7fe61246a480>
2024-08-07 11:34:57,351 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=PutAccountPolicy) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'Logs_20140328.PutAccountPolicy', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'aws-cli/2.17.20 md/awscrt#0.20.11 ua/2.0 os/linux#6.1.97-104.177.amzn2023.x86_64 md/arch#x86_64 lang/python#3.11.9 md/pyimpl#CPython exec-env/CloudShell cfg/retry-mode#standard md/installer#exe md/distrib#amzn.2023 md/prompt#off md/command#logs.put-account-policy'}, 'body': b'{"policyName": "audit-logs", "policyDocument": "{\\"DestinationArn\\": \\"arn:aws:firehose:us-east-1:ACCOUNT_B_ID:deliverystream/AuditLogs\\",\\"FilterPattern\\": \\"{\\\\\\"$.level:\\\\\\"audit\\\\\\"}\\",\\"RoleArn\\": \\"arn:aws:iam:us-east-1:ACCOUNT_A_ID:role/AuditLogger\\"}", "policyType": "SUBSCRIPTION_FILTER_POLICY"}', 'url': 'https://logs.us-east-1.amazonaws.com/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fe60efa4c50>, 'has_streaming_input': False, 'auth_type': None}}
2024-08-07 11:34:57,351 - MainThread - botocore.hooks - DEBUG - Event request-created.cloudwatch-logs.PutAccountPolicy: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fe610ba5690>>
2024-08-07 11:34:57,351 - MainThread - botocore.hooks - DEBUG - Event choose-signer.cloudwatch-logs.PutAccountPolicy: calling handler <function set_operation_specific_signer at 0x7fe612468860>
2024-08-07 11:34:57,352 - MainThread - botocore.credentials - DEBUG - Credentials need to be refreshed.
2024-08-07 11:34:57,352 - MainThread - botocore.credentials - DEBUG - Credentials need to be refreshed.
2024-08-07 11:34:57,353 - MainThread - urllib3.connectionpool - DEBUG - http://localhost:1338 "GET /latest/meta-data/container/security-credentials HTTP/1.1" 200 1404
2024-08-07 11:34:57,354 - MainThread - botocore.credentials - DEBUG - Retrieved credentials will expire at: 2024-08-07 11:49:11+00:00
2024-08-07 11:34:57,354 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2024-08-07 11:34:57,354 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.1
host:logs.us-east-1.amazonaws.com
x-amz-date:20240807T113457Z
x-amz-security-token:*** SECURITY TOKEN ***
x-amz-target:Logs_20140328.PutAccountPolicy

content-type;host;x-amz-date;x-amz-security-token;x-amz-target
0cc6e10729fe6c2d152ecfaa9120188677524224ecff9c8949fa53fe6fff34b8
2024-08-07 11:34:57,355 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20240807T113457Z
20240807/us-east-1/logs/aws4_request
b0aae0310fcbf6f05ed54b2f331cd72a370b23a509a479ac33edf86f92b57314
2024-08-07 11:34:57,355 - MainThread - botocore.auth - DEBUG - Signature:
996240578ad2d785820ac1e4c2701d7b276ca2241bb4eef0b65b01e0f1a6f52d
2024-08-07 11:34:57,355 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://logs.us-east-1.amazonaws.com/, headers={'X-Amz-Target': b'Logs_20140328.PutAccountPolicy', 'Content-Type': b'application/x-amz-json-1.1', 'User-Agent': b'aws-cli/2.17.20 md/awscrt#0.20.11 ua/2.0 os/linux#6.1.97-104.177.amzn2023.x86_64 md/arch#x86_64 lang/python#3.11.9 md/pyimpl#CPython exec-env/CloudShell cfg/retry-mode#standard md/installer#exe md/distrib#amzn.2023 md/prompt#off md/command#logs.put-account-policy', 'X-Amz-Date': b'20240807T113457Z', 'X-Amz-Security-Token': b'*** SECURITY TOKEN ***', 'Authorization': b'AWS4-HMAC-SHA256 Credential=ASIAX6RYLW5TTWUBWNF3/20240807/us-east-1/logs/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=996240578ad2d785820ac1e4c2701d7b276ca2241bb4eef0b65b01e0f1a6f52d', 'Content-Length': '304'}>
2024-08-07 11:34:57,355 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/v2/2.17.20/dist/awscli/botocore/cacert.pem
2024-08-07 11:34:57,356 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): logs.us-east-1.amazonaws.com:443
2024-08-07 11:34:57,683 - MainThread - urllib3.connectionpool - DEBUG - https://logs.us-east-1.amazonaws.com:443 "POST / HTTP/1.1" 400 166
2024-08-07 11:34:57,684 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': '5f9ce527-1dfa-47e8-8d69-8a8cd145839a', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '166', 'Date': 'Wed, 07 Aug 2024 11:34:57 GMT', 'Connection': 'close'}
2024-08-07 11:34:57,684 - MainThread - botocore.parsers - DEBUG - Response body:
b'{"__type":"InvalidParameterException","message":"Could not deliver test message to specified Firehose stream. Check if the given Firehose stream is in ACTIVE state."}'
2024-08-07 11:34:57,685 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': '5f9ce527-1dfa-47e8-8d69-8a8cd145839a', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '166', 'Date': 'Wed, 07 Aug 2024 11:34:57 GMT', 'Connection': 'close'}
2024-08-07 11:34:57,686 - MainThread - botocore.parsers - DEBUG - Response body:
b'{"__type":"InvalidParameterException","message":"Could not deliver test message to specified Firehose stream. Check if the given Firehose stream is in ACTIVE state."}'
2024-08-07 11:34:57,686 - MainThread - botocore.hooks - DEBUG - Event needs-retry.cloudwatch-logs.PutAccountPolicy: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fe60efa68d0>>
2024-08-07 11:34:57,686 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2024-08-07 11:34:57,686 - MainThread - botocore.hooks - DEBUG - Event after-call.cloudwatch-logs.PutAccountPolicy: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7fe60efa6350>>
2024-08-07 11:34:57,687 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "awscli/clidriver.py", line 499, in main
  File "awscli/clidriver.py", line 634, in __call__
  File "awscli/clidriver.py", line 837, in __call__
  File "awscli/clidriver.py", line 963, in invoke
  File "awscli/clidriver.py", line 975, in _make_client_call
  File "awscli/botocore/client.py", line 360, in _api_call
  File "awscli/botocore/client.py", line 739, in _make_api_call
botocore.errorfactory.InvalidParameterException: An error occurred (InvalidParameterException) when calling the PutAccountPolicy operation: Could not deliver test message to specified Firehose stream. Check if the given Firehose stream is in ACTIVE state.

An error occurred (InvalidParameterException) when calling the PutAccountPolicy operation: Could not deliver test message to specified Firehose stream. Check if the given Firehose stream is in ACTIVE state.
[cloudshell-user@ip-10-136-61-149 ~]$
tim-finnigan commented 3 weeks ago

Thanks for following up. Did you follow the documentation here on setting up a role/permissions: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html#FirehoseExample ?

karmaniverous commented 2 weeks ago

Reading through our exchange here I see I only shared the Terraform configuration for the local account (the one generating the logs) not the remote account (the one hosting the stream & archive bucket).

Here's the remote account config:

# FirehoseAuditLogger role.
resource "aws_iam_role" "firehose_audit_logger" {
  name = "FirehoseAuditLogger"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "firehose.amazonaws.com"
        }
        Action = "sts:AssumeRole"
        # Use conditions to lock this down to audit logs from app accounts.
        # Condition = {
        #   StringEquals = {
        #     "sts:ExternalId" = "222222222222"
        #   }
        # }
      }
    ]
  })
}

# FirehoseAuditLogger role policy document.
data "aws_iam_policy_document" "firehose_audit_logger_role" {
  statement {
    effect = "Allow"
    actions = [
      "s3:AbortMultipartUpload",
      "s3:GetBucketLocation",
      "s3:GetObject",
      "s3:ListBucket",
      "s3:ListBucketMultipartUploads",
      "s3:PutObject",
      "s3:PutObjectAcl"
    ]
    resources = [
      "arn:aws:s3:::${aws_s3_bucket.master_audit_log.bucket}",
      "arn:aws:s3:::${aws_s3_bucket.master_audit_log.bucket}/*"
    ]
  }
  statement {
    effect    = "Allow"
    actions   = ["logs:PutLogEvents"]
    resources = ["*"]
  }
}

# Attach FirehoseAuditLogger role policy document to role.
resource "aws_iam_role_policy" "firehose_audit_logger" {
  role   = aws_iam_role.firehose_audit_logger.name
  policy = data.aws_iam_policy_document.firehose_audit_logger_role.json
}

# Create AuditLogsDelivery Firehose delivery stream.
resource "aws_kinesis_firehose_delivery_stream" "audit_logs" {
  name        = "AuditLogs"
  destination = "extended_s3"

  extended_s3_configuration {
    role_arn           = aws_iam_role.firehose_audit_logger.arn
    bucket_arn         = aws_s3_bucket.master_audit_log.arn
    prefix             = "logs/"
    compression_format = "GZIP"
  }
}

To answer your question: I certainly believe I have adhered to the reference you shared, with the caveat that I implemented the resources with Terraform instead of the command line. Am I missing something obvious?

tim-finnigan commented 3 days ago

Hm...I'm not sure what the issue is here. Have you tried reaching out to Terraform regarding this, or do you have any other updates on your need? Regarding the Firehose error, it seems like there are some subtle gotchas with setting the role permissions that people have reported on Stack Overflow in posts like the ones here and here, which may be helpful.