not receiving all cloudtrail events in splunk

[kbroughton]

We deployed trumpet via the CloudFormation template. We noticed a lot of events which we could see in CloudTrail were missing in splunk. What could be causing this?

To investigate, we created equivalent dashboards in Splunk and CloudWatch to group events over the previous 3 days by service and eventName.

CloudWatch fields @timestamp, @message, eventName | stats count(*) as cnt by eventSource, eventName | sort eventSource desc

we get 390 distinct eventName's over the last 3 days in account X. There are 16 distinct ECR eventNames.

Splunk In splunk over the same time period, there are 59 distinct eventNames and only 8 distinct ECR eventNames.

sourcetype="aws-cloudtrail" aws_account_id="X" region="*" | stats count(eventName) as countEventName by eventName, eventSource | sort eventSource

Exporting both to CSV and doing a little munging with pandas I get the following info.

CloudWatch eventNames - Splunk eventNames =

'AssumeRole',
 'CreateKeyPair',
 'CreateSecurityGroup',
and the rest were Describe*, List*, Get*

Similarly, the services in CloudWatch - services in Splunk =

{'acm.amazonaws.com',
 'amazonmq.amazonaws.com',
 'apigateway.amazonaws.com',
 'application-insights.amazonaws.com',
 'autoscaling.amazonaws.com',
 'budgets.amazonaws.com',
 'ce.amazonaws.com',
 'cloudformation.amazonaws.com',
 'cloudfront.amazonaws.com',
 'cloudsearch.amazonaws.com',
 'codedeploy.amazonaws.com',
 'codepipeline.amazonaws.com',
 'cognito-identity.amazonaws.com',
 'cognito-idp.amazonaws.com',
 'compute-optimizer.amazonaws.com',
 'directconnect.amazonaws.com',
 'ds.amazonaws.com',
 'dynamodb.amazonaws.com',
 'ecs.amazonaws.com',
 'eks.amazonaws.com',
 'elasticache.amazonaws.com',
 'elasticbeanstalk.amazonaws.com',
 'elasticfilesystem.amazonaws.com',
 'elasticloadbalancing.amazonaws.com',
 'elasticmapreduce.amazonaws.com',
 'es.amazonaws.com',
 'glacier.amazonaws.com',
 'guardduty.amazonaws.com',
 'inspector.amazonaws.com',
 'kinesis.amazonaws.com',
 'kinesisanalytics.amazonaws.com',
 'kinesisvideo.amazonaws.com',
 'license-manager.amazonaws.com',
 'lightsail.amazonaws.com',
 nan,
 'organizations.amazonaws.com',
 'rds.amazonaws.com',
 'redshift.amazonaws.com',
 'resource-groups.amazonaws.com',
 'route53.amazonaws.com',
 'route53domains.amazonaws.com',
 'secretsmanager.amazonaws.com',
 'ses.amazonaws.com',
 'sns.amazonaws.com',
 'ssm.amazonaws.com',
 'sts.amazonaws.com',
 'support.amazonaws.com',
 'tagging.amazonaws.com',
 'workspaces.amazonaws.com',
 'xray.amazonaws.com'}

I would expect by default that all logs to CloudTrail would show up in Splunk. Note that on the items where data shows up in both CloudWatch and Splunk, the counts were equal, so it seems like more of a filtering issue than a transmission failure issue.

[cool-raj]

Do have any update here ?

[AlDawoode]

I'm not sure if my answer will help, but anyway :

That mean you need to generate a new trumpet template (with CloudTrial selected) and run it in Virginia Region (us-east-1). Because as you saw in the image, IAM pushes all IAM activity to Virginia Region ONLY.