aws-solutions / instance-scheduler-on-aws

A cross-account and cross-region solution that allows customers to automatically start and stop EC2 and RDS Instances
https://aws.amazon.com/solutions/implementations/instance-scheduler-on-aws/
Apache License 2.0
550 stars 266 forks source link

Access denied creating log group for SpokeRegistration lambda #578

Open lorengordon opened 2 weeks ago

lorengordon commented 2 weeks ago

Describe the bug

When the central SpokeRegistration lambda executes (using the Organizations feature), it attempts to create a new log-group, and gets an access denied error. From CloudTrail:

"errorCode": "AccessDenied",
"errorMessage": "User: arn:aws:sts::redacted:assumed-role/instance-scheduler-SpokeRegistrationRole0E2E-7uqzJSJDYDNy/InstanceScheduler-default-SpokeRegistration is not authorized to perform: logs:CreateLogGroup on resource: arn:aws:logs:us-east-1:redacted:log-group:/aws/lambda/InstanceScheduler-default-SpokeRegistration:log-stream: because no identity-based policy allows the logs:CreateLogGroup action",

I believe this happens because the Resource ARN in the SpokeRegistration policy does not match the arn in the CreateLogGroup call:

"Resource": "arn:aws:logs:us-east-1:redacted:log-group:/aws/lambda/InstanceScheduler-default-SpokeRegistration",

I think it needs to be (this is what I use when creating policy for my own lambdas, which are able to create their own log-group):

"Resource": "arn:aws:logs:us-east-1:redacted:log-group:/aws/lambda/InstanceScheduler-default-SpokeRegistration:*",

To Reproduce

  1. Deploy the centralized cloudformation template.
  2. Deploy the remote cloudformation template.
  3. Check the central SpokeRegistration lambda, follow the "Monitoring" link to open the log-group
  4. Check Cloudtrail for the event logs:CreateLogGroup

Expected behavior

The log-group should exist, and log-streams should have log messages. It would be fine/preferred to pre-create the log-group with CloudFormation, instead of relying on the lambda function and its IAM policy. Pre-creating the log-group would allow setting the log expiration.

Please complete the following information about the solution:

To get the version of the solution, you can look at the description of the created CloudFormation stack. For example, "(SO0030) instance-scheduler-on-aws v1.5.1". You can also find the version from releases

aws-khargita commented 2 weeks ago

Hi @lorengordon, thanks for opening this issue. I was able to reproduce the issue and provide the lambda with the correct permissions. We will include this fix in our next release.

On another note, the logs in the specified log group are only the lambda invocation logs. If you are looking for logs on what accounts were registered/deregistered you can find those logs in the instance-scheduler-on-aws-logs log group under the SpokeRegistrations-<date> stream.

Let me know if you have any other questions!

lorengordon commented 2 weeks ago

On another note, the logs in the specified log group are only the lambda invocation logs. If you are looking for logs on what accounts were registered/deregistered you can find those logs in the instance-scheduler-on-aws-logs log group under the SpokeRegistrations-<date> stream.

Yep, understood! I was just clicking through all the Lambdas' invocation logs to ensure they were getting invoked correctly (I use SCPs to protect resources I provision to member accounts, so needed to make sure the SCP was not breaking functionality), and was surprised to when the log-group didn't exist! At first I thought, was this my SCP?