search

ministryofjustice / modernisation-platform

A place for the core work of the Modernisation Platform • This repository is defined and managed in Terraform
https://user-guide.modernisation-platform.service.justice.gov.uk
MIT License
680 stars 289 forks source link

Instance Scheduler can't start some EC2s #2542

Closed gfou-al closed 2 years ago

gfou-al commented 2 years ago

When invoking the Instance Scheduler lambda manually through the AWS web console in core-shared-services account.

The sprinkler-db-mgmt-server is stopped through the lambda, but cannot be started. The lambda logs are not reporting any error because the message to start the instance is successful. However, the EC2 reports the following errors after some time:

State transition reason: Server.InternalError
State transition message: Client.InternalError: Client error on launch

This behaviour only seems to apply to sprinkler-db-mgmt-server whereas a temporal instance I created did not have the same issue.

The following PR unfortunately did not resolve the issue and might have to be reverted:

Add "kms:CreateGrant" to instance scheduler role to try fixing the issue

gfou-al commented 2 years ago

The kms:CreateGrant permission when applied to instance scheduler role in member accounts resolved the issue. We're considering to add some filtering to restrict the permission.

julialawrence commented 2 years ago

https://github.com/ministryofjustice/modernisation-platform/pull/2559 <-- restricting kms:CreateGrant permission