AccessDeniedException for containers trying to access SSM parameters

[MikeTheCanuck]

When deploying a fresh CloudFormation stack to test out the configuration of our services, I'm seeing that while the stack now successfully deploys, CloudWatch logs for the service containers are reporting that for any container attempting to read SSM parameters, they're getting this error for every call to aws ssm get-parameters:

An error occurred (AccessDeniedException) when calling the GetParameters operation: User: arn:aws:sts::845828040396:assumed-role/miketesting20190527-ECSRole-us-west-2/i-081cc40f40abe42cd is not authorized to perform: ssm:GetParameters on resource: arn:aws:ssm:us-west-2:845828040396:parameter/production/2018/API/local-elections/POSTGRES_NAME

https://user-images.githubusercontent.com/6953053/58753129-a9695680-846f-11e9-9709-af7517e02789.png

Originally sub-reported here: https://github.com/hackoregon/hackoregon-aws-infrastructure/issues/63#issuecomment-497974652

[MikeTheCanuck]

I've so far verified:

• the parameter still exists
• this is not a "SecureString" problem - even the unencrypted String parameters are returning AccessDeniedException

So now I'm wondering how we granted the original Role for SSM access (which I can't tell at first glance whether it's for accessing encrypted parameters or for any parameters) to ECS and/or to the tasks and/or to the containers: https://github.com/hackoregon/civic-devops/issues/114

[MikeTheCanuck]

Digging a bit deeper, here's how it appears that ECS containers are getting access to SSM parameters:

• The IAM Policy 2018-api-containers-parameters-access is attached to the IAM Role hacko-integration-ECSRole-us-west-2
• Somehow, the services/tasks/containers running in ECS hacko-integration stack are assuming this IAM role

I notice that in the above error, the "assumed-role" is named miketesting20190527-ECSRole-us-west-2/i-081cc40f40abe42cd, whereas the IAM Role that currently has access to the IAM Policy "2018-api-containers-parameters-access" is named hacko-integration-ECSRole-us-west-2.

It appears by all intents that we've generated the Role and Policy by hand, one-off, and that they're hard-coded to the current hacko-integration stack.

So it looks like we'll have to add some Policy and Role creation logic into the CF templates to be able to map these to whatever we need for each new stack we create (staging, production, test).

[MikeTheCanuck]

Update: tracing the problem, it appears that the root cause of our AccessDeniedException is this:

• the ECS role "miketesting20190527-ECSRole-us-west-2" is probably getting created - in another test, "arn:aws:iam::845828040396:role/testing1512-ECSRole-us-west-2" was successfully created
• it's even getting the inline Policy defined as "ecs-service" here: https://github.com/hackoregon/hackoregon-aws-infrastructure/blob/master/infrastructure/ecs-cluster.yaml#L152
• however, the difference between the working stack and my testing stacks is that the testing stacks aren't assigning the managed policy called 2018-api-containers-parameters-access to the Role - again (because I'm apparently slow) we should note that this is because the Policy was manually generated using aws iam create-policy and then manually attached to the Role using aws iam attach-role-policy

Question to consider: is it possible to inline the SSM policy? If yes, do we need to declare some variables somewhere in the stack templates to name the various resources being allowed in the policy? If not, how else can we anticipate and help someone else using these templates to make sure their own SSM parameter & key resources can be easily substituted for the ones that we currently have hard-coded in our 2018-api-containers-parameters-access policy?

[MikeTheCanuck]

Resolved by https://github.com/hackoregon/hackoregon-aws-infrastructure/pull/66

[MikeTheCanuck]

https://github.com/hackoregon/hackoregon-aws-infrastructure/issues/67 documents an enhancement to refactor the hard-coded SSM resources out of ecs-cluster.yaml and into master.yaml where such configuration parameters belong.