[BUG] Seedfarmer doesn't detect changes from SSM

[gonzalobarbeito]

Describe the bug After a successful deployment using Seedfarmer that reads a parameter from SSM Parameter Store in manifests, if we change the referenced value from console, a redeployment of Seedfarmer won't pick up that change.

To Reproduce Steps to reproduce the behavior:

1. Deploy an app using Seedfarmer, with a parameter in SSM referenced in the manifests
2. Go to SSM in AWS Console
3. Change the value in the console
4. Still in consolde, go to CodePipeline and Release Change so the new value is applied to the deployed app.

Expected behavior While nothing changed in our code, the modification of referenced parameter should trigger a new build to incorporate the changes made in SSM

Actual Behavior No changes are detected, and no new build is started

[chamcca]

This is a known issue and an interesting problem. TL;DR - ensuring consistency of our security model makes this a difficult problem and we'll investigate options.

We use CodeBuild's ability to load SSM Parameters and SecretsManager Secrets as Environment Variables to pass these values to the Module for deployment. The IAM Role dedicated to the Module is the only Role with the ability to read these values. The CLI never reads the values and the Toolchain Role assumed by the CLI does not have permission to access them. This ensures that the values stored in SSM or SecretsManager are never exposed to or by the CLI in plain text.

The issue arises because the CLI is responsible for calculating the change set used to update modules. It does this by inspecting the manifests and the Module source code. But because it does not inspect (or even have access to) the SSM Parameter value, it does not include it in the change set calculation.

Granting the CLI's Toolchain Role the ability to read the SSM or SecretsManager values of the Module Parameters isn't an option; this breaks our security model and potentially exposes these values.

The CLI assumes a Deployment Role in each Target Account in order to check state of modules and read/write metadata. Granting this Role the ability to read these values might be an option, but there is no mechanism for allowing Module builders to grant read permissions to SSM or SecretsManager values to this Role. This is currently only done through the modulestack.yaml policy and attached to the Module Role.

A final option would be to utilize CodeBuild, the various Module Roles, and a PreCheck phase in the CLI to retrieve Versions of the SSM and SecretsManager values during change set detection and compare the current Versions to any previous Versions.

[a13zen]

Just chiming in here.

Adding a --force flag for seedfarmer apply allow us to force re-running the codebuild jobs (even knowing that they might have no change effect) would allow at least manually bypassing this limit.

It also allows us to build CICD pipelines that react to SSM parameter changes and trigger the CICD pipeline that runs seedfarmer apply, in which case we can then use the --force flag.

[a13zen]

Also, would allowing the deployment role permission to only read all parameter versions and not values still comply with the security requirements?

[chamcca]

I originally wanted to use the deployment role to read the parameter versions, but somehow looking through the ssm api and sdk docs I completely missed the describe_parameters method and thought that retrieving the full parameter value was the only way to get to the current version. This would have violated the security model.

But, now that I've found the describe_parameters method, using the deployment role to retrieve current versions of SSM Parameters and SecretsManager Secrets might be an option. Investigating.

[dgraeber]

@a13zen @gonzalobarbeito this has been implemented

[a13zen]

Very cool to see this feature drop so quickly! Thanks for the work!