aws-solutions / aws-control-tower-customizations

The Customizations for AWS Control Tower solution combines AWS Control Tower and other highly-available, trusted AWS services to help customers more quickly set up a secure, multi-account AWS environment using AWS best practices.
https://docs.aws.amazon.com/controltower/latest/userguide/cfct-overview.html
Apache License 2.0
355 stars 205 forks source link

Custom-Control-Tower-CodeBuild step failing due to expired GPG key signature. #144

Closed yotixify closed 1 year ago

yotixify commented 1 year ago

Describe the bug AWS Control Tower Customizations is failing in the Custom-Control-Tower-CodeBuild step of the CodePipeline project due to an expired GPG Key error with yarn. There is an open issue with this on yarns github page located here: https://github.com/yarnpkg/yarn/issues/7866#issuecomment-1403132277

<span><span>
W: GPG error: https://dl.yarnpkg.com/debian stable InRelease: The following signatures were invalid: EXPKEYSIG 23E7166788B63E1E Yarn Packaging <yarn@dan.cx>

When I added curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - to the buildspec file I was able to solve the issue and the build ran through normally.

Edit: It is every step that includes yarn. Initially mentioned the Build stage, however it seems every step depends on yarn packages.

Edit 2: This is really only a bug until AWS updates their default codebuild image, looks like the new key should be valid until 2026. So this really should pop up only every couple years, I don't think it hurts to keep it in but I can also see this as more of an issue with the default Codebuild than the customizations framework. The blocker of not being able to deploy critical infra due to this bug is annoying but not the end of the world as the fix is relatively easy, and I could see the argument made this should fall more on the codebuild image side. If the control tower team doesn't want to add this as a recurring call in the buildspec I can close this as a bug here. It should only pop up once every 3 years, but having to go update the buildspec and be slightly drifted from the project also isn't ideal. 🤷

To Reproduce It seems just running the codepipeline job will trigger the failure. We are unable to deploy without adding the curl command into the buildspec file.

Expected behavior The codebuild job should not fail due to a yarn gpg key error when running.

Please complete the following information about the solution:

To get the version of the solution, you can look at the description of the created CloudFormation stack. For example, "(SO0089) - customizations-for-aws-control-tower Solution. Version: v1.0.0". You can also find the version from releases

Screenshots If applicable, add screenshots to help explain your problem (please DO NOT include sensitive information).

Additional context Add any other context about the problem here.

mathieujobin commented 1 year ago

Are you still running Bionic 20.04 or already on Focal 22.04 ?

hansarh commented 1 year ago

Any updates on this? We are experiencing the same thing. Running on customizations for control tower 2.5.2 in eu-west-1 This stops us from implementing a change

brettcave commented 1 year ago

This is due to an expired GPG key that affects Ubuntu - https://github.com/yarnpkg/yarn/issues/7866.

I am busy testing a manual modification to the buildspec, by adding - curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -

It would be added in here: https://github.com/aws-solutions/aws-control-tower-customizations/blob/main/customizations-for-aws-control-tower.template#L682 after fi;fi;\n,

so it becomes .... fi; fi\n- curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -\n- - apt-get -q update 1....etc

brettcave commented 1 year ago

The issue will occur in all versions of codebuild/standard images - 4, 5 & 6 (Ubuntu 18, 20 & 22).

I have submitted a PR with a fix for the solution. Alternatively, to manually apply the fix, the following needs to be added in each of the 3 build projects:

      - curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -

Retry the failed pipeline.

balltrev commented 1 year ago

We're aware of an ongoing issue with CfCT where customers are unable to deploy infrastructure due to an expired GPG key for the yarn repository that the CodeBuild image used by CfCT depends on. The CodeBuild service team has been engaged and is addressing the issue. No action is required on your side to apply the fix. We are expecting resolution by end of day Pacific time.

balltrev commented 1 year ago

The CodeBuild service team has released a fix addressing the issue and CfCT functionality is restored.

brettcave commented 1 year ago

Thanks @balltrev