search

aws-solutions / aws-control-tower-customizations

The Customizations for AWS Control Tower solution combines AWS Control Tower and other highly-available, trusted AWS services to help customers more quickly set up a secure, multi-account AWS environment using AWS best practices.
https://docs.aws.amazon.com/controltower/latest/userguide/cfct-overview.html
Apache License 2.0
356 stars 205 forks source link

Pipeline broken due to Kernel Version #92

Closed mohamed-caylent closed 2 years ago

mohamed-caylent commented 2 years ago

Describe the bug The pipeline is current broken at the build stage due to a Kernel Version error:

Unpacking libc6-dev:amd64 (2.27-3ubuntu1.5) over (2.27-3ubuntu1.4) ...
--
250 | Preparing to unpack .../libc-dev-bin_2.27-3ubuntu1.5_amd64.deb ...
251 | Unpacking libc-dev-bin (2.27-3ubuntu1.5) over (2.27-3ubuntu1.4) ...
252 | Preparing to unpack .../linux-libc-dev_4.15.0-169.177_amd64.deb ...
253 | Unpacking linux-libc-dev:amd64 (4.15.0-169.177) over (4.15.0-151.157) ...
254 | Preparing to unpack .../libc6_2.27-3ubuntu1.5_amd64.deb ...
255 | ERROR: Your kernel version indicates a revision number
256 | of 255 or greater.  Glibc has a number of built in
257 | assumptions that this revision number is less than 255.
258 | If you\'ve built your own kernel, please make sure that any
259 | custom version numbers are appended to the upstream
260 | kernel number with a dash or some other delimiter.
261 |  
262 | dpkg: error processing archive /var/cache/apt/archives/libc6_2.27-3ubuntu1.5_amd64.deb (--unpack):
263 | new libc6:amd64 package pre-installation script subprocess returned error exit status 1
264 | Errors were encountered while processing:
265 | /var/cache/apt/archives/libc6_2.27-3ubuntu1.5_amd64.deb
266 | E: Sub-process /usr/bin/dpkg returned an error code (1)
267 |  
268 | [Container] 2022/03/01 22:45:46 Command did not exit successfully apt-get update && apt-get upgrade -y exit status 100
269 | [Container] 2022/03/01 22:45:46 Phase complete: INSTALL State: FAILED
270 | [Container] 2022/03/01 22:45:46 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: apt-get update && apt-get upgrade -y. Reason: exit status 100

To Reproduce Just run the pipeline using the defaults that AWS providers.

Expected behavior The dependencies step should at least pass no problem

Please complete the following information about the solution:

Screenshots N/A

Additional context I tried changing the image used in the Codebuild to: aws/codebuild/standard:5.0 and also a non-latest version of aws/codebuild/standard:4.0 and no luck.

mohamed-caylent commented 2 years ago

It seems that some of the lib6* packages were upgraded today and they are causing the pipeline to fail. As a temporary workaround, you can add this line - sudo apt-mark hold libc6* before - apt-get update && apt-get upgrade -y in the buildspec.yml file in the Codebuild Build stage.

Edit: As mentioned by @WarrEagle below, you will need to add this one line to the build stage, the SCP stage and the Cloudformation stage for the entire pipeline to work.

jakemraz commented 2 years ago

check this :) https://github.com/aws-solutions/aws-control-tower-customizations/pull/93 it works!

mbeacom commented 2 years ago

Related upstream issue: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1962225

WarrEagle commented 2 years ago

It seems that some of the lib6* packages were upgraded today and they are causing the pipeline to fail. As a temporary workaround, you can add this line - sudo apt-mark hold libc6* before - apt-get update && apt-get upgrade -y in the buildspec.yml file in the Codebuild Build stage.

This affects stages 2,3, and 4 in the same manner. This one-liner added to each stages buildspec.yaml allowed my orgs deployments to complete successfully for our automatic account provisioning triggers.

stumins commented 2 years ago

We are aware of this issue and are working with the AWS CodeBuild team to resolve the issue upstream. AWS CodeBuild is actively working on a fix and we do not expect needing to patch CFCT as part of the resolution.

As mentioned above, not upgrading libc6 at runtime by adding sudo apt-mark hold libc6* before apt-get upgrade in buildspec stages should work if you require an immediate workaround.

We will provide additional details as we receive more information. Thank you for your patience.

mohamed-caylent commented 2 years ago

For those of you following this thread, I started getting errors like /codebuild/output/tmp/script.sh: 4: /codebuild/output/tmp/script.sh: sudo: not found with the workaround so make sure to remove the sudo in the beginning of the command and it will work fine.

In other words, add this line - apt-mark hold libc6* before - apt-get update && apt-get upgrade -y in the buildspec.yml for all stages Build, SCP and Cloudformation.

stumins commented 2 years ago

The CodeBuild team has released a fix that should resolve this. Please reopen this issue if you continue to experience this problem.

mohamed-caylent commented 2 years ago

Confirming that the issue was fixed. Thank you!