search

aws / aws-codebuild-docker-images

Official AWS CodeBuild repository for managed Docker images http://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref.html
Other
1.13k stars 979 forks source link

Long Provisioning Times on aws/codebuild/amazonlinux2-x86_64-standard:1.0 #296

Closed hounddog22030 closed 4 years ago

hounddog22030 commented 4 years ago

We are experiencing long provisioning times using the image aws/codebuild/amazonlinux2-x86_64-standard:1.0

I opened a support ticket and was told I "that CodeBuild service team confirmed that builds using image "aws/codebuild/amazonlinux2-x86_64-standard:1.0" can cause significant increase in the build time due to an internal change (they are in process of deprecating the image "aws/codebuild/amazonlinux2-x86_64-standard:1.0"). "

I am compiling .netcore2.1 for aws lambda (does not support 3.x yet).

I changed to "aws/codebuild/amazonlinux2-x86_64-standard:2.0" but that does not support the " runtime-versions" in the buildspec (specifically dotnet: 2.2).

It does seem to have .net core 3.x installed, but even with some trials of trying to build and test with "--framework netcoreapp2.1", I seemed to get painted into different corners.

Additionally I was told "One of the most common ways that the provisioning stage of a CodeBuild execution can be sped up is to enable Docker layer caching".

I don't see how using a Docker Layer Cache will speed up the PROVISIONING time and am hestitant to try to make this change as it would require quite a bit of change.

Once the build is "Provisioned" everything is very fast. But some of the Provisioning is taking 3-5 minutes whilst the rest of the phases are cumulative less than 90 seconds.

Can anyone confirmation or advise on the above?

subinataws commented 4 years ago

@hounddog22030 - The issue that you were facing was due to standard:1.0 image no longer being cached on the build fleet for a few hours. When possible, please use the latest version of the CodeBuild provided image1, which are the only ones guaranteed to be cached on the build fleet and lower the provisioning time.

asmontas-i commented 4 years ago

Hi, I haven't used CodePipeline/CodeBuild much, however I'm constantly experiencing long provisioning times (around 5-6mins) with latest images:

Is this caused by some kind of a misconfiguration on my side or this is normal behaviour ?

subinataws commented 4 years ago

@asmontas-i - which compute type and AWS Region did you experience 5-6 mins provisioning delay when using the CodeBuild provided image?

asmontas-i commented 4 years ago

eu-central-1 region and build.general1.small compute type.

However, when switched to use aws/codebuild/standard:3.0 instead of latest standard:4.0, provisioning phase time got significantly reduced from 5-6min to around 30s.

nickfreemandesign commented 2 years ago

We are also experiencing this issue with 3+ minute time spent in buildType PROVISIONING using image: aws/codebuild/standard:6.0 with environment type: Linux (Ubuntu latest), compute: 3 GB memory, 2 vCPUs and region: us-east-1

subinataws commented 2 years ago

@nickfreemandesign - Some of the docker layers for the new standard:6.0 and the latest AL2 image isn't cached yet, leading to the provisioning delays. Team is working on a fix for the regression identified and will roll that forward the fix, after this is tested.

hounddog22030 commented 2 years ago

@nickfreemandesign - Some of the docker layers for the new standard:6.0 and the latest AL2 image isn't cached yet, leading to the provisioning delays. Team is working on a fix for the regression identified and will roll that forward the fix, after this is tested.

When you say "isn't cached yet", I envision that there are physical machines that haven't yet seen/used the "new standard:6.0 and the latest AL2 image". I find that difficult to swallow. If "isn't cached yet" because some scripts/process haven't been run to explicitly bring them down, that I can consider a possibility. Are they "isn't cached yet" because some scripts/process haven't been run yet? It doesn't seem like some software change would need to be deployed in order to make the servers cache a new version of the layers/image.

jakubadamw commented 2 years ago

@nickfreemandesign - Some of the docker layers for the new standard:6.0 and the latest AL2 image isn't cached yet, leading to the provisioning delays. Team is working on a fix for the regression identified and will roll that forward the fix, after this is tested.

Is there an ETA for this, please?

hounddog22030 commented 2 years ago

Holy Cow...

image

NinoMaj commented 2 years ago

@nickfreemandesign - Some of the docker layers for the new standard:6.0 and the latest AL2 image isn't cached yet, leading to the provisioning delays. Team is working on a fix for the regression identified and will roll that forward the fix, after this is tested.

@subinataws any updates on this? We are still seeing long provision times (75 seconds) for ~50% of our builds.

hounddog22030 commented 2 years ago

We switched to GitHub actions. We reduced our build and deployment time from 26 minutes on CodePipeline/CodeBuild to 8 minutes on GitHub . It’s way faster, there’s very little latency between steps and there’s a ton of reusable components. We’re still deploying to AWS.