Got weird error while referencing from Codebuild source = buildspec.yml

[zoonderkins]

:question: General Issue

The Question

I try to use codebuild with compose docker in docker and deploy to ECR hub.

While using buildSpec: BuildSpec.fromSourceFilename I encountered this error:

If the Project's source is NoSource, you need to provide a concrete buildSpec

xxxProject/node_modules/constructs/lib/construct.ts:407
        throw new Error(`Validation failed with the following errors:\n  ${errorList}`);
              ^
Error: Validation failed with the following errors:
  [Bob-Pipeline-dev] Stage 'Build' must have at least one action

If I use buildSpec: BuildSpec.fromObject, it works @@, no idea

Environment

• CDK CLI Version: 1.32.1
• OS: : OSX Catalina
• Language: Typescript

Sample code with buildspec.yml

File structure:

File: pipeline.ts

    const pipeline = new Pipeline(
      this,
      `Pipeline-${environment.ENV}`,
      {
        pipelineName: `pipeline-${environment.ENV}`,
      },
    )

    const sourceStage = pipeline.addStage({
      stageName: 'Source',
    })

    const buildStage = pipeline.addStage({
      stageName: 'Build',
      placement: {
        justAfter: sourceStage,
      },
    })

    const sourceOutput = new Artifact()

    const sourceAction = new GitHubSourceAction({
      actionName: `codebuild-action-${environment.ENV}`,
      owner: 'xxx,
      repo: 'xxx',
      oauthToken: cdk.SecretValue.secretsManager('tGitHubToken'),
      output: sourceOutput,
      branch: `${environment.branch}`,
      trigger: GitHubTrigger.WEBHOOK,
    })

    sourceStage.addAction(sourceAction)

const buildRole = new iam.Role(
      this,
      `IamRole-${environment.ENV}`,
      {
        assumedBy: new iam.ServicePrincipal('codebuild.amazonaws.com'),
      },
    )

    const codeBuild = new Project(
      this,
      `CodeBuildProject-${environment.ENV}`,
      {
        role: buildRole,
        environment: {
          buildImage: LinuxBuildImage.fromDockerRegistry('docker:dind')
        },
        buildSpec: BuildSpec.fromSourceFilename('./buidspec.yml')

    const buildAction = new CodeBuildAction({
      actionName: 'Build',
      input: sourceOutput,
      project: codeBuild,
    })

    buildStage.addAction(buildAction)
  }

File: buildspec.yml

version: 0.2

phases:
  install:
    runtime-versions:
      nodejs: 12
      python: 3.8
    commands:
      - apk add --no-cache python py-pip jq
      - pip install awscli
      - echo $AWS_ACCESS_KEY_ID
      - echo $AWS_SECRET_ACCESS_KEY

  build:
    commands:
      - $(aws ecr get-login --no-include-email --region ${AWS_REGION_ECR})
      - |
        for dir in `find packages -type d -mindepth 1 -maxdepth 1 -not -name utils -not -name types -exec basename {} \;`;
        do
          docker build -f Dockerfile.${dir} -t ${REPOSITORY_URL}:${dir} .
          docker push ${REPOSITORY_URL}:${dir}
        done

  post_build:
    commands:
      - i=1
      - CURRENT_SERVICE_COUNT=`aws ecs list-services --region ${AWS_REGION_ECS} --cluster ${CLUSTER_NAME} | jq --raw-output ".serviceArns" | jq length`
      - CURRENT_SERVICES=`aws ecs list-services --region ${AWS_REGION_ECS} --cluster ${CLUSTER_NAME} | jq --raw-output ".serviceArns"`
      - |
      while [ "$i" -le "$CURRENT_SERVICE_COUNT" ]; do
        SERVICE=`echo $CURRENT_SERVICES | jq --raw-output ".[$i-1]"`
        aws ecs update-service --region ${AWS_REGION_ECS} --service ${SERVICE} --cluster ${CLUSTER_NAME} --force-new-deployment
        i=$(( i + 1 ))
      done

[skinny85]

Hey @ookangzheng ,

change Project to PipelineProject in your code, and everything will work again.

Thanks, Adam

[zoonderkins]

I got this error when I put buildspec.yml under root directory of my cdk project, I think when I exec cdk synth it doesnt upload my buildspec.yml to S3

2020/04/15 06:14:31 CODEBUILD_SRC_DIR=/codebuild/output/src356100413/src
2020/04/15 06:14:31 Phase complete: DOWNLOAD_SOURCE State: FAILED
2020/04/15 06:14:31 Phase context status code: YAML_FILE_ERROR Message: stat /codebuild/output/src356100413/buildspec.yml: no such file or directory

Screenshot:

[skinny85]

Of course - you put your buildspec in the root of your project, which corresponds to codebuild.BuildSpec. fromSourceFilename('buildspec.yml'), but in your code you have codebuild.BuildSpec. fromSourceFilename('../buildspec.yml').

BTW - buildspec.yml in the root of your project is the default, so just get rid of the buildSpec property completely in PipelineProject.

[zoonderkins]

Nope, this wont work either. Changed code to : codebuild.BuildSpec. fromSourceFilename('buildspec.yml') and place buildspec.yml under cdk root dir, still not working.

How can I make sure when execute cdk synth and cdk deploy will automatic sync buildspec.yml to AWS Codebuild? , cuz under cdk.out/ doesn't seem to have any buildspec.yml or xxx.zip file.

I comment out buildSpec completely in PipelineProject. still not found...

[skinny85]

The BuildSpec path is relative to the root of the project, not relative to the directory the CDK code is in. So, since its in lib in your project, you need codebuild.BuildSpec.fromSourceFilename('lib/buildspec.yml').

[zoonderkins]

Actually I realized that I split CDK and my code into 2 different separate Github Repositories. While I use codebuild.BuildSpec. fromSourceFilename('buildspec.yml'), I have to put buildspec.yml under my code repo.

[skinny85]

Great, glad you got it resolved!

[yash982000]

@ookangzheng @skinny85 In this small world, I have fallen into this error too and glad that I could find one such wonderful thread. I hope you can provide me some guidance since it's been a while I have been stuck on this.

Here is the error snippet:-

I run this command for nonprod synth:- cdk synth --context env_name=nonprod

class ecsCodeBuild(Construct):
    def __init__(  
            self,
            scope: Construct,
            id: str,
            codebuild_project_name,
            **kwargs       
        ):

        super().__init__(
                scope,
                id,
                **kwargs
            )
       git_hub_source = codebuild.Source.git_hub(
        owner="xxx",
        repo="xxx",
        webhook=True,  # optional, default: true if `webhookFilters` were provided, false otherwise
        webhook_triggers_batch_build=True,  # optional, default is false
        webhook_filters=[
        codebuild.FilterGroup.in_event_of(codebuild.EventAction.PUSH).and_branch_is("develop").and_commit_message_is("the commit message")
        ]
    )

        codebuild.GitHubSourceCredentials(
            self, 
            "CodeBuildGitHubCreds",
            access_token=SecretValue.secrets_manager("git-key-mastery-service-cdk-yash")    
    )

        # 3) Environment
        # Build Environment Certificate

        codebuild.Project(
            self, 
            "Project",
            environment=codebuild.BuildEnvironment(
                build_image=codebuild.LinuxBuildImage.from_code_build_image_id("aws/codebuild/standard:5.0"),
            )
        )

        # 4) Buildspec

        codebuild.Project(
            self, 
            "Project",
            source=codebuild.Source.git_hub(
                owner="SavvasLearning",
                repo="rbs-mastery-service"
            ),

            build_spec=codebuild.BuildSpec.from_source_filename('buildspec.yml')
            )

        # 5) Batch Configuration

        # 6) Artifacts
        # bucket: s3.Bucket
        project = codebuild.Project(
            self, 
            "MyProject",
            build_spec=codebuild.BuildSpec.from_source_filename('buildpsec.yml'),
        )

       #7 Logs

The above one is basically my initial basic version of the Code block to run Codebuild through CDK for an AWS CI/CD pipeline for one of the services.

I generally get it from the docs. But, here I am getting the error of NoSource. The CDK docs and code format everything seems to be perfect but not sure what is the problem here. Somewhere I was not sure if Github was properly accessible to that particular repo or not but that seems to be fine. If not, will you please guide me to know the best way to recheck and verify it?

Appreciate it a lot if you can help somehow. Happy to give more inputs as required.

[skinny85]

@yash982000 as the error says, you're missing the build_spec property in

        codebuild.Project(
            self, "Project",
            environment=codebuild.BuildEnvironment(
              build_image=codebuild.LinuxBuildImage.from_code_build_image_id(
               "aws/codebuild/standard:5.0"),
            ),
        )

[yash982000]

@skinny85 I don't understand one thing.

As per the docs, there is no build_spec in the environment class. While I do provide build_spec differently below, how does it differ in this case?

[skinny85]

Not sure what you mean exactly, sorry. The build_spec property should be set on the Project class, as you already do in the other instances of using that class.

[yash982000]

Okay @skinny85 . I was able to resolve the error as per your suggestions. Thanks a lot.

While proceeding with the Artifact part, I have an error like this. I understand I don't have the bucket listed however, I want to store the output of the Artifact to a new s3 bucket. So, what is the right approach to debug it.

        project = codebuild.Project(
            self, "MyProject",
            build_spec=codebuild.BuildSpec.from_object(
            {
                "version": "0.2"
            }),
            artifacts=codebuild.Artifacts.s3(
                bucket=s3.Bucket.from_bucket_name(self, "Bucket", "my-bucket"),
                include_build_id=False,
                package_zip=True,
                path="another/path",
                identifier="AddArtifact1"
            )
        )

Below is the screenshot of error:-

[skinny85]

Right. Python is telling you that s3 is not defined.

[yash982000]

Yes, @skinny85. I was able to do that by importing aws_s3 as s3 Thanks a lot.

I had a blockage from the docs to understand regarding the environment variable in the environment part for code build CDK. How to pass environment variable? Also, do I need to use Build EnvironmentVariable in Build Environment or is there any other way?

I am not able to understand perfectly from the docs library and not able to get more supporting material regarding this.

Also, the same is in the case of VPC reference.

Much help is appreciated.

Thanks.

[skinny85]

Take a look at this example from the CDK examples repository.

[yash982000]

Great @skinny85 . This worked really well. Thanks a lot.

Do we have any such example for VPC?

[skinny85]

Search that repository I've linked to - that's your best bet. You can also Google for stuff about the CDK.

[DimitriosKay]

@skinny85 Great thread, thank you. Could I ask for a little more clarification on the difference between Project and PipelineProject in the context of a CodeBuild deployed and ran under a pipeline ? What does PipelineProject have that Project doesn't ?

[skinny85]

@skinny85 Great thread, thank you. Could I ask for a little more clarification on the difference between Project and PipelineProject in the context of a CodeBuild deployed and ran under a pipeline ? What does PipelineProject have that Project doesn't ?

The only difference is that PipelineProject sets the sourceType property of the Project correctly. That’s the only difference between the two.