(pipelines): FileSets rendering of `cdk.out` artifacts conflicts with artifacts codebuild partial

[jeffb4]

Describe the bug

The pipelines module in the CodePipeline construct, when setup to be self-mutating, takes any primary_output_directory property defined for the synth ShellStep as well as any additional output directories added, and renders into a codebuild artifacts: partial section. If the construct user adds an artifacts: partial to codebuild options to the CodePipeline constructor (code_build_defaults or synth_code_build_defaults) then it is rendered with 2 artifacts: sections in the buildspec defined for the CodeBuild resource. This results in a jsii synth error of jsii.errors.JSIIError: Error: Only one build spec is allowed to specify artifacts.

Example result

artifacts:
  secondary-artifacts:
    dist:
      base-directory: dist
      files:
        - "**/*"
artifacts:
    base-directory: cdk.out
    files: 
      **/*

Expected Behavior

I would expect some sort of hash merge to occur between FileSets defined for the ShellStep (whether default added cdk.out directory or something custom from the library user) as well as any artifacts: section in any codebuild partials in play.

Current Behavior

Currently two artifacts: hashes exist in the same rendered codebuild partial and jsii is choking on it.

Reproduction Steps

synth_code_build_defaults = pipelines.CodeBuildOptions(
            partial_build_spec=codebuild.BuildSpec.from_object({"artifacts":{
             "secondary-artifacts":{"dist":{"base-directory":"dist","files":["**/*"]} } } })
          )
synth_shell_step = pipelines.ShellStep(
            "Synth",
            input=pipelines.CodePipelineSource.connection(redacted),
            install_commands=["./install.sh"],
            commands=["./build.sh"]
        )
pipeline=pipelines.CodePipeline(
            self,
            "Pipeline",
            pipeline_name="test-Pipeline",
            synth=synth_shell_step,
            synth_code_build_defaults=synth_code_build_defaults,
        )
pipeline.build_pipeline()

and synth should fail with jsii.errors.JSIIError: Error: Only one build spec is allowed to specify artifacts.

Possible Solution

I tried to dig into how pipelines/codepipeline.ts is handling artifacts (and how that relates to aws-codepipeline/pipeline.ts and really wasn't able to.

Additional Information/Context

No response

CDK CLI Version

2.43.0 (build 487870a)

Framework Version

No response

Node.js Version

v14.17.0

OS

Linux

Language

Python

Language Version

Python 3.8.5

Other information

No response

[rix0rrr]

If I'm interpreting the report correctly, it's:

• I want to add artifacts to each and every CodeBuild step in my pipeline, and CDK Pipelines is not letting me.

The error Only one build spec is allowed to specify artifacts. is correct, this is currently not possible. I'm not entirely clear where or why you are seeing a YAML file with two artifacts: sections: we should never have generated anything of the sort, and I don't understand the code path that would generate that. But also, it doesn't really matter, because as I said the error is right.

I'm struggling to understand the use case behind this request. What are you trying to do?

[github-actions[bot]]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[jeffb4]

I have a buildspec.yaml for CI builds that defines artifacts. It would be most ergonomic for me to read in that buildspec.yaml and define (for my synth step) an artifacts partial that exactly matches the buildspec file, instead of converting the artifacts section into FileSets and the like (which will then get rendered as artifacts). I cannot do so, because the cdk.out artifacts expected from the synth stage conflict with the partial. I think, behind the scenes, FileSets are rendered into an artifacts section and appended to the in-memory build spec for the step, and aren't being merged in in a data structure way, but it's very difficult for me to glean what is exactly going on.

[jeffb4]

In the alternate, there is no documentation that adding artifacts to the buildspec partial will irretrievably fail, and the jsii error is very unhelpful, so this limitation should be documented (but it should not be a limitation)

[rix0rrr]

Ah, yes. The issue is that buildspec.yaml interferes with pipelines in an annoying way. CDK really wants to hide things like names of artifacts for you (because they need to be the same in buildspec.yaml and in the pipeline definition, but they are not interesting at all), but it cannot do so because CDK does not control the buildspec.yaml.

The only thing I would say really needs to be in a buildspec.yaml (because it is tied to the source, and not the rest of the infrastructure) is build commands -- and those could be put into a ./my-build.sh file as well, and invoking that script from a pipeline-side BuildSpec.

[jeffb4]

I understand what you're saying. We have a buildspec.yaml that is used for CI branch builds, and while I'm reading it in my python/cdk code, there's no expectation that it would necessarily play well with BuildSpec. I would like to add an artifacts partial and have it be properly merged as a map with the artifacts section generated internally by FileSets. Partials and FileSets and the like are all being rendered together to generate a CloudFormation-internal buildspec, so this feels like it should be possible to do in the jsii code.

[bestickley]

I'd like to be able to set "enable-symlinks": "yes" to the synth build spec like below but I cannot because of the same error. +1 on this issue.

     ...
     synthCodeBuildDefaults: {
        partialBuildSpec: BuildSpec.fromObject({
           artifacts: {
             "enable-symlinks": "yes",
           },
         }),
      },

I cannot event .addPropertyOverride() because source.buildSpec is a token :/