awslabs / aws-deployment-framework

The AWS Deployment Framework (ADF) is an extensive and flexible framework to manage and deploy resources across multiple AWS accounts and regions based on AWS Organizations.
Apache License 2.0
662 stars 226 forks source link

Unable to use custom Docker Image in every stage of the pipelines #382

Closed stemons closed 1 year ago

stemons commented 3 years ago

Hello,

we are trying to use a custom Docker Image in every stage of the pipelines, but it seems that the image is used only in the first build phase using this configuration, while the other stages (plan and apply) use the standard image (STANDARD_5)

pipelines:
  - name: sample-s3-terraform
    default_providers:
      source:
        provider: codecommit
        properties:
          account_id: "111111111111" 
      build:
        provider: codebuild
        properties:
          environment_variables:
            TERRAFORM_VERSION: "1.0.0" # terraform version
          image: 
            repository_arn: arn:aws:ecr:eu-west-1:111111111111:repository/iac
            tag: "0.0.2"
      deploy:
        provider: codebuild
        properties:
          environment_variables:
            # TARGET_ACCOUNTS: 111111111111,222222222222 # target accounts
            TARGET_OUS: /exceptions/sandbox # target OUs
            MASTER_ACCOUNT_ID: 111111111111 # master account
            REGIONS: eu-west-1 # target regions
    params:
      restart_execution_on_update: true
    targets:
      - name: terraform-plan
        properties:
          spec_filename: tf_plan.yml # terraform plan
      - approval # manual approval
      - name: terraform-apply
        properties:
          spec_filename: tf_apply.yml # terraform apply

We tried to use also this configuration by specifying the repository_arn properties in every stage of the pipelines, but in this case we get an error.

pipelines:
  - name: sample-s3-terraform
    default_providers:
      source:
        provider: codecommit
        properties:
          account_id: "111111111111"
      build:
        provider: codebuild
        properties:
          environment_variables:
            TERRAFORM_VERSION: "1.0.0" # terraform version
          image: 
            repository_arn: arn:aws:ecr:eu-west-1:111111111111:repository/iac
            tag: "0.0.2"
      deploy:
        provider: codebuild
        properties:
          environment_variables:
            # TARGET_ACCOUNTS: 111111111111,222222222222 # target accounts
            TARGET_OUS: /exceptions/sandbox # target OUs
            MASTER_ACCOUNT_ID: 111111111111 # master account
            REGIONS: eu-west-1 # target regions
    params:
      restart_execution_on_update: true
    targets:
      - name: terraform-plan
        properties:
          spec_filename: tf_plan.yml # terraform plan
          image: 
            repository_arn: arn:aws:ecr:eu-west-1:111111111111:repository/iac
            tag: "0.0.2"
      - approval # manual approval
      - name: terraform-apply
        properties:
          spec_filename: tf_apply.yml # terraform apply
          image: 
            repository_arn: arn:aws:ecr:eu-west-1:111111111111:repository/iac
            tag: "0.0.2"
[Container] 2021/08/23 09:18:58 Running command cdk synth --app adf-build/cdk/generate_pipeline_stacks.py 1> /dev/null
2021-08-23 09:19:00,524 | INFO | __main__ | ADF Version 3.1.2 | (generate_pipeline_stacks.py:25)
2021-08-23 09:19:00,524 | INFO | __main__ | ADF Log Level is INFO | (generate_pipeline_stacks.py:26)
2021-08-23 09:19:00,529 | INFO | cdk_stacks.main | Pipeline creation/update of sample-s3-terraform commenced | (main.py:31)
jsii.errors.JavaScriptError: 
  Error: There is already a Construct with name 'custom_repo' in Stack [sample-s3-terraform]
      at Node.addChild (/tmp/jsii-kernel-bpFzVV/node_modules/constructs/lib/construct.js:529:19)
      at new Node (/tmp/jsii-kernel-bpFzVV/node_modules/constructs/lib/construct.js:38:28)
      at new ConstructNode (/tmp/jsii-kernel-bpFzVV/node_modules/@aws-cdk/core/lib/construct-compat.js:166:28)
      at Object.createNode (/tmp/jsii-kernel-bpFzVV/node_modules/@aws-cdk/core/lib/construct-compat.js:37:42)
      at new Construct (/tmp/jsii-kernel-bpFzVV/node_modules/constructs/lib/construct.js:576:32)
      at new Construct (/tmp/jsii-kernel-bpFzVV/node_modules/@aws-cdk/core/lib/construct-compat.js:35:9)
      at new Resource (/tmp/jsii-kernel-bpFzVV/node_modules/@aws-cdk/core/lib/resource.js:22:9)
      at new RepositoryBase (/tmp/jsii-kernel-bpFzVV/node_modules/@aws-cdk/aws-ecr/lib/repository.js:16:1)
      at new Import (/tmp/jsii-kernel-bpFzVV/node_modules/@aws-cdk/aws-ecr/lib/repository.js:231:17)
      at Function.fromRepositoryArn (/tmp/jsii-kernel-bpFzVV/node_modules/@aws-cdk/aws-ecr/lib/repository.js:240:16)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "adf-build/cdk/generate_pipeline_stacks.py", line 37, in <module>
    main()
  File "adf-build/cdk/generate_pipeline_stacks.py", line 33, in main
    PipelineStack(app, stack_input)
  File "/codebuild/output/src721917786/src/adf-build/jsii/_runtime.py", line 83, in __call__
    inst = super().__call__(*args, **kwargs)
  File "/codebuild/output/src721917786/src/adf-build/cdk/cdk_stacks/main.py", line 101, in __init__
    adf_codebuild.CodeBuild(
  File "/codebuild/output/src721917786/src/adf-build/jsii/_runtime.py", line 83, in __call__
    inst = super().__call__(*args, **kwargs)
  File "/codebuild/output/src721917786/src/adf-build/cdk/cdk_constructs/adf_codebuild.py", line 40, in __init__
    build_image=CodeBuild.determine_build_image(scope, target, map_params),
  File "/codebuild/output/src721917786/src/adf-build/cdk/cdk_constructs/adf_codebuild.py", line 177, in determine_build_image
    repo_arn = _ecr.Repository.from_repository_arn(
  File "/codebuild/output/src721917786/src/adf-build/aws_cdk/aws_ecr/__init__.py", line 1910, in from_repository_arn
    return jsii.sinvoke(cls, "fromRepositoryArn", [scope, id, repository_arn])
  File "/codebuild/output/src721917786/src/adf-build/jsii/_kernel/__init__.py", line 126, in wrapped
    return _recursize_dereference(kernel, fn(kernel, *args, **kwargs))
  File "/codebuild/output/src721917786/src/adf-build/jsii/_kernel/__init__.py", line 349, in sinvoke
    response = self.provider.sinvoke(
  File "/codebuild/output/src721917786/src/adf-build/jsii/_kernel/providers/process.py", line 371, in sinvoke
    return self._process.send(request, InvokeResponse)
  File "/codebuild/output/src721917786/src/adf-build/jsii/_kernel/providers/process.py", line 335, in send
    raise JSIIError(resp.error) from JavaScriptError(resp.stack)
jsii.errors.JSIIError: There is already a Construct with name 'custom_repo' in Stack [sample-s3-terraform]
Subprocess exited with error 1

[Container] 2021/08/23 09:19:00 Command did not exit successfully cdk synth --app adf-build/cdk/generate_pipeline_stacks.py 1> /dev/null exit status 1
[Container] 2021/08/23 09:19:00 Phase complete: BUILD State: FAILED
[Container] 2021/08/23 09:19:00 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: cdk synth --app adf-build/cdk/generate_pipeline_stacks.py 1> /dev/null. Reason: exit status 1
[Container] 2021/08/23 09:19:00 Entering phase POST_BUILD
[Container] 2021/08/23 09:19:00 Running command python adf-build/cdk/clean_pipelines.py
stemons commented 2 years ago

Hello team, any chance to add this in the next release?

sbkok commented 2 years ago

Hi @stemons, I will look into the effort required to assess if this could be added as part of the next release. I will keep you posted here.

sbkok commented 1 year ago

Thank you for your patience. I am happy to inform you that this issue has been resolved in our latest release v3.2.0 just now. I'm hereby closing this issue. Please open a new issue if you are experiencing any issues with the latest release.