search

aws / aws-cdk

The AWS Cloud Development Kit is a framework for defining cloud infrastructure in code
https://aws.amazon.com/cdk
Apache License 2.0
11.69k stars 3.93k forks source link

[aws-codepipeline-actions] EcrSourceAction with ECR as a source does not trigger pipeline change when the source is updated #10901

Closed jpSimkins closed 2 years ago

jpSimkins commented 4 years ago

When creating a code pipeline and using EcrSourceAction, I would expect when the latest tag is updated, for this to update the pipeline that is using this as a source. This is how it works when I create this manually but I cannot get the CDK to do this.

I am thinking this is a bug because this is not what happens when I build the pipeline manually.

Reproduction Steps

I have a pretty large construct to build our pipeline so I will only post the part I think is relevant to the issue.

This is how I define the source action for the pipeline to use the Repository as a source:

new EcrSourceAction({
  actionName: 'BaseImage',
  repository: this._props.projectSourceEcrRepo, // This is an ECR Repository object
  imageTag: 'latest',
  output: this._artifactProjectBaseECR,
}),

Whenever I update the ECR repo, this does not trigger the pipeline to update at all. Basically, adding this as a source is pointless as the entire purpose is to allow this to update the pipeline when this is changed. Otherwise, I can simply pull the image when the project code is update.

To give a better idea of what I am doing:

The only part of the pipeline not working is the ECR source does not trigger the pipeline to build when it is updated.

What did you expect to happen?

I expected that when I update the ECR source, for it to trigger the pipeline to build as this is what it does when I build this manually.

What actually happened?

When the ECR source is updated, nothing happens. No errors that I could find, simply nothing. Like it is not aware of the connection to the repo. For me to trigger a change, I need to update the code to force the pipeline to trigger

Environment

This is :bug: Bug Report

skinny85 commented 4 years ago

Hey @jpSimkins ,

thanks for reporting the issue. Can you tell me whether you see a AWS::Events::Rule resource in your template, and show its content if you do?

Thanks, Adam

jpSimkins commented 4 years ago

Hello @skinny85,

This is the resource with AWS::Events::Rule:

CDKTestBaseRepositoryPipelineTestCDKTestProjectPipeline3CAA437FSourceEventRuleC4D400A3:
    Type: AWS::Events::Rule
    Properties:
      EventPattern:
        source:
          - aws.ecr
        detail-type:
          - AWS API Call via CloudTrail
        detail:
          requestParameters:
            repositoryName:
              - Ref: CDKTestBaseRepository90CAC7C5
            imageTag:
              - latest
          eventName:
            - PutImage
      State: ENABLED
      Targets:
        - Arn:
            Fn::Join:
              - ""
              - - "arn:"
                - Ref: AWS::Partition
                - ":codepipeline:us-east-1:31120948XXXX:"
                - Ref: CDKTestProjectPipeline664B0BF2
          Id: Target0
          RoleArn:
            Fn::GetAtt:
              - CDKTestProjectPipelineEventsRoleEBA1D6BA
              - Arn
    Metadata:
      aws:cdk:path: PipelineTest/CDKTestBase/Repository/PipelineTestCDKTestProjectPipeline3CAA437FSourceEventRule/Resource

I notice that eventName has PutImage. My process is to build the base image, test it, then I have another codeBuild that updates the latest tag once it is approved using the approval action. Just mentioning in case this matters. Still, this works as expected when I set this up manually.

skinny85 commented 4 years ago

My process is to build the base image, test it, then I have another codeBuild that updates the latest tag once it is approved using the approval action

Sorry, can you clarify this part? What do you mean exactly by "updates the latest tag"?

jpSimkins commented 4 years ago

Once the testing is done, given this is a base image it needs to be vetted first, I have a manual approval action that will trigger the updated tag to be added to the image.

Basically, I pull the image, add the latest tag, then push it. I assume this would still trigger PutImage but I am not sure if that is the case as this is an existing image just added the latest tag to it.

The Code may help:

const codebuildProjectUpdateLatestImage = new PipelineProject(this, 'UpdateLatestTag', {
  buildSpec: BuildSpec.fromObject({
    version: '0.2',
    phases: {
      install: {
        'runtime-versions': {
          python: '3.8',
        },
      },
      pre_build: {
        commands: [
          'echo Defining variables...',
          "BUILD_IMAGE=$(cat imageDetail.json | jq '.ImageURI')",
          'echo Removing quotes from string...   ',
          "BUILD_IMAGE=$(sed -e 's/^\"//' -e 's/\"$//' <<< $BUILD_IMAGE)",
          'echo $BUILD_IMAGE',
          "REPOSITORY_URL=$(cut -d':' -f1 <<< $BUILD_IMAGE)",
          'echo $REPOSITORY_URL',
          'echo Logging in to Amazon ECR...',
          "aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $(cut -d'/' -f1 <<< $REPOSITORY_URL)",
        ],
      },
      build: {
        commands: [
          'echo Build started on `date`',
          'echo Pulling the Docker base image...',
          'docker pull $BUILD_IMAGE',
          'echo Tagging the Docker image...',
          'docker tag $BUILD_IMAGE $REPOSITORY_URL:latest',
        ],
      },
      post_build: {
        commands: [
          'echo Pushing the Docker image...',
          'docker push $REPOSITORY_URL:latest',
          'echo Build completed on `date`',
        ],
      },
    },
  }),
  description: 'Updates the latest tag for the ECR Repo',
  environment: {
    buildImage: LinuxBuildImage.AMAZON_LINUX_2_3,
    computeType: ComputeType.SMALL,
    privileged: true, // Needed to run docker build commands
  },
  projectName: this._id + '-UpdateLatestTag',
  role: this._role,
});
// Stage - Build Update Latest Tag
this._codepipelinePipeline.addStage({
  stageName: 'UpdateLatestTag',
  actions: [
    new CodeBuildAction({
      actionName: 'UpdateLatestTag',
      project: codebuildProjectUpdateLatestImage,
      input: this._artifactBaseBuild,
    }),
  ],
});
boraerbas commented 4 years ago

Seems I am also experiencing the same issue, EcrSourceAction is not triggering the build action when the Image is updated.

solovievv commented 3 years ago

I have the same issue, checked CloudWatch events, and found different events. CDK creates an event that doesn't work

CDK created event (it doesn't work):

{
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "source": [
    "aws.ecr"
  ],
  "detail": {
    "requestParameters": {
      "repositoryName": [
        "ecr-repo-name"
      ],
      "imageTag": [
        "latest"
      ]
    },
    "eventName": [
      "PutImage"
    ]
  }
}

Web console created such event (it works well):

{
  "source": [
    "aws.ecr"
  ],
  "detail": {
    "action-type": [
      "PUSH"
    ],
    "image-tag": [
      "latest"
    ],
    "repository-name": [
      "ecr-repo-name"
    ],
    "result": [
      "SUCCESS"
    ]
  },
  "detail-type": [
    "ECR Image Action"
  ]
}

My fix for this issue (python):

        ecr_repo = ecr.Repository.from_repository_name(self, "ecr-repo", settings.backend_ecr_name)
        source_output_ecr = pipeline.Artifact()
        source_action_ecr = actions.EcrSourceAction(
            action_name="ECR",
            repository=ecr_repo,
            image_tag="latest",  # optional, default: 'latest'
            output=source_output_ecr
        )
        rule = events.Rule(
            self, "ecr-rule",
            event_pattern=events.EventPattern(
                source=["aws.ecr"],
                detail={
                    "action-type": [
                        "PUSH"
                    ],
                    "image-tag": [
                        "latest"
                    ],
                    "repository-name": [
                        settings.backend_ecr_name
                    ],
                    "result": [
                        "SUCCESS"
                    ]
                }
            ),
        )
        ......pipeline_backend creation
        rule.add_target(targets.CodePipeline(pipeline_backend))
skinny85 commented 3 years ago

Did that make the CodePipeline trigger @solovievv ?

solovievv commented 3 years ago

@skinny85 Yes, It works well after the fix

jpSimkins commented 3 years ago

I was able to get the base image to trigger the pipeline with Typescript using:

      const eventRule = new Rule(this, 'base-ecr-rule', {
        eventPattern: {
          source: ['aws.ecr'],
          detail: {
            'action-type': ['PUSH'],
            'image-tag': ['latest'],
            'repository-name': [this._props.projectSourceEcrRepo.repositoryName],
            result: ['SUCCESS'],
          },
        },
      });
      eventRule.addTarget(new CodePipelineTarget(this._codepipelinePipeline));

So far, this seems to work as expected.

luca-ferreri commented 3 years ago

same issue here!

However, @solovievv workaround fix it.

luca-ferreri commented 3 years ago

Good news: the @solovievv's workaround is not anymore needed. Did AWS fix something in Cloudwatch?

skinny85 commented 3 years ago

Interesting @luca-ferreri... it definitely seemed like this problem was happening only for some customers (I was never able to reproduce myself, for example).

luchees commented 3 years ago

I am experiencing exactly the same issue in typescript. I'm using cdk 107.0. After the fix of @jpSimkins I was able to resolve it.

EventRule generated by the EcrSourceAction that does not trigger the pipeline

"GraphQlGwEcrservergraphqlgwpipelinedevGraphqlApiECSPipelineD471B6D0SourceEventRuleF2E828D6": {
      "Type": "AWS::Events::Rule",
      "Properties": {
        "EventPattern": {
          "source": [
            "aws.ecr"
          ],
          "detail-type": [
            "AWS API Call via CloudTrail"
          ],
          "detail": {
            "requestParameters": {
              "repositoryName": [
                "server-dev-graphql-gw-ecr"
              ],
              "imageTag": [
                "latest"
              ]
            },
            "eventName": [
              "PutImage"
            ]
          }
        },
        "State": "ENABLED",
        "Targets": [
          {
            "Arn": {
              "Fn::Join": [
                "",
                [
                  "arn:",
                  {
                    "Ref": "AWS::Partition"
                  },
                  ":codepipeline:ap-southeast-1:",
                  {
                    "Ref": "AWS::AccountId"
                  },
                  ":",
                  {
                    "Ref": "GraphqlApiECSPipeline4C249FA2"
                  }
                ]
              ]
            },
            "Id": "Target0",
            "RoleArn": {
              "Fn::GetAtt": [
                "GraphqlApiECSPipelineEventsRole0098D078",
                "Arn"
              ]
            }
          }
        ]
      },
      "Metadata": {
        "aws:cdk:path": "server-graphql-gw-pipeline-dev/GraphQlGwEcr/servergraphqlgwpipelinedevGraphqlApiECSPipelineD471B6D0SourceEventRule/Resource"
      }
    }

EventRule Created by the workaround of @jpSimkins that does trigger the pipeline

"GraphqlGwPutImageRule34EA2A1C": {
      "Type": "AWS::Events::Rule",
      "Properties": {
        "EventPattern": {
          "source": [
            "aws.ecr"
          ],
          "detail": {
            "action-type": [
              "PUSH"
            ],
            "image-tag": [
              "latest"
            ],
            "repository-name": [
              "server-dev-graphql-gw-ecr"
            ],
            "result": [
              "SUCCESS"
            ]
          }
        },
        "State": "ENABLED",
        "Targets": [
          {
            "Arn": {
              "Fn::Join": [
                "",
                [
                  "arn:",
                  {
                    "Ref": "AWS::Partition"
                  },
                  ":codepipeline:ap-southeast-1:",
                  {
                    "Ref": "AWS::AccountId"
                  },
                  ":",
                  {
                    "Ref": "GraphqlApiECSPipeline4C249FA2"
                  }
                ]
              ]
            },
            "Id": "Target0",
            "RoleArn": {
              "Fn::GetAtt": [
                "GraphqlApiECSPipelineEventsRole0098D078",
                "Arn"
              ]
            }
          }
        ]
      },
      "Metadata": {
        "aws:cdk:path": "server-graphql-gw-pipeline-dev/GraphqlGwPutImageRule/Resource"
      }
    },
fasatrix commented 3 years ago

We have exactly the same problem however the workaround suggested @jpSimkins is no working for us. It does create the rule as indicated however the pipeline is not triggered. We have a cross account configuration where ECR is in AWS Account A and Codepipeline in Account B. After deploying CDK initially the image is sourced correctly from our ECR and deployed without any problems. However, any subsequent changes made to the image there after, fails to get deployed. Any idea to what could go wrong? Could it be something related to the Events bus on different accounts (Rule is listening for changes on Account B bus which never gets the Event, whereas ECR creates events on Account A's bus), Is anyone ( maybe @skinny85 ? ) clear about how this should work cross account? Our cross account role should work fine as otherwise would not be able to source the image at all (first deployment works).

We have even tried to create a Rule that includes the account and the region as additional parameters. As you can see below the CloudFormation is created properly however still not triggering codePipeline

  const eventRule = new Rule(this, 'base-ecr-rule', {
      ruleName: `CodePipelineTriggerImageChanges-${id}`,

      eventPattern: {
        source: ['aws.ecr'],
        account: [props.registryId],
        region: ['ap-southeast-2'],
        detail: {
          'action-type': ['PUSH'],
          'image-tag': [this.imageTag],
          'repository-name': [this.ecrImageRepoName],
          result: ['SUCCESS'],
        },
      },
    });

    eventRule.addTarget(new CodePipeline(this.pipeline));
{
  "source": [
    "aws.ecr"
  ],
  "detail": {
    "repository-name": [
      "MyRepo"
    ],
    "result": [
      "SUCCESS"
    ],
    "action-type": [
      "PUSH"
    ],
    "image-tag": [
      "latest"
    ]
  },
  "region": [
    "ap-southeast-2"
  ],
  "account": [
    "AccountA"
  ]
}
skinny85 commented 3 years ago

@fasatrix make sure you have CloudTrail configured to deliver these events.

fasatrix commented 3 years ago

@fasatrix make sure you have CloudTrail configured to deliver these events.

@skinny85 thanks for the prompt answer.. Do you mean configure CloudTrail to send events from Account A to CloudWatch Logs still in Account A or would I have to send those events from Account A to B like this doc

So basically what I am asking here is, will a rule created using EcrSourceAction be able to listen to events generated in a different account (giving one has the right permissions granted via cross account role) , say Account A (with a conf similar to the one mentioned above -- add account and region to the rule) or those events must to be shipped/shared across Accounts too? Thanks again

skinny85 commented 3 years ago

The EcrSourceAction should take care of cross-account Events for you - it should create a separate Stack that grants account A (where the repository is) permissions to publish events to account B (where the pipeline is).

It should be a stack with a single resource, AWS::Events::EventBusPolicy.

fasatrix commented 3 years ago

The EcrSourceAction should take care of cross-account Events for you - it should create a separate Stack that grants account A (where the repository is) permissions to publish events to account B (where the pipeline is).

It should be a stack with a single resource, AWS::Events::EventBusPolicy.

ok that explains. I cannot find this AWS::Events::EventBusPolicy in my yaml output.. is this a bug and how can we troubleshoot it? CDK version we use 1.106

skinny85 commented 3 years ago

Can you show me how do you set up your EcrSourceAction in your code?

fasatrix commented 3 years ago

Can you show me how do you set up your EcrSourceAction in your code?

It is part of the following function..

    const createImageAction = (
      imageRepoName: string,
      imageTag?: string,
    ): CreatePipelineActionResponse<Artifact, EcrSourceAction> => {
      const imageArtifact = new Artifact(`ImageArtifact-${id}`);
      const imageRepository = Repository.fromRepositoryArn(
        this,
        'ImageRepository',
        `arn:aws:ecr:ap-southeast-2:<MyAWSAccount>:repository/${imageRepoName}`,
      );
      //latest will be taken as the default image tag
      const imageSourceAction = new EcrSourceAction({
        actionName: `ImageSource-${id}`,
        repository: imageRepository,
        imageTag: imageTag ?? 'latest',
        output: imageArtifact,
        role: Role.fromRoleArn(this, 'CentralEcrAccessRole', props.crossAccountEcrRole),
      });
      return { artifact: imageArtifact, action: imageSourceAction };
    };
skinny85 commented 3 years ago

Does the Role you use here, with the ARN from props.crossAccountEcrRole, is in the same account as the ECR Repository, with the ARN arn:aws:ecr:ap-southeast-2:<MyAWSAccount>:repository/${imageRepoName}?

fasatrix commented 3 years ago

crossAccountEcrRole

correct, they are from the same account (which is the account that host ECR)

fasatrix commented 3 years ago

The EcrSourceAction should take care of cross-account Events for you - it should create a separate Stack that grants account A (where the repository is) permissions to publish events to account B (where the pipeline is). It should be a stack with a single resource, AWS::Events::EventBusPolicy.

ok that explains. I cannot find this AWS::Events::EventBusPolicy in my yaml output.. is this a bug and how can we troubleshoot it? CDK version we use 1.106

@skinny85 it is consistent on our side! Our yaml does not include AWS::Events::EventBusPolicy out of the box and if we do create a custom policy like the following, everything works just fine. 

    new CfnEventBusPolicy(this, `EventBusPolicy-${id}`, {
      statementId: `EventBusPolicy-${id}`,
      statement: {
        Effect: 'Allow',
        Action: 'events:PutEvents',
        Resource: "arn:aws:events:ap-southeast-2:<AccounB>:event-bus/default"
        Principal: {
          AWS: `arn:aws:iam::<AccountA>:root`,
        },
      },
    });

Does it look like it could be a bug to you?

skinny85 commented 3 years ago

Yes, it's possible it is. Let me try and reproduce to confirm.

skinny85 commented 3 years ago

I think I understand the problem now. It's actually a different thing than this issue talks about. Let's move the conversation to a dedicated issue I've created: https://github.com/aws/aws-cdk/issues/16245.

fasatrix commented 3 years ago

Thanks for you support @skinny85. With your guidance I was able to understand how this lib works..

github-actions[bot] commented 2 years ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.