ECR assets: referencing outside of the CDK

[eladb]

Originally posted by @caruso-billfire in https://github.com/aws/aws-cdk/pull/5733#issuecomment-578246447

How would you reference the image outside of the CDK? Is that use case supposed to even be supported? What would the best practices for that be?

[eladb]

@caruso-billfire can you please provide a bit more details about your use case for consuming CDK docker assets outside the CDK? Generally, we design assets to be consumable from CDK constructs.

If you are looking for a way to publish docker images to ECR, you should be able to do this with any supporting CI/CD tool.

[eladb]

@EduardTheThird wrote:

Previously when creating an ECS service and using the ECR asset functionality, an ECR repo was created for that service. Having the service have it's own ECR repo allows us to easier manage it as it was linked to a service. With the shared repository, tags such as latest, qa, dev now conflict.

Ideally when using ECS.ContainerImage.fromAsset to create the Docker image, we would like to specify what the ECR repo and tag it should create for it.

The central ECR repo could be retained, however, we would appreciate the option to retain the old or similar workflow.

Reproduction Steps Call ContainerImage.fromAsset to build local Docker image for the service, a central ECR repo is created, aws-cdk/assets, not cdk/servicename.

Ideally, we would like to be able to specify what ECR repo and image tag it should create.

[iippis]

Even in the case of using LinuxBuildImage.fromAsset and LinuxBuildImage.fromEcrRepository for a better decoupling this is complicated - it's easy to come up with a tag or just go with latest but hard to come up with the hash.

[EduardTheThird]

Hi @eladb Thank you for the prompt reply 😸

I'm a huge fan of the CDK and we have been using it since your first very early versions last year. Keep up the good work 👍

As you mentioned in https://github.com/aws/aws-cdk/issues/5976, the intent of assets might be different from our use-case, which I will try to explain as best as possible.

When deploying new services to our AWS environments, we copy the application's build artifact into a folder within our AWS CDK TypeScript solution. This application's artifact contains a Dockerfile and is configured for the environment.

The workflow:

1. Get published artifacts from the application's build pipeline. No release pipeline or external ECR repository exists at this stage.
2. Copy artifact to an artifacts folder within the AWS CDK TypeScript solution.
3. The service is configured in TypeScript for the AWS CDK:
4. The application's Docker image is built as part of the CDK deployment to our account:
5. Once the AWS CDK deployment is complete, the service is running in a steady state. The initial seed is now complete.
6. We now use this repository created for the service, by the CDK and incorporate it into our release pipeline for the application. The service now gets deployed via the CI/CD pipeline and uses the repository created by the CDK.

Caveats and possible improvements on the process:

1. Ideally, we would like to be able to specify the name and tag of the created repository. Rather than having cdk/< some lengthy name >< guid > we would prefer just < servicename >.
2. The container image after CDK deployment is marked with the latest tag, but the service task definition uses the SHA of the pushed image. Ideally, we would be able to specify the tag the task definition uses.
3. Using a shared repository, as introduced in the latest version of the CDK breaks this process, as the latest, dev, qa or build ID that might be used as tags by the release pipelines, could possibly not be unique for each service in the shared ECR assets repository.

Should you need more clarification on any of this, or more examples, I'd be more than happy to assist in any way possible. If it was possible for us to create, name and assign ECR repositories to ECS EC2 and ECS Fargate Services, and seed them with a locally built docker image, it would simplify and cleanup our workflow greatly! 🚀 🎉🎉

[eladb]

@EduardTheThird I must say that I still can't fully understand your use case. What do you mean by:

We now use this repository created for the service, by the CDK and incorporate it into our release pipeline for the application. The service now gets deployed via the CI/CD pipeline and uses the repository created by the CDK.

Can you elaborate?

I feel this is where you guys are doing something that's outside of what we considered initially.

[EduardTheThird]

Certainly, let me elaborate on our use-case for the repositories. 😃

Let us consider service "sometestservice" as our example. After the CDK has deployed the ECS service, a repository, cdk/sometestservice4b11af is created (Custom::ECRAdoptedRepository resource).

We continue to use cdk/sometestservice4b11af as the service's main repository for the environment. On each new release, a new image is pushed to it and the service updated.

I've created an example pipeline which hopefully might illustrate its place in our release pipeline:

The AWS CDK is only used to seed the service to the new environment, releases are then done in Azure DevOps.

[eladb]

The AWS CDK is only used to seed the service to the new environment, releases are then done in Azure DevOps.

In that case, I would argue that you don't need to use docker image assets at all. Just define an ECR repository (with or without an explicit physical name) and use ContainerImage.fromEcrRepository:

Sketch:

const repo = new ecr.Repository(this, 'MyRepo', { repositoryName: 'repository-for-my-service' });
const image = ContainerImage.fromEcrRepository(repo);

Then, have your CI/CD pipeline push to repository-for-my-service and you are golden.

What am I missing?

[EduardTheThird]

That is almost exactly what we need, the only missing piece of the puzzle is the option to reference an image that's constructed directly from sources on disk.

ContainerImage.fromAsset is able to reference our artifact folder in the solution:

ContainerImage.fromEcrRepository has no directory option, if it could be added, we will be golden :

Note: This was tested on AWS CDK 1.21.1

[eladb]

@EduardTheThird wrote:

reference an image that's constructed directly from sources on disk.

Something still doesn’t add up for me... you mentioned that at runtime you actually want to reference the image pushed to ECR from your CI/CD pipeline, not the one built from disk.

[EduardTheThird]

Aah, let me clarify.

Let us consider service "sometestservice" again as our example.

After the AWS CDK has deployed the ECS service, a repository, cdk/sometestservice4b11af with an image cdk/sometestservice4b11af@sha:1234353543 is created. This image is built from disk, by the AWS CDK.

At this stage, inside our newly created ECS service's task definition, cdk/sometestservice4b11af@sha:1234353543 is used.

This is where the AWS CDK stops and the manual labor begins 😄 Via the AWS Console, we now manually update the task definition of the ECS service to use cdk/sometestservice4b11af:latest and update the ECS service to use the new version of the task definition that references cdk/sometestservice4b11af:latest.

In Azure DevOps, we now update our CI/CD pipeline for "sometestservice". On the release pipeline we now have it push newer docker images that are built on our build server, to the cdk/sometestservice4b11af ECR repository, using the latest tag (cdk/sometestservice4b11af:latest). Once the image is pushed, the ECS service is updated via the AWS CLI and the deployment complete.

We continue to use cdk/sometestservice4b11af as the service's main repository for the environment. On each new release, a new image is pushed to it and the service updated.

We could potentially bypass many of the manual processes by being able to use ContainerImage.fromEcrRepository, should it be able to build images from disk. 🥇

[eladb]

@EduardTheThird Why does the first image come from disk and the rest come from the CI/CD pipeline? What makes this first image special and what value to you get from actually consuming it as an asset ("from disk")?

Another question: why not let the CDK always build & push the image to ECR? If you invoke cdk deploy from your CI/CD pipeline, it should simply build the docker image, push it ECR and wire it to your stack. It will create a healthy coupling between your infra code and your app code and ensure that the image you consume is always aligned with the image in your source repo.

[EduardTheThird]

Excellent question, I've noticed that Cloudformation would sometimes get stuck if the newly created ECS service does not pass health checks. It seems to wait for the service to reach a steady state. Which it never will as it is referencing an empty newly created repository.

The first image is only needed to allow the service to pass health checks, reach a steady-state and allow Cloudformation to complete.

Regarding the coupling of the infrastructure code and app code in our CI/CD pipeline, I love the idea, it is certainly something that we will aim to implement as soon as we worked out all the kinks 😺

[eladb]

I am closing this for now. Please feel free to reopen if you wish to continue the discussion or provide more use cases.

[djheru]

@EduardTheThird Why does the first image come from disk and the rest come from the CI/CD pipeline? What makes this first image special and what value to you get from actually consuming it as an asset ("from disk")?

Another question: why not let the CDK always build & push the image to ECR? If you invoke cdk deploy from your CI/CD pipeline, it should simply build the docker image, push it ECR and wire it to your stack. It will create a healthy coupling between your infra code and your app code and ensure that the image you consume is always aligned with the image in your source repo.

I am trying to accomplish something very similar. When the stack is first deployed, I would like to pass the taskImageOptions an image created using ContainerImage.fromAsset and local assets. I have a pipeline set up as part of the stack, where pushes to CodeCommit will trigger a build from CodeBuild and then an ECS Deployment. I also have a pipeline for CDK itself, to deploy CDK changes. My goal is to have the cdk deploy only use the ContainerImage.fromAsset on the initial deployment (so the stack will pass health checks and deploy successfully) but then after that, have the pipeline CodeBuild/ECS Deploy build the new images and push them up to ECR, then deploy to ECS, rather than allowing the CDK to control it via CloudFormation rolling updates to the task definition. Is this possible?

[GhyslainBruno]

Hi all, First of all, thanks for your awesome work, your guys rock !

@djheru I am currently trying to achieve the exact same thing you want to do, and I assume there's something I'm not getting here as this seems a pretty common need.

@eladb I actually tried to implement it the way you mentioned (as it seemed logic to me) but the thing is that when doing it, as @EduardTheThird noted, the service created doesn't pass the health checks anymore, because of the ECS part (taskDefinition.addContainer)

I mean, we are actually saying to the newly created ECS to use an image from a repository newly created, which doesn't contains any image for now, so it just waits there... (That's my understanding, I'm really new to IaC and even more to CDK, so please, don't be too rough on me if I say dumb things).

@eladb I think I will do this by doing exactly what you suggested here, but I'm kind of anxious there, because I can't think about it as a production ready workflow (I mean, destroying/recreating the whole infrastructure at every git push... Plus, it's quite a long process to do it every time in a CI/CD pipeline.

Anyway, I'm kinda stuck in here, so any help would be awesome (idk, almost a year since last comment, maybe someone found a way :-D).

[apptimise]

The AWS CDK is only used to seed the service to the new environment, releases are then done in Azure DevOps.

In that case, I would argue that you don't need to use docker image assets at all. Just define an ECR repository (with or without an explicit physical name) and use ContainerImage.fromEcrRepository:

Sketch:

const repo = new ecr.Repository(this, 'MyRepo', { repositoryName: 'repository-for-my-service' });
const image = ContainerImage.fromEcrRepository(repo);

Then, have your CI/CD pipeline push to repository-for-my-service and you are golden.

What am I missing?

Thanks for the info, @eladb, but AWS advise against using hardcoded names and we have faced many problems updating the stack when repositoryName is specified. If we let AWS to generate a name, how do we push the image in our deployment scripts? especially when there are multiple environments. Thanks!