Guidance on how to use AWS ECR as private registry

[aad]

Is your feature request related to a problem? Please describe. I can not find the document or use case how to use AWS ECR as private registry.

Describe the solution you'd like we can run aws ecr get-login-password to get the authorization token and config the registry like below to deploy cf. But the token is only valid for 12 hours.

system_registry:
  add_image_pull_secrets: true
  hostname: https://${ecr_hostname}
  username: AWS
  password: ${ecr_token}

doc mentioning the expiration: https://docs.aws.amazon.com/cli/latest/reference/ecr/get-login-password.html

Describe alternatives you've considered It would be useful to suggest how to use ecr as private registry or is it an option?

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/175894512

The labels on this github issue will be updated when the story is started.

[amalagaura]

Related to #345

[jamespollard8]

Thanks @aad, we don't yet have cf-for-k8s documented or explored fully on entirely AWS but we'll work to prioritize this as part of the epic captured by #345

marked as a Feature Request

cc my pair @acosta11

[aad]

Thanks @jamespollard8, we have switched our app_registry from ecr to dockerhub since we confirmed its a known issue that we can not push image to ecr without creating the image (called repository in ecr) in advance.

to simulate the case when we cf push which will build a uuid image

eval $(aws ecr get-login --no-include-email)
ecr_hostname="$(aws sts get-caller-identity --query 'Account' --output text).dkr.ecr.${region}.amazonaws.com"
aws ecr create-repository --repository-name test-dynamic-repo
docker pull busybox
new_image="test-dynamic-repo/$(uuidgen)"
docker tag busybox ${ecr_hostname}/${new_image,,}
docker push ${ecr_hostname}/${new_image,,} # expect `name unknown: The repository with name 'test-dynamic-repo/dc41b118-f518-4c02-a53e-15e95b575850' does not exist in the registry`

[jamespollard8]

We provided instructions and an example structure for an ECR app_registry config here: https://github.com/cloudfoundry/cf-for-k8s/blob/07f7898ae1bcd6a2dccc3ed0181474c16741bf8a/sample-cf-install-values.yml#L108-L115

[dishant-openxcell]

what does it mean by # ensure ${ecr_hostname}/my_env/cf-default-builder exists where can I bring the cf-default-builder image from?

I am facing this issue here, I have configured all the necessary access policies, and created a repo named cloudfoundry, and using the following configuration in cf-values.yml

cat << EOF >> ${TMP_DIR}/cf-values.yml
app_registry:
   hostname: "https://$(aws sts get-caller-identity --query 'Account' --output text).dkr.ecr.${region}.amazonaws.com"
   repository_prefix: "$(aws sts get-caller-identity --query 'Account' --output text).dkr.ecr.${region}.amazonaws.com/cloudfoundry"
   username: AWS
   password: "$(aws ecr get-login-password)"

remove_resource_requirements: true
add_metrics_server_components: true
allow_prometheus_metrics_access: true
use_external_dns_for_wildcard: true
enable_automount_service_account_token: true
metrics_server_prefer_internal_kubelet_address: true
use_first_party_jwt_tokens: true

load_balancer:
  enable: true
EOF

Its giving me this error:

fail: reconcile builder/cf-default-builder (kpack.io/v1alpha1) namespace: cf-workloads-staging
6:45:55PM:  ^ Encountered failure condition Ready == False:  (message: HEAD https://473798018071.dkr.ecr.ap-northeast-1.amazonaws.com/v2/cloudfoundry/cf-default-builder/blobs/sha256:3646b32e05b2ff078dc82ba83427077212b237be6abcfc8a71ec6412dc21c79e: unsupported status code 400)

it was tring to get cloudfoundry/cf-default-builder so I created another repo with name cloudfoundry/cf-default-builder in ecr, but error is still the same.

[aad]

@dishant-openxcell, i didn't encounter the error you have.

FYI, the last test we did (it was few month ago, against v1.0), ecr was not suitable for app_registry since its doesn't support dynamic repository. Which is required as all cf app will have unique image ids and we couldn't pre-recreate the ecr repositories beforehand.

ecr as system_registry is fine.