using aws-ecr/build_image results in multiple images pushed, some of which appear invalid

johnmayock commented 2 months ago

Orb version

v9.3.2

What happened

I am moving from version 8.1.3 to 9.3.2 (to use a later executor image - aws-ecr/default). After making the necessary changes to the pipelines to account for new(er) required properties, the pipeline completed successfully. This is the step which is successful.

aws-ecr/ecr_login: account_id: <> region: <>
aws-ecr/build_image: path: <> build_path: <> account_id: <> region: <> repo: <> extra_build_args: <> tag: ${COMPONENT_VERSION} push_image: true

However, whereas with 8.1.3, only one image is pushed to ECR as I expect, 9.3.2 (and 9.3.1 tested) push the built image but they also appear to taint the ecr repo with other images with the image name of "-".

In this image, there are 3 images added following the execution of aws-ecr/build_image.

One of the images has the same size as the legitimate image, whereas one has a reported size of zero.

When you inspect the image with zero size, via the AWS mngt console, it returns LayerError.

Expected behavior

Only one image should be pushed to the ECR repo by aws-ecr/build_image.

I am of the view (I have searched for documentation) that this should not be expected behaviour, certainly compared to prior version of 8.1.3, it is not seen.

I would welcome feedback if this is expected or not. Thank you.

marboledacci commented 2 months ago

The problem seems to be the builder, this is using a builder with docker-container endpoint set here I was testing using docker default build and it works fine. I'm investigating if there's a reason to use docker-container driver, otherwise will update to docker driver.

marboledacci commented 2 months ago

@johnmayock After some research, I found this is probably a bug with ECR visualization for multi arch images. The docker-container driver is designed to support cache and multi arch images, and when it is used the images on ECR will be shown on this way, in theory your images would be working fine even if they are showed like this, but ECR will show it in a confusing way. You can find more information in this aws issue.

Please verify if this behavior is actually the same bug on ECR and you can keep working normally with the images, or if there's something else that should be done on orb side.

johnmayock commented 2 months ago

Hello @marboledacci, thank you for that link. I did try to find references to this prior to raising but clearly, I didn't do a good enough job on it. I have read it through fully.There is common debate regarding how confusing the representation is.

There is suggestion in the thread that setting provenance to false, disabled buildx from generating a build attestation which is stored in the OCI image metadata.

https://docs.docker.com/build/metadata/attestations/slsa-provenance/

However, I have no grounds to demand this change from the ORB. This issue does seem to be a consequence of the interpretation within the ECR UI of dockers mutli-arch manifest format and not a defect on the orb itself.

Please feel free to close this issue and thank you so much for your investigation and support.

marboledacci commented 2 months ago

We could add the provenance param as a default, or a new parameter to defined when wanted. But based on the thread it is not a full solution, and not always works so it is not a priority now.

I'm closing the issue here as it is not a problem with the orb, and will add the revision of provenance arg to our backlog.

CircleCI-Public / aws-ecr-orb