search

cnabio / cnab-to-oci

Tool to convert CNAB bundle.json to OCI index
Apache License 2.0
54 stars 41 forks source link

Image copy fails when image contains duplicate layers #128

Closed carolynvs closed 2 years ago

carolynvs commented 2 years ago

Some images have duplicate layers, for example docker/whalesay:

$ docker image inspect docker/whalesay
...
"RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:1154ba695078d29ea6c4e1adb55c463959cd77509adf09710e2315827d66271a",
                "sha256:528c8710fd95f61d40b8bb8a549fa8dfa737d9b9c7c7b2ae55f745c972dddacd",
                "sha256:37ee47034d9b78f10f0c5ce3a25e6b6e58997fcadaf5f896c603a10c5f35fb31",
                "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
                "sha256:b26122d57afa5c4a2dc8db3f986410805bc8792af3a4fa73cfde5eed0a8e5b6d",
                "sha256:091abc5148e4d32cecb5522067509d7ffc1e8ac272ff75d2775138639a6c50ca",
                "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
                "sha256:d511ed9e12e17ab4bfc3e80ed7ce86d4aac82769b42f42b753a338ed9b8a566d",
                "sha256:d061ee1340ecc8d03ca25e6ca7f7502275f558764c1ab46bd1f37854c74c5b3f",
                "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
            ]
        },

Note that 5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef is in there multiple times. This can happen when a RUN command doesn't modify the file system, such as running COPY + RUN chmod vs COPY --chmod.

We are simultaneously kicking off copies for each layer, and if there are duplicates, then a race begins over which copy will run successfully and which will notice that it didn't write the expected number of bytes (because the layer already exists).

This causes the following error in Porter (see https://github.com/getporter/porter/issues/2320):

error preparing the bundle with cnab-to-oci before pushing: failed commit on ref "layer-sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4": unexpected size 64, expected 32.

We should only copy the unique set of layers in an image, and not kick off extra copies for duplicate layers.