Unable to import rock into the docker daemon when packing with python plugin and staging python3-dev

[arturo-seijas]

Hi!

I've just noticed skopeo fails to import the resulting rock when packing with the python plugin:

name: indico
summary: Indico rock
description: Indico OCI image for the Indico charm
version: "1.0"
base: ubuntu:22.04
build-base: ubuntu:22.04
license: Apache-2.0
platforms:
  amd64:
parts:
  indico:
    stage-packages:
      - python3-dev
      - python3-venv
    source: plugins
    plugin: python

skopeo --insecure-policy copy oci-archive:indico_1.0_amd64.rock docker-daemon:localhost:32000/indico:latest seems to execute successfully but when I run docker image ls there are no results. However, it works if I change the plugin:

name: indico
summary: Indico rock
description: Indico OCI image for the Indico charm
version: "1.0"
base: ubuntu:22.04
build-base: ubuntu:22.04
license: Apache-2.0
platforms:
  amd64:
parts:
  indico:
    stage-packages:
      - python3-dev
      - python3-venv
    source: plugins
    plugin: nil

After some changes in my rockcraft.yaml to provide a narrower use case, I noticed the following does work with the python plugin, so this likely has to do with the staged packages, either python3-dev or pythone-venv:

name: indico
summary: Indico rock
description: Indico OCI image for the Indico charm
version: "1.0"
base: ubuntu:22.04
build-base: ubuntu:22.04
license: Apache-2.0
platforms:
  amd64:
parts:
  indico:
    stage-packages:
      - build-essential
    source: plugins
    plugin: python

Let me know if there's anything else you'd like me to provide in order to pinpoint the issue.

Thank you

[cjdcordeiro]

Right...this one was tricky. The cause is:

Error processing tar file(duplicates of file paths not supported):

What this means is that, somewhere in the image's layers, there are one (or more) blobs with duplicated content.

Skopeo not detecting that is indeed troublesome, and maybe even worth putting an issue upstream, although one could argue that Skopeo is not to be blamed, since this it a compression/decompression matter, and maybe this Skopeo copy operation doesn't look into the contents of each blob, but just the OCI structure and references.

Having said that, I cannot reproduce your use case to the letter (missing the plugins source), but I was able to get the same error with some other random Python source.

In my case, the following paths are duplicated inside the parts' layer created by Rockcraft:

usr/bin/pip
usr/bin/pip3
usr/bin/pip3.10
usr/bin/python3
usr/bin/python3.10

How can you see this? Well, just do

skopeo --insecure-policy copy oci-archive:indico_1.0_amd64.rock oci:indico
cd indico
# the layer created by Rockcraft is the 2nd one
manifest=`jq -r .manifests[].digest < index.json | cut -d ':' -f 2`
cd blobs/sha256
blob_name=`jq -r .layers[1].digest < $manifest | cut -d ':' -f 2`
tar -tvf $blob_name | awk -F' ' '{print $6}' | sort | uniq -d

---

Now the question is: why are these files being attached twice into the tarball? @tigarmo any idea?

[tigarmo]

The issue here looks like a bad interaction between our usrmerge handling and craft-parts' Python plugin. The plugin creates a virtual env where things like python-packages and the python code get installed; this creates a bin/python3 symlink pointing to /usr/bin/python3.10.

However, in the ubuntu base /bin is a symlink to /usr/bin, so /bin/python3 becomes /usr/bin/python3 which already exists on the base. There's clearly a bug there where rockcraft isn't taking into account the fact that /usr/bin/python3 already exists on the base, but you can see the original well-intentioned handling on the log:

2023-02-14 12:27:12.004 :: 2023-02-14 15:27:06.892 Adding to layer: /root/prime/bin/python3 as 'usr/bin/python3'

The question here is, what's the correct fix for this bug? The two sets of symlinks (the venv one and the one from the main python installation) are incompatible; in one python3.10 points to python3, and in the other it's the opposite. One idea is to have the Python plugin install the venv on a separate directory to bypass the issue altogether, but then we lose the easy and common /bin/python3 access.