Manylinux public container removal

[AdamYoblick]

In release.yml we reference a public container for manylinux builds.

image: quay.io/pypa/manylinux2014_x86_64:latest

This violates the recent company security policies. We have to figure out a way to deal with this.

[AdamYoblick]

@zooba @karthiknadig @int19h Do you have any ideas for how we can get around this?

[karthiknadig]

@AdamYoblick We can create a internal registry using Azure Container Registry and upload a custom built image for this based on the pypa spec for the container. Here is where we can find the definition: https://github.com/pypa/manylinux

[int19h]

The reason why we want to build in a container is because that's the only reliable way to ensure that Linux wheels don't end up with dependencies on the wrong glibc version. This container is the official "blessed" one from PyPA that runs an old enough version of RedHat to produce binaries that run on anything newer.

But as I understand it, the problem here isn't the use of a container per se, but rather the fact that we don't own it? In which case we can just fork the container spec, review it, and build it ourselves.

[int19h]

Side note, it doesn't have to be manylinux2014 specifically - we can take something newer, it all depends on which glibc versions (= which distros) we want to support. There's nothing in either debugpy or pydevd that requires anything newer than manylinux1, so the only time I update our build scripts to point to a newer container is when Azure build agent & tasks stop working on the older one. But we can always rev it for other reasons, if the newer version is more convenient in some other way.

[AdamYoblick]

@AdamYoblick We can create a internal registry using Azure Container Registry and upload a custom built image for this based on the pypa spec for the container. Here is where we can find the definition: pypa/manylinux

Thank you! I don't know anything about this, but it looks like that link does have the instructions on how to generate the docker images from the source. I'm not sure if that is sufficient to satisfy the requirement, but it might not be since the source is public and can be an attack vector. So maybe we have to generate it from scratch based on the spec, like you said :)

[zooba]

Unfortunately, there are other requirements about using up to date OS, so we're not going to win there.

Best shot might be to get as close as we can with a regular build and run a validation step in the container to make sure we aren't relying on anything too new.

[int19h]

@zooba The problem is that when you link to glibc, the symbols get versioned. So e.g. if you use printf, your binary's import table will have printf@GLIBC.x.y.z, and will refuse to run on anything lower. I'm not sure if anything changed in this department recently, but last I looked into it, the official answer was still "build against the lowest version of glibc that you intend to support" (either in container, or in chroot which is not fundamentally different in this case).

zig cc can link to specific glibc versions on any platform (even when cross-compiling from Windows), but they jump through a lot of hoops to make it all work. But if that is a feasible option, it could greatly simplify our build, I think.

[AdamYoblick]

Unfortunately, there are other requirements about using up to date OS, so we're not going to win there.

Best shot might be to get as close as we can with a regular build and run a validation step in the container to make sure we aren't relying on anything too new.

Ah, you're saying that using a too-old OS in the image is itself a violation? Hmm, sounds like a catch-22 if you want to support manylinux. Maybe we need to find other teams at MS that produce manylinux wheels and see what they are doing.

[zooba]

Corporate compliance requirements are inspired by catch-22 😆

If we can figure a way to link against early glibc without actually running an OS based around that, it'll be easier to get the exception we need to run tests in the actual container. Otherwise, we need to justify getting an exception to build production artifacts in an obsolete OS, which is a much bigger ask (and the first question will be whether we can do what I suggested earlier, so let's run that one out before we go straight to begging forgiveness).