Use Nix as build system

[katexochen]

Our current build with make and docker are not only chaotic, but also aren't reproducible.

Nix is a build system that enables reproducible builds, which is a hard requirement if we want to do meaningful remote attestation.

Nix can build

• Software packages
• Container images
• OS images

in a reproducible way.

I propose we use Nix as a build system. A transition to Nix would need to be done incrementally.

This also doesn't have to be an all-or-nothing decision. Nix can do a lot of things. I think using it as dev environment alone to provide tools needed for development would be a big benefit.

Considered alternatives

Major alternative to Nix to consider is Bazel. We have been using Bazel for quite some time at Edgeless, but recently added Nix in addition, as Bazel could not fulfill all the requirements. Especially, Bazel requires you to also convert transitive dependencies to be built with Bazel. In Nix, we can use nixpkgs for our transitive dependencies, which is a huge and up to date package collection. As CAA has many similarities to Constellation regarding builds of binaries, containers and OS images, I think we would hit the same limitations. There are other systems like buck2, but those are rather niche ecosystems.

[mkulke]

What would be the impact on build time w/ nix? At the moment Docker-bases builds take considerable time, esp for the rust-components that we have to build, since target folders cannot be shared.

[katexochen]

In general, Nix builds aren't as fine granular, compared to Bazel builds for example. Nix doesn't share a cache between different cargo builds (but there is upstream work on this). However, it is input-addressed, so you only rebuild packages that changed and their dependents. For our use case, fast/cached builds within a single binary aren't that important. We rather need to ensure we only rebuild a binary when necessary, and that is done by nix through only rebuilding a package when its inputs change. And while Bazel has the fancier caching, it comes at a high cost, as you basically have to declare every single input and output file in Starlark, whereas you can just invoke cargo/make/cmake/whatever in Nix.

So build time would for sure be reduced (by large, I assume), as we update the source of the binaries rarely.

In other scenarios where we might not want to build within nix right now, but with the help of nix, nix could enable us to use a local cache while also using pinned toolchains. For example, if we use nix develop with mkosi. And of cause, we wouldn't need to build some things at all, as parts of our dependencies are already packaged in nixpkgs and can be substituted from a binary cache.

Does that answer your question @mkulke?

[mkulke]

Yup, I understand that we can expect some speedup over building with docker-containers, but in any case don't expect any regression 👍

[mkulke]

two questions that came to my mind:

In my limited experience with nix, when building binaries that are dynamically linked it's not possible to deploy them to non-nix system, as nix uses its own dynamic loader. I used patchelf to replace it with the system's linux-ld to work around it, but that's probably not the proper way. How would we handle this in our scenario?

In some cases we might have exotic build dependencies e.g. for tdx attestation. How would be handle that, if the CPU vendor only provides ubuntu packages, would it be on us to maintain a nix package for those?

[katexochen]

when building binaries that are dynamically linked it's not possible to deploy them to non-nix system, as nix uses its own dynamic loader. I used patchelf to replace it with the system's linux-ld to work around it, but that's probably not the proper way. How would we handle this in our scenario?

So first, I'd say we could start doing static builds again. If I remember correctly, the case for dynamic linking was that security patches could be applied easier. As soon as the root FS is read-only and protected by dm-verity, there is no way to update any libraries, so we can also just do static builds.

If we want to run dynamic binaries built with Nix in the fedora image, we would copy the full closure (all nix paths that are needed during runtime) into the image, and I think that would include the loader that is hard coded in the binary? I need to re-check this.

In some cases we might have exotic build dependencies e.g. for tdx attestation. How would be handle that, if the CPU vendor only provides ubuntu packages, would it be on us to maintain a nix package for those?

Yes, we would need to package these. I would upstream these dependencies to nixpkgs and hope that other people use and maintain them, too. There are already SGX related things packaged in nixpkgs. I know this is an initial investment, but I don't think that it will be hard to maintain over time.

[mkulke]

So first, I'd say we could start doing static builds again.

I think the rationale against static musl builds was not necessarily about receiving security updates during runtime. There was pragmatic reason, since having static binaries w/ musl and rust -sys (like tss-esapi) packages that are themselves linked against system libs e.g. openssl turned out to be tricky. The other security-related rationale would be that at least during build-time we'd ship the latest fixes, which we wouldn't necessarily if we ended up vendoring openssl.

we would copy the full closure

That would probably work, but it feels odd to me that we have to leak build-system specifics into the artifact. The patchelf (patchelf --set-interpreter /usr/lib64/ld-linux-x86-64.so.2 target/release/bla) hack generally seems to works, but I suspect there is some nix-toggle to perform this change (I did not find anything in a casual search, though).

Yes, we would need to package these

For az we might get around vendor-specific dependencies on the PodVM for the time being, but I'm not sure. Those libraries are not known having a nice dx and we might require some of those in the future or if plan to support other CSPs. That's a commitment we have to be aware of and consciously decide to accept.

[katexochen]

That would probably work, but it feels odd to me that we have to leak build-system specifics into the artifact.

Well, I wouldn't see it that way. Nix is many things. It's rather using nix-the-package-manager in the image than leaking nix-the-build-system into it. :wink: You wouldn't mind traces of dnf in the image, would you?

[katexochen]

Btw, this is also how Nix builds containers:

After the new layer has been created, its closure [...] will be copied in the layer itself.

[mkulke]

It's rather using nix-the-package-manager in the image than leaking nix-the-build-system into it. 😉 You wouldn't mind traces of dnf in the image, would you?

looking at it from this perspective the behaviour makes sense. but in this we're opting for nix not merely as a build system, but also as a package manager. that might or might not be ok, not sure, we probably want to discuss this eplicitly in the sync.

the upside would reproducible builds, which is something that we probably want once we get further along with the measurement of the podvm disk

[katexochen]

Maybe the title of this issue including "build system" isn't ideal. I thought of a bit more, including developer environments and toolchains.

Just to emphasize this again, I don't think this must be an all-or-nothing decision. To get reproducible builds, we could go top-down, pinning next lower layer that is not yet reproducible. For example, given we don't have reproducible binaries yet, we can build reproducible OS images given a reproducible mkosi toolchain and a pinned set of binaries (e.g. from a CAS). We can then work on the next stage, like making binary builds reproducible.

[bpradipt]

@katexochen @mkulke I want to highlight few things here and also ask a few questions

1. Official libvirt library only provides cgo bindings and hence we have a hard requirement to support dynamically linked binaries for cloud-api-adaptor. Ref: https://github.com/confidential-containers/cloud-api-adaptor/issues/119 Also imho, it might not be practical to have a hard requirement on static builds given the fast changing nature of the CoCo project resulting in new components getting introduced.

2. Is there a documented set of challenges with reproducible builds when using docker and make build system ? It'll be a good starting point for discussion on which problem and persona (developer, user, downstream providers etc) to prioritise.

[katexochen]

Thanks for the feedback @bpradipt :)

1. Official libvirt library only provides cgo bindings and hence we have a hard requirement to support dynamically linked binaries for cloud-api-adaptor.

So just to clarify a little: Nix is totally capable of building dynamic binaries. However, you will usually link against the libraries provided by nix, not the one of your host (or target system). There are ways around this (like copying the nix libraries, the so-called closure to the target, or patching the binary after build, as Magnus described).

My suggestion was to build the binaries in the podVM statically, not CAA. And as said, there are many ways to use nix. If we copy the closure anyway, we don't have to care about building statically...

2. Is there a documented set of challenges with reproducible builds when using docker and make build system ?

I'm not exactly sure what you would like to document. It is not as if make+docker would be a build system comparable to nix that is just lacking some features one could implement. It lacks basically everything that would characterizes a build system: foremost dependency management and proper sandboxing.

Regarding make: In the C/C++ sense, make is a recipe that describes which commands to execute to build an artifact. It does neither provide the toolchain (gcc) nor the libraries you want to link against. You are responsible for managing these. And nix is a way to do exactly this. Nix is at a higher level and executes make (or meson, ...) in case it is the way to build a project. It provides the sandboxing, dependencies, toolchains and also the source code.

Using docker for build provide you with a way to declare dependencies in some way (apt install hello), but there is no way to pin these inputs in a reproducible way. It uses chroot, but that's all about the sandboxing. Nix and bazel do not allow network access in the sandbox.

[katexochen]

I would like to discuss this more in depth and maybe show some examples and code, but I feel this would break the frame of our weekly meeting. Maybe we can setup an extra slot for it?

[bpradipt]

I would like to discuss this more in depth and maybe show some examples and code, but I feel this would break the frame of our weekly meeting. Maybe we can setup an extra slot for it?

Great suggestion. We can setup a separate one to discuss this topic. Let's finalise it in today's syncup.

[bpradipt]

Thanks for the feedback @bpradipt :)

1. Official libvirt library only provides cgo bindings and hence we have a hard requirement to support dynamically linked binaries for cloud-api-adaptor.

So just to clarify a little: Nix is totally capable of building dynamic binaries. However, you will usually link against the libraries provided by nix, not the one of your host (or target system). There are ways around this (like copying the nix libraries, the so-called closure to the target, or patching the binary after build, as Magnus described).

Got it

My suggestion was to build the binaries in the podVM statically, not CAA. And as said, there are many ways to use nix. If we copy the closure anyway, we don't have to care about building statically...

2. Is there a documented set of challenges with reproducible builds when using docker and make build system ?

I'm not exactly sure what you would like to document. It is not as if make+docker would be a build system comparable to nix that is just lacking some features one could implement. It lacks basically everything that would characterizes a build system: foremost dependency management and proper sandboxing.

Regarding make: In the C/C++ sense, make is a recipe that describes which commands to execute to build an artifact. It does neither provide the toolchain (gcc) nor the libraries you want to link against. You are responsible for managing these. And nix is a way to do exactly this. Nix is at a higher level and executes make (or meson, ...) in case it is the way to build a project. It provides the sandboxing, dependencies, toolchains and also the source code.

If we pin the base docker image by using it's sha256 hash, pin the code commits, tool chain versions, then what prevents us from having a reproducible build? This is the part I was hoping can be documented as a baseline list of challenges.

Using docker for build provide you with a way to declare dependencies in some way (apt install hello), but there is no way to pin these inputs in a reproducible way. It uses chroot, but that's all about the sandboxing. Nix and bazel do not allow network access in the sandbox.

Will installing a specific version of the dep solve the reproducibility issue with docker builds (eg apt-get install package=version) ?

We can discuss these questions in the call.

[katexochen]

Will installing a specific version of the dep solve the reproducibility issue with docker builds (eg apt-get install package=version)?

Not from my perspective. I would say reproducible builds have one simple building block: an artifact that is pinned by hash. Yes, you could pin the base image, and that is reproducible. Installing a package with a specific version (apt-get install package=version) is not reproducible, as it does not pin the hash of what you install. I don't say it is impossible to archive this with docker (as I've already done it), but you need to build a system around it to archive it (like we already started with versions.yaml), and at that point you are just rebuilding Nix/Bazel (and that is not a good idea, obviously).

If we pin the base docker image by using it's sha256 hash, pin the code commits, tool chain versions, then what prevents us from having a reproducible build? This is the part I was hoping can be documented as a baseline list of challenges.

So here is a really quick and incomplete list my perspective on the requirements:

• Build results SHOULD be bit-by-bit reproducible [^1]
• All inputs MUST be pinned by hash
• Management of pinned inputs SHOULD scale well
• Builds MUST happen in a strict sandbox (stricter than the default Bazel sandbox) [^2]
• ...[I'll add more points as they come to my mind]

An important point that is somewhere in between the lines is the correct handling of metadata: timestamps, randomness etc are usually a source of non-reproducibility. Depending on the type of artifact you are building (binary, container image, OS image), these can be introduced through many different things: the build environment, your toolchain, the way things are packaged or how your package manager works.

[^1]: Notice that not all build outputs in nixpkgs are bit-by-bit reproducible (yet). However, nixpkgs provides a collection of input-addressed packages, which is closes as you get regarding reproducibility of a full system today. Your own binaries could be build bit-by-bit reproducible with nix nevertheless. [^2]: Notice that this is only strictly required in case of input-addressed build systems, but for practical reasons you'll likely want it anyway to discover impurities.