search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
18.02k stars 14.03k forks source link

Create vcpkg builder #330859

Open Guekka opened 3 months ago

Guekka commented 3 months ago

vcpkg is a C++ package manager. We recently created a Nix package for it, but it can only be used in Nix shells and not for packages.

There were some discussions recently about creating a builder. Let's centralize the information here

Guekka commented 3 months ago

@gracicot: Do you know how well this solution fits with any future plans to build projects with manifests and external repositories as nix derivations? If you're unsure I'll do some research of my own, but I was hoping you had some more insider knowledge :)

Originally posted by @h7x4 in https://github.com/NixOS/nixpkgs/issues/329939#issuecomment-2254300611

Guekka commented 3 months ago

@h7x4 sadly my only "insider knowledge" I have is the few comments I got from their lead developer, and the few conversations I participated in their repo.

For projects that have a manifest, I guess we could try building derivations from libraries packaged by vcpkg using vcpkg ports, then making cmake aware of the install prefix. I don't think this patch impede with this future. On the contrary, I think it's gonna help.

We could also try to follow what cargo does. I'm not exactly sure how it works, but I'm sure there's a way to make it just as convenient.

Originally posted by @gracicot in https://github.com/NixOS/nixpkgs/issues/329939#issuecomment-2254306724

Guekka commented 3 months ago

I think the most viable way for now would be the rely on Fixed Output Derivation

h7x4 commented 3 months ago

I have been experimenting with this for a little while now. Based on what I found, I think I would categorize the different builds into these 3 groups, ordered by difficulty.

  1. Classic mode project without a baseline

This is relatively straight forward. Using the already packages vcpkg instance with cli flags --downloads-root or --x-asset-sources, we can store the inputs needed to build offline in a FOD. Depending on how old the software is, it should go smooth. It does not need a .git directory. If it does not build, you can override vcpkg's revision to an older version (and open an issue upstream).

  1. Classic mode project with a baseline / Manifest mode project with no external registries

Not as straight forward. You will need the .git directory to check out different revisions for different ports. For context, .git is not deterministic. #328937 and #329939 are attempts to make this work.

Another thing to note is that you will need to bring the modified vcpkg instance with you to the build derivation or at least be able to check out the baseline in the offline sandbox. If the baseline is incorrect during buildtime, vcpkg will try to pull versions of the dependencies that you do not have stored in the FOD with assets/downloads. If we are going to store modified vcpkg instances in FODs, we'll need to make sure there are no /nix/store references here. Maybe disable patchShebangs, or at least add a config flag to the package so we can have non-patched versions for FOD use in the builder?

As a side note, I have gotten a classic project with baseline to build by overriding the nixpkgs-package's revision + hash, and and patching the project source to remove the baseline. This effectively transforms it into a classic mode project without a baseline.

  1. Manifest project with external registries

These are projects which pull resources for vpckg to use from multiple sources named "registries". We might want to create a FOD for each registry to let the user patch them individually. Each registry will itself depend on its git history. Just like the previous group, both the vcpkg instance and also the registries with correctly configured baselines will needed in the offline build sandbox.

There are also things like external registries with toolchain files that needs patching, but most of that can probably be handled on a case by case basis for now.

There also something inbetween here, where you can add a single vcpkg overlay to a classic project, mimicking the behaviour of a manifest project with a single external registry. Not entirely sure how to fit that puzzle piece into the picture, but you can at least download its correct revision as a FOD and point to it with the environment variable VCPKG_OVERLAY_PORTS.