Support immutable vcpkg root and standard unix filesystem structure

[gracicot]

Is your feature request related to a problem? Please describe.

Right now vcpkg assumes that /opt or /usr/bin or wherever vcpkg is installed to be mutable. This is not a problem when you install it manually in /opt/vcpkg and set the permissions to your user, but this is a problem when packaging vcpkg.

This is especially a problem when trying to create a nix package for vcpkg, since the nix store is immutable. It's also difficult to install files in a standard unix structure with directories like bin/ and share/

Proposed solution

What I would like: be able to install the vcpkg executable under $prefix/bin/vcpkg and the data, port, version and everything to $prefix/share/vcpkg/. vcpkg should be able to pickup those paths when no VCPKG_ROOT is defined, from the path of the executable.

Some of the requirements that would need to support this feature:

• The vcpkg executable should be able to take filesystem lock under /var/run/<hash-of-absolute-vcpkg-path>/.vcpkg-root which is common practice for package manager, and a known mutable location.
• The vcpkg executable should have a configurable installed/, buildtrees/ and packages/ locations, or assume a known mutable place like ~/.vcpkg/<hash-of-absolute-vcpkg-path>/ for those directories

Of course, <hash-of-absolute-vcpkg-path> can be anything that uniquely identify this particular installation of vcpkg.

Describe alternatives you've considered

The considered alternative for me was to package vcpkg with a wrapper script for the executable in path that takes a hash of the absolute path and create a mutable directory under ~/.vcpkg/<hash-of-absolute-vcpkg-path>/ and simlink everything from inside the VCPKG_ROOT, and set the root there instead. I found this alternative brittle. It should be supported natively by vcpkg.

Proof of concept nix file with this workaround:

{
  stdenv,
  fetchFromGitHub,
  gcc,
  ninja,
  cmake,
  lib,
  git,
  cacert,
  runtimeShell
}:

stdenv.mkDerivation rec {
  pname = "vcpkg";
  version = "b18b17865cfb6bd24620a00f30691be6775abb96";

  src = fetchFromGitHub {
    owner = "microsoft";
    repo = "vcpkg-tool";
    rev = "2021-12-09";
    sha256 = "0027fbm6vd82qiddg92wslabsi2dvpfxgyin67fna97xz2hnzr9l";
  };

  vcpkg_src = fetchFromGitHub {
    owner = "microsoft";
    repo = "vcpkg";
    rev = "b18b17865cfb6bd24620a00f30691be6775abb96";
    sha256 = "1yfb1vhfgz7mgl6iwvq6qyrpzc1m95jr821pk1d36j5bj1n31qhh";
  };

  buildInputs = [cmake ninja git cacert];

  vcpkgScript = ''
    #!${runtimeShell}
    vcpkg_hash=$(echo -n "${placeholder "out"}" | sha256sum | cut -f1 -d ' ')
    vcpkg_root_path="$HOME/.vcpkg/roots/$vcpkg_hash"
    if [[ ! -d $vcpkg_root_path ]]; then
      mkdir -p $vcpkg_root_path
      ln -s ${placeholder "out"}/share/vcpkg/{docs,ports,scripts,triplets,versions} $vcpkg_root_path/
    fi
    VCPKG_DOWNLOADS=$vcpkg_root_path/downloads VCPKG_ROOT=$vcpkg_root_path ${placeholder "out"}/share/vcpkg/vcpkg "$@"
  '';

  passAsFile = [ "vcpkgScript" ];
  installPhase = ''
    cmake --build . --target=install --config=Release
    mkdir -p $out/share/vcpkg
    cp --preserve=mode -r ${vcpkg_src}/{docs,ports,scripts,triplets,.vcpkg-root,versions} $out/share/vcpkg
    mv $out/bin/vcpkg $out/share/vcpkg/vcpkg
    cp $vcpkgScriptPath $out/bin/vcpkg
    chmod +x $out/bin/vcpkg
    touch $out/share/vcpkg/vcpkg.disable-metrics
  '';
  enableParallelBuilding = true;
}

save under vcpkg.nix and run it using nix-shell -p 'with (import <nixpkgs> {}); callPackage .vcpkg.nix {}' --pure.

However with proper support for that, VCPKG_ROOT could be left to $out/share/vcpkg and the wrapper script would simply point to other directory for buildtrees and all without making simlinks. Or even better, vcpkg could use the home directory by default like other package managers for build trees, cache and user-wide packages.

Additional context

https://discourse.nixos.org/t/installing-vcpkg-on-macos-using-nix/16852/8 https://discourse.nixos.org/t/known-mutable-location-in-filesystem-for-lock-files/17337/2

[gracicot]

@JackBoosY should I open a discussion in the discussion board?

[JackBoosY]

@gracicot Just keep this here.

[dg0yt]

Right now vcpkg assumes that /opt or /usr/bin or wherever vcpkg is installed to be mutable. This is not a problem when you install it manually in /opt/vcpkg and set the permissions to your user, but this is a problem when packaging vcpkg.

This is a particular problems of third-party packaging of vcpkg. As a regular user, you can use vcpkg anywhere in your home directory, as many times as you like.

[gracicot]

@dg0yt indeed. This is only a problem when packaging vcpkg for distributions. As far as I experienced, none of the prepackaged vcpkg (which are easier than manually compiling vcpkg) works properly perhaps beside the nix wrapper script I wrote for my own usage. I tried arch's AUR, brew and some other.

Of course, you can go through the manual steps to configure vcpkg, then adding the VCPKG_ROOT to your env, then having all build scripts using that environment variable, oh and then you have to update it manually, on all the workstation that you have setup in the past. This is much more tedious than having it pre-packaged, and having automation installing it correctly, and having a package manager keeping it up to date.

Most program can be packaged correctly. Most package managers like npm, pip, cargo, can all be installed using a distribution package, and they will all work out of the box. But vcpkg won't. It won't just work out of the box unless you... compile it from source and manually update it?

I don't want to disable users doing the manual setup, or having it as an embedded subproject, I simply want to enable a new way to use it that fits standards and make it easier to automate installation and upgrades.

[JackBoosY]

@gracicot Maybe https://github.com/microsoft/vcpkg-ce ?

[gracicot]

@JackBoosY I fail to see how it solves vcpkg not being installable and unable to be packaged in other system package managers like brew, nix, pacman or dpkg. They are extremely suited to manage programs and tools on a workstation.

[dg0yt]

@gracicot I only commented about your claim that vcpkg makes assumptions about certain system paths. It doesn't. This wasn't a general judgement about the issue but only addressed the quoted part.

It won't just work out of the box unless you... compile it from source and manually update it?

AFAICT for major platforms it just works out of the box and uses a precompiled binary. It has got a simple and clear way to update which works on my Ubuntu 18.04 as on my Macbook Air as in MSYS2. And the Ubuntu 18.04 system packages of other package managers are frozen on old versions.

Note that for versioning in manifest mode, the vcpkg root must be a git checkout, not a plain filesystem.

[dg0yt]

... other system package managers like brew, nix, pacman or dpkg. They are extremely suited to manage programs and tools on a workstation.

Note that vcpkg doesn't aim to manage programs and tools for users. It is focused on libraries. Pure tools and applications are rejected. (Libraries seem to be accepted regardless of quality or relevance.) (And I don't say that it has to be this way. It is rather a limitation, because I have to install autotools etc. with a different package manager on each host.)

[gracicot]

@dg0yt Oh sorry this isn't what I meant. I meant that system package managers are extremely suited to manage pure tools and applications. Since they are well suited for the task, it's a shame that vcpkg cannot be installed through those. I want to keep vcpkg for C++ libraries, and use AUR, nix, brew to install vcpkg itself.

As you can see from my first comment, it's not possible to install vcpkg on nix without the massive hack of making symlinks of everything in the home directory just so vcpkg agrees to put the downloads there. A simple environment variable to tell vcpkg where it can mutate the filesystem freely would fit x the issue and make vcpkg mostly packagable.

[dg0yt]

It means that for any app that need to be packaged, it cannot use vcpkg as a build dependency, thus it cannot use vcpkg to download dependencies.

Why do you think so? AFAICS you can always clone a particular state of the repo, bootstrap, and installed defined versions of the desired dependencies in manifest mode. With the manifest managed in your source code repository. And you can cache the vcpkg binary together with other build artifacts.

[gracicot]

AFAICT for major platforms it just works out of the box and uses a precompiled binary.

Well, doesn't matter that you have to clone the repo instead of using a tool such as pacman, apt, brew, or even chocolatey.

Note that for versioning in manifest mode, the vcpkg root must be a git checkout, not a plain filesystem.

This is very bad news, but let's focus on the fact that vcpkg assume the filesystem is mutable first.

[gracicot]

AFAICS you can always clone a particular state of the repo, bootstrap, and installed defined versions of the desired dependencies in manifest mode

It's the problem I'm trying to highlight. I don't have to do that with any other package managers out there. If I'm writing an MAKEPKG file, I cannot put vcpkg as build dependency in that script and use the manifest to install dependencies. It will fail right now.

I don't have to clone the repository of pip to use it, and I don't have to clone the repository of npm to use it either. They are installed by my system package manager and they will work out of the box, and I had no problem compiling a package that had cargo as builddep. There are no package doing that with vcpkg because it doesn't work with vcpkg, since you have to clone the repo somewhere, then specify where is vcpkg to the build script.

[github-actions[bot]]

This is an automated message. Per our repo policy, stale issues get closed if there has been no activity in the past 180 days. The issue will be automatically closed in 14 days. If you wish to keep this issue open, please add a new comment.

[gracicot]

I am writing a useless comment to "add new content", sorry for all the notifications I'm causing. This issue is not solved yet.