simpler, saner cross-compilation

ghost commented 1 year ago

This is a long-term goal; none of this can happen overnight. This is also an incomplete work-in-progress.

TL;DR: Short Example

{ pkgsOnHost          # simple dependencies
, pkgsOnBuild         # build tools that don't emit code (`grep`, `jq`...)
, pkgsOnBuildForHost  # build tools that emit code for the hostPlatform
}:

{

  # setup.sh will put this stuff in the $PATH at compile time
  depsInPATH = with pkgsOnBuild; [
    cmake
    makeWrapper
    pkgsOnBuildForHost.gfortran     # to compile fortran code for use at run time
  ];

  # setup.sh will not put any of this stuff into the $PATH
  deps = with pkgsOnHost; [
    openssl
  ];

}

Wishlist

Eliminate splicing

I made splicing (nativeDrv and crossDrv are older, but were done differently). Splicing is bad. I never wanted it. I causes major headaches when getting anywhere near bootstrapping. It is gross. It is probably slow. It should be removed. -- https://github.com/NixOS/nixpkgs/issues/204303#issuecomment-1383282367

Splicing is too magical. Probably three[^1] people really understand it. Dozens of people think they understand it but really don't, and get pissed off when it breaks in weird ways. This is a major part of why people hate on cross compilation.

Eliminating splicing likely means that package expressions will need to take package sets explicitly as top-level arguments. See next section.

Eliminate special exceptions to `depsFooBar` naming

Basically, do this (thanks @Artturin):

https://github.com/NixOS/nixpkgs/issues/28327

`targetPlatform==null` for packages that do not emit code

The vast majority of packages in nixpkgs don't emit code. Instead of setting their targetPlatform equal to their hostPlatform, we should set it to null.

Using targetPlatform==hostPlatform erases the distinction between packages that emit code for their hostPlatform and packages that don't emit code. It also creates ambiguity: these packages can be moved freely between (say) depsBuildHost and depsBuildTarget with no change in behavior, so the difference between depsBuildHost and depsBuildTarget is harder for people to learn.

Prefix all binaries that have a targetPlatform

AKA resurrect this PR:

https://github.com/NixOS/nixpkgs/pull/44583

Right now we have an artificial distinction between:

pkgsBuildTarget for build=X target=Y
pkgsHostTarget for host=X target=Y

The first kind won't have a aarch64-linux- prefix on their code-emitting binaries (like gcc). The latter kind will. This is silly. Prefix all the binaries, eliminate the distinction.

For compilers (clang, rust) that have a single binary entry point for all platforms and use some kind of --target= flag, we simply wrap that entry point with a wrapper for each targetPlatform which adds the --target= flag.

Use the pythonPackages `onBuildForHost` naming

pkgsFooBar is extremely unergonomic. Nobody can remember what Foo is for and what Bar is for. These should be changed to pkgsOnBuildForHost like pythonPackages does; with pkgsOnFooForBar least that way there is a reminder of what Foo and Bar are: pkgsOnFooForBar runs on Foo and emits code for Bar.

Packages with no target (see previous heading) get shorter names: pkgsOnBuild and pkgsOnHost.

No more `depsOnXForY`, only `pkgsOnXForY`

The distinction between the various depsFooBar attributes of a derivation serves two purposes:

"Desplicing" the correct derivation from the spliced package
Ensuring that only depsBuildX stuff goes in the $PATH at build time

Instead of (six? more?) different derivation attributes we only need two:

depsInPATH  = [ ... ]  # stuff that is put into $PATH
deps        = [ ... ]  # stuff that is not put into $PATH

Long Example

This example might look complicated, but it exercises all possible arguments and attributes. Think about that. This is the most complicated attrset you'll ever see.

Non-code-emitting packages will only have, at most, the first five attributes (one of which is never used in practice). In practice most packages will have only three arguments: pkgsOnHost, pkgsOnBuild, and pkgsOnBuildForHost.

{ pkgsOnHost          # simple dependencies
, pkgsOnBuild         # build tools that don't emit code (`grep`, `jq`...)
, pkgsOnBuildForHost  # build tools that emit code for the hostPlatform
# the three arguments above are enough for nearly all packages in nixpkgs

, pkgsOnHostForHost   # theoretical possibility only: zero uses in nixpkgs
, pkgsOnBuildForBuild # build tools that emit code for the buildPlatform

# **only code-emitting packages** (compilers) will have the following:
, pkgsOnHostForTarget   # theoretical posibility only: canadian cross
, pkgsOnBuildForTarget  # e.g. the bintools used to link the libraries bundled with gcc
, pkgsOnTarget          # target libraries bundled with the compiler (eg libstdc++)
, pkgsOnTargetForTarget # code-emitting target libraries (JITs), like libgccjit
}:

{

  # setup.sh will put this stuff in the $PATH at compile time
  depsInPATH = with pkgsOnBuild; [
    cmake
    makeWrapper
    pkgsOnBuildForHost.gfortran     # to compile fortran code for use at run time
    pkgsOnBuildForBuild.stdenv.cc   # to compile+run a C program at build time
  ];

  # setup.sh will not put any of this stuff into the $PATH
  deps = with pkgsOnHost; [
    openssl
    pkgsOnTarget.libstdc++
  ];

}

Problems

Note: there is a possible solution to both of the following two problems, which brings additional benefits.
1. This needs to be harmonized with callPackage. It is really unfortunate that the Nix language does not let functions declare deep attributes as arguments, like {foo.bar, ...}: ....
2. In its current form this makes .override very painful to use, since it don't cope well with overriding a sub-attribute of an argument. It should be extended to provide an ergonomic way to override sub-attributes of an argument.

Questions

Should deps and depsInPATH be combined into a single attribute? mkDerivation could decide whether or not to put each dependency into the $PATH based on whether stdenv.buildPlatform.canExecute that dependency's hostPlatform.

In practice there are very very very few situations where you don't want all the possibly-executable dependencies put into the path; we could provide an "escape hatch" to mark specific dependencies as "don't belong in the $PATH even if we could execute them".
with is a footgun, and this scheme encourages with pkgsOnBuild. We could make deps and depsOnBuild be attrsets (whose attrnames are ignored); this would allow the use of the much-safer inherit syntax. A better long-term solution would be RFC 110.
Should pkgsOnBuildForHost be an attribute within the pkgsOnBuild attrset? For example, pkgsOnBuild.forHost. This would mean only two arguments for non-compiler-like packages.

[^1]: I am not one of these people. But I'm also aware that I don't fully understand splicing.

ghost commented 1 year ago

Possible solution to the .override problem proposed in https://github.com/NixOS/nixpkgs/issues/204303#issuecomment-1403662186

Artturin commented 1 year ago

Ensuring that only depsBuildX stuff goes in the $PATH at build time

Note that the dependency attributes are used for more than that

https://github.com/search?q=repo%3ANixOS%2Fnixpkgs%20targetOffset&type=code

https://github.com/NixOS/nixpkgs/blob/4f322852fdaf6009f53ae1256ef5b48572b50c4f/pkgs/build-support/setup-hooks/role.bash

https://github.com/NixOS/nixpkgs/blob/4f322852fdaf6009f53ae1256ef5b48572b50c4f/pkgs/stdenv/generic/setup.sh#L518

kjeremy commented 1 year ago

From the peanut gallery I've only been using nix for about a year and a half now and have been continuously confused by the naming around cross-compiling so I think this is a good direction to go in.

One thing that continuously trips me up is the name Host. I always have to pause and think "is this the build host or the target host?". When cross-compiling an app I never think of the word "host", it is always "build" (on what building occurs) and "target" (on what the artifact is executed). Basically: Is there a better word than Host (what is being hosted here)? My head always reaches for "target" but that seems to be taken. Maybe if host is to be used we could still use the word but make it clear what is being hosted: pkgsOnBuildHost, pkgsOnTargetHost.

Artturin commented 1 year ago

@kjeremy We're following the autoconf definitions gnu.org/software/autoconf/manual/autoconf-2.68/html_node/Specifying-Target-Triplets.html while you're thinking of something else

Artturin commented 1 year ago

Eliminate special exceptions to depsFooBar naming

initial work https://github.com/NixOS/nixpkgs/pull/227502

ghost commented 1 year ago

We're following the autoconf definitions

Yeah, autoconf made some less-than-ideal choices there (mainly "host") but we are sort of stuck with them. Trying to change the terminology creates even more confusion (cough llvm cough).

I do think that nixpkgs makes the situation a bit worse by saying that non-code-emitting packages (like bash) have a targetPlatform. This is why I propose to set targetPlatform=null for packages that don't emit code (i.e. most packages). Getting an error about targetPlatform being null should help correct a lot of host/target mixups and help people learn the standard terminology more quickly.

ghost commented 1 year ago

@Artturin thanks for pointing out activatePackage().

https://github.com/NixOS/nixpkgs/blob/e6899c3b249d57f5c702e4a31e3bd088ff369da2/pkgs/stdenv/generic/setup.sh#L647-L650

https://github.com/NixOS/nixpkgs/blob/e6899c3b249d57f5c702e4a31e3bd088ff369da2/pkgs/stdenv/generic/setup.sh#L683-L698

Ugh. I sort of wish hooks didn't exist, at least as bash scripts... I wish they were managed at the level of Nix code and stitched together by mkDerivation. But getting to that point is an even bigger job than eliminating splicing, so I should stop hoping.

My initial instinct is that we could stash these offsets in a __offset attribute of the packages. So for a build==host situation pkgsOnBuild.bash != pkgsOnHost.bash but only because of this attribute -- (pkgsOnBuild.bash // {__offset=null;}) == (pkgsOnHost.bash // {__offset=null;}). And the attribute should probably warn people that it exists only to implement setup hooks and shouldn't be used for anything else since it could go away in some far-future world where we assemble and sort the hook list in nix instead of in bash.

initial work #227502

Awesome, will review.

ghost commented 1 year ago

Another solution to the .override problem, which is a bit more radical but fits really well with this change, is @roberth's idea of always using mkDerivation(finalAttrs: style and making deps a passed-through attribute of every derivation. Then .override goes away, because you can do what it does using .overrideAttrs (I see this as a massive benefit).

The example below is copied from this issue below the text "instead it could look like" since I can't seem to link directly to that example:

pkgs:

pkgs.stdenv.mkDerivation (self: {
  # deps here is a "local" variable, not passed to builtins.mkDerivation
  deps = {
    inherit (pkgs) pkg1 pkg2;
  };
  buildInputs = [ self.deps.pkg1 self.deps.pkg2 ];
})

The "let .overrideAttrs subsume .override" idea seems to have undergone further development but the above was the most recent version that didn't require modules.

In the context of getting rid of splicing, this would mean pkgs being an attrset-of-packagesets (pkgs.onBuild, pkgs.onHostForTarget, etc plus two special exceptions: lib and stdenv), and what I call deps in the first comment of this issue being renamed to something like inputs (there is still only one such attribute -- no more buildInputs/nativeBuildInputs/etc). So something approximately like:

pkgs:

pkgs.stdenv.mkDerivation (self: {
  # deps here is a "local" variable, not passed to builtins.mkDerivation
  deps = {
    inherit (pkgs.onBuild) cmake makeWrapper;
    inherit (pkgs.onBuildForHost) gfortran;
    inherit (pkgs.onHost) openssl;
  };
  inputs = with self.deps; [ cmake makeWrapper gfortran openssl ];
})

With splicing gone and buildInputs/nativeBuildInputs/depsFooBar combined into inputs, this raises the question of whether mkDerivation can simply default to:

  inputs = builtins.attrValues self.deps;

I think it can, which means we can drop the inputs = ...; line except in very unusual situations. This is an added bonus, because it is very tempting to use with deps in inputs (as I did above) which can lead to unexpected results during treewide changes because with clauses have their own separate lexical scoping.

Artturin commented 1 year ago

Eliminate special exceptions to depsFooBar naming

@Ericson2314's issue https://github.com/NixOS/nixpkgs/issues/28327

nixos-discourse commented 5 months ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/is-this-the-current-state-of-things/45944/13

nixos-discourse commented 4 months ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/frustrations-about-splicing/49607/1

nixos-discourse commented 3 months ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/frustrations-about-splicing/49607/16

nixos-discourse commented 1 week ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/is-it-possible-for-us-to-remove-builtins-functionargs/51960/5

roberth commented 1 week ago

pkgs:

pkgs.stdenv.mkDerivation (self: {
  # deps here is a "local" variable, not passed to builtins.mkDerivation
  deps = {
    inherit (pkgs.onBuild) cmake makeWrapper;
    inherit (pkgs.onBuildForHost) gfortran;
    inherit (pkgs.onHost) openssl;
  };
  inputs = with self.deps; [ cmake makeWrapper gfortran openssl ];
})

This looks pretty good to me.

Possible limitation, but don't know how bad it is: Some derivation may currently depend on a package x that has a tool and a library:

strictDeps = true;
nativeBuildInputs = [ x ];
buildInputs = [ x ];

and this would work, because the attributes like nativeBuildInputs accurately describe the role of the package.

I don't know if this would be good enough:

deps = {
  x_build = pkgs.onBuild.x;
  x_host = pkgs.onHost.x;
};
inputs = with deps; [ x_build x_host ];

when non-cross, we don't have the info to infer the intended role, because x_build.stdenv == x_host.stdenv. Simply put, having only inputs, we lose strictDeps. That's a significant regression, because it removes a little "forcing function" that makes non-cross package authors do the right thing.

So we'd have to bring back the buildInputs, nativeBuildInputs etc attributes, largely duplicating the same info in deps.

It seems that we have to choose two of {strictDeps, low boilerplate, no splicing}, but I hope I'm wrong.

It does make me wonder if splicing could be done right. The documentation is lacking and it seems underdeveloped. The other day I've tried to implement a newScope helper and failed. It seems that the splicing code is quite ad hoc, and we're actually missing some of the primitives that would make implementing a nicer newScope helper easy. Either that, or maybe I'm just overly optimistic.

Atemu commented 1 week ago

I think it'd be best to separate deps between those that should be in executable PATH and deps that should be in library discovery paths.

I've actually been cooking a design for a "mkDerivation v2" over the past few weeks where I've addressed pain points like this and many others in a complete re-design. I've been adding bits as I come across bad APIs or UX in our current derivation wrappers and I need to think about a few things some more, so it's not quite done yet but stay tuned for that.
That is very much a "vision wishlist" rather than something we could practically implement today with any efficiency.

NixOS / nixpkgs