Standard gentoo-like "use" flags across nixpkgs

copumpkin commented 8 years ago

Before telling me nix already supports this, hear me out :smile:

Take a "headless" Nix (NixOS or otherwise) system. A whole load of packages have slightly different options to disable GUI components, but none of them are standardized, and thus we can end up with a completely headless server bundling a ton of graphics stuff because it depends for example on pinentry which optionally supports a GUI prompt and thus pulls in a ton of graphical crap.

What if we instituted a set of standard flags that would be available through stdenv that would help us determine those options? For example:

{ stdenv, fetchurl, mesa, gnome }:

stdenv.mkDerivation {
  buildInputs = stdenv.lib.optional (!stdenv.flags.headless) [ mesa gnome ];
}

And then our closures get small again. Individual packages could still keep ad-hoc flags in their parameter set, but this would help us standardize on a set of common flags of this sort.

Off the top of my head, I'd start with these common flags:

headless: disable any GUI components
~~hardened: use any additional package-specific hardening measures available (the stdenv would also apply some default hardening steps, but certain packages might need additional knowledge of this)~~ [this is already on by default, now]
monolingual: get rid of language translations (and default to whatever you specify if not english)

There are probably others, but I think we could get a lot of value out of just these two.

cc:

@edolstra @shlevy @vcunat: general knowing of things
@globin @thoughtpolice: probably care about the hardened flag

copumpkin commented 8 years ago

I guess this is basically the config field, but cast in a slightly different light, with a centrally determined schema.

cleverca22 commented 8 years ago

one issue i have noticed is that while there is a config.pulseaudio to force it on/off, every app has its own default, so its a total mess until you set it, and once you do, half of them miss the binary cache

copumpkin commented 8 years ago

I'd like to keep the space of flags fairly small so that we can conceivably get Hydra to build common combinations of them. Some sort of audio enable/disable flag also sounds reasonable, but I might just lump it under headless to keep the combination set small.

domenkozar commented 8 years ago

I think we all agree on this one, we need to define an API and the rest will follow :)

vcunat commented 8 years ago

Well, just a single flag may double the number of jobs on Hydra, and I imagine the hardened flag would want to affect some mass-rebuild stuff such as glibc. But we could have a smaller jobset for the hardened config.

What's missing in the config field? Centrally defined defaults and perhaps centrally checked basic consistency (like nixos types)? Why not add such features to config instead?

jagajaga commented 8 years ago

I suppose there was a discussion somehow related to this question.

jagajaga commented 8 years ago

Ooops, here is a link: https://github.com/NixOS/nixpkgs/pull/6968

wmertens commented 8 years ago

Good idea! The current implementation of this is basically the profiles like https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/profiles/headless.nix , but having actual boolean values to check would indeed be better

spacekitteh commented 7 years ago

@grahamc policy discussion, security

vcunat commented 7 years ago

The hardening is on by default, as amended in the issue, so I don't see any security-related stuff remaining here.

spacekitteh commented 7 years ago

the security aspect comes from disabling unnecessary options. for example, a web server has no use for audio support or screen reader support. additionally, it allows the specifying of more-secure-yet-less-performing replacements, such as specifying constraints on allowable crypto algorithms and implementations.

vcunat commented 7 years ago

Side note: if we do have such extreme cases, I think we should remove those options from the default config (and perhaps add a packageFull version as we have in a few cases already).

spacekitteh commented 7 years ago

Reguarding the actual proposal, though, I am highly in favour of it. I think with the purely functional approach we can avoid most of the pitfalls of Gentoo's implementation (I spent a week trying to sort out a broken gentoo install due to conflicting flags, which happened because I performed installs in the wrong order)

If we put some thought into our flags and their defaults, we could probably have a large range of them yet not blow up hydra - one major thing to think about is sets of related flags (e.g. a "hardened" set) which sets multiple individual flags. Then hydra could just build the flagset defaults, and let tinkerers build locally.

oxij commented 7 years ago

This is exactly what the original version of #6968 did with nixpkgs.config.common option (see several of my first comments in that PR), but those changes were cut as unorthodox and the PR was trivialized to the point of doing nothing in that regard.

However, I still rebase those original changes on top of master since I can't live without them (they bitrotted considerably after two years of rebases, though).

Personally, I do some crazy stuff with nixpkgs.config, I turned it into something like "NixOS options", pretty much. But even old mild version of that was rejected as unorthodox.

Still, I think I can extract something agreeable for everyone from my personal branch

if we can agree that nixpkgs.config option is useful and not actually deprecated,
we force *Support and with* package options defaults into package definitions (this is the part I constantly fix on rebasing)

pkg = callPackage ./something.nix {
   someSupport = true;
};

breaks the thing. That someSupport need to be in ./something.nix, not in all-packages.nix (it's also cleaner that way, IMO).

(optionally) we agree to rename all *Support options into with* or vice versa, or you'd have

nixpkgs.config.common = {
  x11Support = false;
  withX = false;
  withX11 = false;
};

in your configuration.nix, which is meh.

What I do in my branch to achieve this is simple: I have callPackages look at nixpkgs.config.common and apply those options to everything (my version does much more, including several different callPackages, but we shall ignore that for now).

But that change alone is useless since defaults in all-packages.nix will override the common options. The change needed to make it actually work for all packages is huge.

I'm not extracting it from my wizardry (it's like a day of work, if not more) until people say they sure they want it.

MarcWeber commented 7 years ago

http://lists.science.uu.nl/pipermail/nix-dev/2017-March/023248.html Well -> having flags doesn't mean all combinations have to be provided as binaries by Hydra or that all combinations must be tested. A warning line could be enough. Having a sink to express the need and setup such flags (even without providing binaries) could be helpful. Flags can also be commented

copumpkin commented 7 years ago

Yeah, as long as people don't go adding them willy-nilly for tiny package-specific flags that's probably fine. I'd want there to be a bit of governance around what makes sense to go into a flag. It also gets interesting when you consider that we support more than one platform.

spacekitteh commented 7 years ago

Yeah, single-package-specific flags don't make sense as global flags. On the other hand, things like "headless" still have to be implemented carefully - e.g. openjdk needs some AWT components, even in headless mode, in order to support generating images, and currently the headless flag there overlooks that.

twhitehead commented 6 years ago

You could technically add common flags to the top level callPackage namespace (see pkgs/top-level/splice.nix). This way they would automatically become available to satisfy the arguments of all packages.

I think the real key, however, would be to take the NixOS option declaration and checking machinery and apply it to defining and declaring these options. This way they would be documented, defaulted, checked, and ideally automatically extracted and made available on a website to lookup as with NixOS options.

Imagine if the top-level config attribute was not just sucked in blindly from ~/.config/nixpkgs/config.nix, but rather generated from this file in the manner of the NixOS machine configuration. That is, verified against a declaration of valid options, which itself is declared in a modular manner.

Packages could also have the option of extending the config with package specific options by providing an optional options attribute (or perhaps meta.options?). This would functions in the same manner as the NixOS modules options attribute. So, for example, a config.emacs entry ~/.config/nixpkgs/config.nix would be matched up and checked against an emacs.options declaration.

twhitehead commented 6 years ago

One way to test this would be to extend the current nixpkgs NixOS module to provide a verified config attribute set to nixpkgs (i.e., one constructed using the NixOS option machinery) instead of just the current anything goes nixpkgs.config option.

Ultimately you could imagine nixpkgs itself being changed to not default the config attribute to just importing ~/.nixpkgs/config.nix, but rather to use the NixOS module system to treat ~/.config/nixpkgs/config,nix as a module config declaration for nixpkgs.config to be merged with the system declared one from /etc/nixos/configuration.nix to generate the actual config attribute in a verified manner.

Note also how this is nicely orthogonal to the use of overlaying and overriding which provide a nice modular way for the user to actually make changes to nixpkgs itself without having to fork it.

oxij commented 6 years ago

Anyone still interested in this, please see https://github.com/NixOS/nixpkgs/issues/36229#issuecomment-370180008 It describes how SLNOS implements this feature (and gives some social commentary on related issues).

oxij commented 5 years ago

A demo implementation #56110 with some discussion of what a proper implementation will take.

Ekleog commented 4 years ago

(triage) I think this will require an actual RFC in order to move forward. Leaving open until someone actually writes it, though, so this issue can track the need to write down the RFC.

stale[bot] commented 4 years ago

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

Search for maintainers and people that previously touched the related code and @ mention them in a comment.
Ask on the NixOS Discourse.
Ask on the #nixos channel on irc.freenode.net.

Pitometsu commented 4 years ago

For me it's important.

wmertens commented 4 years ago

I agree, this is important, and as @Ekleog said this should be an RFC with a single proposal for the way forward. Any champions amongst our midst?

stale[bot] commented 3 years ago

I marked this as stale due to inactivity. → More info

Pitometsu commented 3 years ago

And that is still quite important.

WillPower3309 commented 2 years ago

Anyone interested in this may want to also look at #143860

NixOS / nixpkgs

Standard gentoo-like "use" flags across nixpkgs #12877