shlevy
commented
6 years ago :+1: from me. I think some asserts are true sanity checks about argument compatibility, but the rest should switch to meta.
Closed 7c6f434c closed 2 years ago
shlevy
commented
6 years ago :+1: from me. I think some asserts are true sanity checks about argument compatibility, but the rest should switch to meta.
7c6f434c
commented
6 years ago Is there any reason not to move argument compatibility sanity checks to meta.broken, too? After all, a package with withGtk==true and gtk == null is for all practical purposes broken…
Ericson2314
commented
6 years ago Additinally, with https://github.com/NixOS/nixpkgs/issues/36226 I don't think there is any more reason for somePkg ? null. I mean, it isn't like we're using packages in nixpkgs in a smaller package set, right?.
7c6f434c
commented
6 years ago Well, sometimes these have to be extracted from a nested package set (like pythonPackages)
Ericson2314
commented
6 years ago I'd say, if the assert only results from user error, and not any arbitrary broken packages situation, then the assert is fine. For example:
{ withFoo ? foo.meta.available, foo }:
assert withFoo -> foo.meta.available;The assertion respects the default, so it would only fail in the presence of user overrides. And users should in fact only do overrides to make things less enabled. If some optional feature is needed downstream, but it in that downstream package's meta.broken.
rnhmjoj
commented
6 years ago So, I want to make sure I understand correctly. For example is this ok?
{ x11Support ? stdenv.isLinux, mesa }:
assert x11Support -> libX11.meta.available && libXext.meta.available [...];
[...]
buildInputs = [...] ++ optionals x11Support [ libX11 libXext mesa libXxf86vm ];@Ericson2314 from time to time people use some kind of a deep override; it would be nice if a global deep override of libxcb to null worked in a comprehensible way.
What is the benefit of not putting this condition into meta.broken?
oxij
commented
6 years ago :+1: on the idea, as this is on SLNOS TODO list too.
SLNOS motivation: derivations should be lazy (#33057) as this gives a lot of nice bonuses.
But I also completely agree with
I think some asserts are true sanity checks about argument compatibility, but the rest should switch to meta.
My motivation: I want to fuzz package builds that depend on package options (use flags).
I want to check that all allowed combinations of flags actually build and (eventually) that all disallowed don't.
The latter would be implemented with tests for flags (or combinations of them). Possible tests include:
In general, most asserts are either of the two kinds:
assert gtkSupport -> x11Support || waylandSupport,nulls like assert gtkSupport -> gtk2 != null.I think the second kind should simply not exist (https://github.com/NixOS/nixpkgs/issues/35906#issuecomment-369465393). When all derivations are lazy you don't need to null them to hide the ones that won't work on a given system, after #33057 you can still access their attributes, hence you don't need to check they are not nulls anymore. nulls now just get in the way of debugging actual package dependencies.
Switching the first kind of asserts from assert to broken is fine for me, but I would prefer we switch to several tiers of those (names are subject to change):
meta.makesNoSense - logically incompatible combination of flags,meta.wontBuild - building this would fail (should be implied by and eventually equivalent to makesNoSense) (this is easy to fuzz),meta.wontRun - would build, but won't start (ideally, such expressions should fail to build, but there're things that are hard to test in nix, like they require you to run an Xserver or a database) (this can be fuzzed with nixos tests),meta.wontWork - would build and run, but won't work as you would expect (same thing) (same thing) (maybe this should be squashed with wontRun, I'm not yet sure),meta.broken - broken, but no reason given.I also would like those to be more verbose than just bools. Say, like asserts and warnings of NixOS.
Also note that all of this plays really well when you explicitly mark the use flags. Are you still sure you don't want them (#12877. Wink, wink!)?
7c6f434c
commented
6 years ago Of all things, running X during a build is trivial: https://github.com/7c6f434c/lang-os/blob/master/firefox-profile.nix
Whether we want it is another question.
USE flags, multiple kinds of brokenness etc. require maintenance effort to continue making sense, which makes achieving buy-in harder…
Ericson2314
commented
6 years ago @oxij look up rust portability lint. Fuzzing would never exhaust space but we could with SAT solver.
@rmhnjoj. I don't think so, because a Linux host platform does not always ensure those packages are available. If it did however, that would be good.
7c6f434c
commented
6 years ago The problem is that we are talking about build failures, and that cannot be just fed to SAT solver. The only thing you can hope for is increasing the chance of a problem being found per minute of build.
oxij
commented
6 years ago I think you over-complicate the problem. Most of the time you can see if the buildInput list is correct by the configure output log. It's super-cheap. So you can reasonably exhaustively check use flags for most of the packages of nixpkgs.
For packages with massive amounts of use flags (like ffmpeg) fuzzing *Support flags one by one would already be uber-helpful and would find 99% of the errors in minutes.
I imagine that to work like this:
it sets all use flags to false, computes buildInput (lazy!), runs configure, checks the log for references,
then it overrides flags one by one with true, computes new buildInputs, runs configure, looks at the log, checks that it now uses the new buildInputs.
All of that would not work with overrides with nulls.
To answer https://github.com/NixOS/nixpkgs/issues/35906#issuecomment-370157300
nullprovides a way to quickly remove some dependencies without adding extra support into packages.
I agree. But I consider that to be a hack and I think it should be treated as a hack. Normally, things should be put under flags.
Also, note that the composition of use flags and fuzzing makes maintenance burden less, not more: you add a new use flag, then you add the corresponding buildInputs, ask the fuzzer to fuzz it, get to see whenever the build would be likely to succeed and do what you expect in seconds instead of hours.
7c6f434c
commented
6 years ago Also, note that the composition of use flags and fuzzing makes maintenance burden less, not more: you add a new use flag, then you add the corresponding
buildInputs, ask the fuzzer to fuzz it, get to see whenever the build would be likely to succeed and do what you expect in seconds instead of hours.
I think we don't have enough acceptance for multiple versions to have people even look at the fuzzer.
oxij
commented
6 years ago (Posting this as a separate comment to refer back to specifically this. This seems like a good place as any to remind you that all of those ideas I constantly talk here combined produce some awesome benefits.)
Writing this despite the fact that, most likely, unless I frustrate myself to the point at which this thread will become a flamewar with many non-pretty words, this would just get ignored :(
Anyway.
Anybody skeptic of that not having nulls can be made short and beautiful is requested to proceed to read pkgs/applications/audio/cmus/default.nix on master (because it's an exemplary expression of AFAIK the best thing that can be done with master's infrastructure), and then proceed to imagine an expression that has no duplication of
{ stdenv,
...
, cddbSupport ? true, libcddb ? null
...
}:
let
opts = [
...
(mkFlag cddbSupport "CONFIG_CDDB=y" libcddb)
...
];
in
stdenv.mkDerivation {
...
configureFlags = ... ++ concatMap (a: a.flags) opts);
buildInputs = ... ++ concatMap (a: a.deps) opts;
};kind, but instead does this
{ lib }: with lib; mkUFDerivation (attrs: with attrs; {
required = mkRequired [ "stdenv" "pkgconfig" ];
# mkYNFlag : (defaultValue : Bool) -> (configureFlag : String) -> (buildInputs : [ String ]) -> mkUFDerivationMagic
darwin = mkYNFlag stdenv.isDarwin null [ "CoreAudio" ];
alsa = mkYNFlag stdenv.isLinux "ALSA" [ "alsaLib" ];
jack = mkYNFlag false "JACK" [ "libjack" ];
samplerate = mkYNFlag useFlags.jack "SAMPLERATE" [ "libsamplerate" ];
...
cddb = mkYNFlag true "CDDB" [ "libcddb" ];
...
}) (attrs: with attrs; stdenv.mkDerivation {
...
nativeBuildInputs = [ pkgconfig ];
configureFlags = useFlags.configureFlags;
buildInputs = useFlags.buildInputs;
...
})Note how short that is.
Adding a new dep with a use flag and configure options? Its a single line!
Disabling a dep? nixpkgs.config.<package>.dep = false!
It's shorter than your shortest packageOverrides thing with nulls and it can also properly check any deps between flags (which .override fails to do and nulls forbid you to even think about, btw).
But "in nixpkgs nixpkgs.config is deprecated, you should use overrides instead!" Right? Pfft!
Support for other build systems is really simple too: just make another kind of mkYNFlag.
Isn't that much better than what master has?
SLNOS says "Yes!". Hence let me introduce you into how the above works line by line.
lib.mkUFDerivation that does all the magic.
Note that we tie the knot with attrs there. That attrs feels like attrs = { inherit useFlags pkgs; } // pkgs; to the rest of the function.stdenv and pkgconfig to required pkgs.lib.mkYNFlag to make up required pkgs and useFlags attributes.mkUFDerivation) gets the same attrs and does the rest pretty much as usual, but now it can use useFlags for awesome things.Calling this, as you might have noticed, does require a slightly different kind of infrastructure behind callPackage.
Merging such expressions with changes to nixpkgs is paniful, so SLNOS doesn't use them much (yet).
However, in the light of that expression above I fail to see why nixpkgs goes out of the way to reject use flags (#12877). Like, seriously, just, WHY?!
Now, to completely blow your mind, look at the above expression and then remind yourself that many, if not most, packages in nixpkgs differ only in deps, src and some meta attributes, the rest is very copy-pastey.
Wouldn't it be cool if all you needed to do to define most packages was: call some magical lib.mkUFNoCopyPasteyDerivation, specify use flags with deps, specify src, meta.license and meta.description, and it figures out the rest for you (homepage from src, platforms from deps, etc).
Now, do you see the hidden agenda behind #35075 and
As a side note,
meta.platformsshould be checked (or just autogenerated, or maybe both, maybe withconfig.checkMetaPlatformsor something) against platforms produced by intersecting platforms of derivations frombuildInputs. But I have not implemented that properly yet.
of https://github.com/NixOS/nixpkgs/issues/35906#issuecomment-369465393 ?
Do you want to welcome such an agenda now (if you do, I can port mkUFDerivation infra to nixpkgs) or do you want to wait another 3-5 years until, maybe, somebody with commit access implements something like this (usually, worse) with little to no discussion.
Let me remind you that this is, pretty much, my experience with nixpkgs merge process, e.g. see anything nixos related from https://github.com/NixOS/nixpkgs/pulls?page=1&q=is%3Apr+author%3Aoxij+is%3Aunmerged&utf8=%E2%9C%93 , which gave birth to SLNOS.
I want SLNOS to be less than a 100 patches on top of master (most of which would be in OS department), but master makes good things hard to archive in less than 400, which is really painful to merge all the time.
Why can't nixpkgs move in a better direction with use flags? Seriously. So much time is wasted by nixpkgs contributors doing copy-pastey things, and then by SLNOS contributors undoing those things and merging with SLNOS. So frustrating! Grrr.
Can't anything be done with bikeshedding and "not invented here" syndrome in nixpkgs? Like, did you read GuixSD's sources yet? No? You should go read GuixSD's sources! Grrr.
/cc @edolstra @shlevy @vcunat on this
7c6f434c
commented
6 years ago Oh well. The real problem is: you say «worse», what you mean is it allows strictly less, I agree that the solution in Nixpkgs will be worse, but there is the position that it will do less and will be in some sense more lightweight and so it will be in some sense «better».
I have already implemented and committed multiple solutions to remove duplication between buildInputs and the function parameters, and all of them were explicitly criticised (not silently ignored) even when I was only committing them into packages that nobody but me updated. So I assume that some things will never happen — not because of stalling or problems with decision-making, but because of explicit opposition of many people. Powerful usable overrides are considered harmful, oh well. I don't want so many overrides to justify not using Hydra binaries for the rest of the system.
Also, I tried to keep a couple of latest versions by default for libraries, and that also was explicitly rejected. I think there is some idea about too many combinations making things untestable or the exact use pattern of library versions hard to establish or risk of having different default dependency versions for downstream packages and problems when combining them further downstream.
The thing that happens with 3–5 years and limited implementation is not based as much on NIH, as on accumulation of support for some feature that makes reverting a small enough implementation unattractive.
The detail with packageOverrides is that the complicated packageOverrides expression is really a one-time cost, not per-package.
Ignoring all these flags etc. and just passing null leads to shorter default-case expressions anyway (and that is what too many people care about). As for nulls not allowing to check something — why exactly? I can check for null just fine.
Copy-pastey expressions are not enough. when they are enough, if I fully buy all the idea that a separate version should not be specified, come at 16 non-blank lines with 10 lines to actually fill. Platforms from deps is iffy, unifying deps and arguments meets pushback, and the rest actually requires human input. So when you find out that people want the most limited packages possible, there is just nothing to generate.
It is probably better to make as much of the distinctions inside Nixpkgs as possible library functions, and not these splits into the lists like nativeBuildInputs. The most expensive part of Nixpkgs is the leaf package expressions, and if one could just give them a different stdenv.mkDerivation, the package version/patch database could be maintained together by everyone and the stdenvs could be different.
In the context of meta.broken, that probably means putting everything into meta.broken; maybe with optional-trace-generating wrappers around groups of conditions to hint whether the failure is likely to be a buildtime or a runtime one.
oxij
commented
6 years ago I have already implemented and committed multiple solutions to remove duplication between
buildInputsand the function parameters, and all of them were
I'm not aware of anything except composableDerivation and things similar to how cmus expression is done. Any pointers?
I think there is some idea about too many combinations making things untestable or the exact use pattern of library versions hard to establish or risk of having different default dependency versions for downstream packages and problems when combining them further downstream.
I can get behind that argument. But I also see a solution: simply hide non-default versions by default with something like knownVulnerabilities, e.g. knownBetterVersions :)
You need it? And it doesn't meet the default nixpkgs inclusion conditions? You just have to enable it explicitly.
The thing that happens with 3–5 years and limited implementation is not based as much on NIH, as on accumulation of support for some feature that makes reverting a small enough implementation unattractive.
That is another plausible explanation, yes, but seeing how NixOS slowly adopts things that were proposed and wholeheartedly rejected years ago I wonder if that is actually the case.
My experience shows that most people hate reading others' source code. In SLNOS we have "show you did your research first" rule (which, you might have noticed, I try to follow in my nixpkgs PRs too). Anything non-trivial has to provide relevant references like "it's a remake of/solution to such and such nixpkgs/Triton/GuixSD/Gentoo/Arch/Debian/etc PR/issue/etc".
and just passing
nullleads to shorter default-case expressions anyway (and that is what too many people care about).
Eh? My whole point that it is not. Did you read the above?
As for
nulls not allowing to check something — why exactly? I can check fornulljust fine.
By
it can also properly check any deps between flags (which
.overridefails to do andnulls forbid you to even think about, btw)
I mean that flags are strictly more expressive than nulls. How do you properly disable x11Support? You set it to false. How you do that with nulls? You override all the X11 libraries with nulls? Seriously? Flag dependencies are easy to check, ad-hoc null overrides are much harder to check properly (expressions get large and ugly pretty fast).
But if one almost never overrides things because hydra cache then, yes, all of that is purely academic.
Platforms from deps is iffy,
But completely optional and independent from the rest of the above (it's just that I like the idea). Because I like all the ideas that lead to more automation. Computers are cheap. My time is very limited. I hate cross-compiling shit for hours at a time to find out that what I actually needed is not actually supported.
unifying deps and arguments meets pushback,
"Yes, but why?"
and the rest actually requires human input.
If you mean "the expressions need to be rewritten" then, yes, they do.
So when you find out that people want the most limited packages possible, there is just nothing to generate.
=/
It is probably better to make as much of the distinctions inside Nixpkgs as possible library functions, and not these splits into the lists like
nativeBuildInputs.
It just came to me that you can do that with the SLNOS mkUFDerivation too, like so
mkUFDerivation (x: {
native = mkNativeBuildInputs [ "pkgconfig" ];
}) bodyand a proper stdenv.mkDerivation (or whatever else) wrapper in the wrapper.
The most expensive part of Nixpkgs is the leaf package expressions, and if one could just give them a different
stdenv.mkDerivation, the package version/patch database could be maintained together by everyone and the stdenvs could be different.
Hm. That is quite an interesting idea, actually. But I don't see that working for complicated packages with intricate interplay between inputs and build phases, but most packages can be written this way, yes.
I don't see that happening in nixpkgs, though. Such a thing is just too far away from what the core of contributors cares about.
In the context of
meta.broken, that probably means putting everything intometa.broken; maybe with optional-trace-generating wrappers around groups of conditions to hint whether the failure is likely to be a buildtime or a runtime one.
Eh? But why? It isn't like nix-env depends specifically on meta.broken or something. We can do pretty much whatever with meta attributes. Why not do something fuzzable then?
aneeshusa
commented
6 years ago I find that I run into a bad interaction of having asserts at the top level of a package expression with overrides. Namely, if an assert fires due to an a change from an overlay, then a future use of .override on that package which would normally pass the assert doesn't work. An example (not that I actually want to remove http2 support, but can't remember which package I was working with when this bit me):
# in an overlay
self: super: {
nghttp2 = null; # causes curl to fire assert, as curl defaults http2Support to true
curl = super.curl.override { http2Support = false; } # doesn't fix the issue; happens too late
}The only way I've found to fix this is to make the assert lazier by placing it right before nghttp2 is used, i.e. replace
optional http2Support nghttp2 ++with
optional http2Support (assert nghttp2 != null; nghttp2) ++While I do like having these checks, it's a bit annoying to have to manually lazy-ify each one. I hope a better design here can avoid this problem; I haven't spent the time to read through this thread and all the proposals, but for example I think something like https://github.com/NixOS/nixpkgs/issues/36229#issuecomment-370138378 might still be prone to this issue.
7c6f434c
commented
6 years ago I have already implemented and committed multiple solutions to remove duplication between
buildInputsand the function parameters, and all of them were I'm not aware of anything exceptcomposableDerivationand things similar to howcmusexpression is done. Any pointers?
Well, for example helperArgNames (up to beginning 2016)
You need it? And it doesn't meet the default nixpkgs inclusion conditions? You just have to enable it explicitly.
I wonder is a bot could maintain nixpkgs-attic repository, which would automatically make the version-addressed snapshots of Nixpkgs master. If some version is not present in master anymore, it just stops getting any changes. A policy could be that the versions that are still present in Nixpkgs should be changed there (this would be ‘‘enforced’’ by the bot actually overwriting the changes), and for old versions anything goes if someone cares.
That is another plausible explanation, yes, but seeing how NixOS slowly adopts things that were proposed and wholeheartedly rejected years ago I wonder if that is actually the case.
I have seen the full spectrum of reactions to changes from committers pushed to master — reverts, negotiating the scale down, complaints with no action. Basically, there is a desire for minimalism, but then there are practical considerations that slowly erode the will to oppose the feature creep.
and just passing
nullleads to shorter default-case expressions anyway (and that is what too many people care about). Eh? My whole point that it is not. Did you read the above?
Yes. Passing the null often allows the expression that doesn't care. Your list of supported overrides is in itself longer than many expressions I have written.
How do you properly disable
x11Support? You set it tofalse. How you do that withnulls?
Just libX11. Yes, flags are more expressive, but they are more explicit, and have to be paid some attention during package updates, and that last part means they will not gain support in the next three years.
But if one almost never overrides things because hydra cache then, yes, all of that is purely academic.
Most high-volume Nixpkgs committers don't use a complex set of overrides (either because Hydra or because they want a set of expressions that receives the most community testing), so your arguments for USE flag — valid arguments — are arguing for an undesirable tradeoff.
unifying deps and arguments meets pushback, "Yes, but why?"
Maybe evaluation times, actually. Or evaluation RAM footprint. Maybe also the distant hope of having static typechecking for Nix one day.
and the rest actually requires human input. If you mean "the expressions need to be rewritten" then, yes, they do.
No, I mean that of the 8 remaining lines, you can generate zero with Nix. You cannot set maintainer, you do need human input for platforms, you cannot calculate the license in pure Nix.
Hm. That is quite an interesting idea, actually. But I don't see that working for complicated packages with intricate interplay between inputs and build phases, but most packages can be written this way, yes.
Maintaining the interplay between flags and build phases is something that a lot of people in Nixpkgs want to avoid. So this will have to live in a separate repo.
We-ell, if you can beat LibreOffice into submission without going mad, I can Just Commit It and I know from experience that people just take whatever LibreOffice version someone else updates. As long as it resembles a working LibreOffice version. And I also will gratefully accept work from someone else for LibreOffice updates.
In general if the expression is hard enough to get right even for one set of options, I would expect supporting more build variants to be even harder.
I don't see that happening in nixpkgs, though. Such a thing is just too far away from what the core of contributors cares about.
Nixpkgs packages have pressure to minimalism. We just need to steer it into a direction that is slightly more convenient to parse from outside. I mean, most Nixpkgs expressions are already there — if you pass them fake stdenv, you get all the valuable information there is (inputs, version, patches) with little effort.
In the context of
meta.broken, that probably means putting everything intometa.broken; maybe with optional-trace-generating wrappers around groups of conditions to hint whether the failure is likely to be a buildtime or a runtime one. Eh? But why? It isn't likenix-envdepends specifically onmeta.brokenor something. We can do pretty much whatever withmetaattributes. Why not do something fuzzable then?
I have been in a discussion whether some attributes in meta should not ever exist. A wrapper function which is just a trace-if-desired could be simpler to negotiate into Nixpkgs mainline, and easy to override in the downstream use-case.
7c6f434c
commented
6 years ago I think meta.broken is currently lazy enough.
oxij
commented
6 years ago @aneeshusa, right. This exactly my point in
It's shorter than your shortest
packageOverridesthing withnulls and it can also properly check any deps between flags (which.overridefails to do andnulls forbid you to even think about, btw).
nulls are evil.
oxij
commented
6 years ago Well, for example
helperArgNames(up to beginning 2016)
Hm. From what I can surmise from git log -S helperArgNames and #4210 the issue with builderDefsPackage was that it was more copy-pastey than plain mkDerivations for packages with small number of flags (which is the most packages of nixpkgs), that is. See e.g. cc9547dcf99f63e1046b25b9273bbc4b3999b341 or 8878e8ec05845b92fc5bca7f22c5bb0a0f19dad4. This is not the case with mkUFDerivation.
Any more examples? I seriously want to know why every other attempt failed. Please teach me what to search with. I looked at git log --numstat pkgs/build-support/ | grep -E '^\s*0\s*[1-9][0-9]*' | sed 's/^[^p]*//' | sort | uniq but there's nothing else (except already mentioned builder-defs) that looks like it might be the thing I want.
ATM I see no drawbacks to mkUFDerivation except marginally longer evaluation times.
I wonder is a bot could maintain
nixpkgs-atticrepository
If nix could partially evaluate and dump code as data (Guix has this for free because they do it in LISP, btw) mkUFDerivation would cost nothing and attic bot could be implemented pretty easily by just partially evaluating, dumping, and then deduplicating expressions from all-packages.nix on every master commit.
Yes. Passing the null often allows the expression that doesn't care. Your list of supported overrides is in itself longer than many expressions I have written.
In which case you could just dump all of it into required and get exactly the same lack of extensibility.
so your arguments for USE flag — valid arguments — are arguing for an undesirable tradeoff.
But I also argue that adding fuzzing on top of mkUFDerivation changes that tradeoff. Well, whatever. I see your point, you see my point.
Maybe also the distant hope of having static typechecking for Nix one day.
A-ha-ha. (It would be cool, indeed, but knowing how nixpkgs does things now: "a-ha-ha".) I do want to write an alternative interpreter (with incremental and partial evaluation) for nix in Haskell or Rust eventually, though.
you do need human input for platforms, you cannot calculate the license in pure Nix
Well, actually... You know about my fascination with autogenerating platforms already. But you could do that for licenses too. I actually did implement a license checker for checkMeta that checked that you didn't link (with buildInputs) against GPL packages in non-GPL packages. And got a ton, A TON, of evaluation errors :) The reality is that nobody actually cares about the licenses. If you start properly checking them nothing ever evaluates. But the moral of that story is that if people did actually care about the licenses, you could derive them from package deps too. Just sayin'.
We-ell, if you can beat LibreOffice into submission without going mad, I can Just Commit It and I know from experience that people just take whatever LibreOffice version someone else updates. As long as it resembles a working LibreOffice version. And I also will gratefully accept work from someone else for LibreOffice updates.
:) No thanks. But I'll ask in SLNOS. But the positive result is unlikely. :)
In general if the expression is hard enough to get right even for one set of options, I would expect supporting more build variants to be even harder.
I think fuzzing changes that.
I don't see that happening in nixpkgs, though. Such a thing is just too far away from what the core of contributors cares about.
Nixpkgs packages have pressure to minimalism. We just need to steer it into a direction that is slightly more convenient to parse from outside. I mean, most Nixpkgs expressions are already there — if you pass them fake
stdenv, you get all the valuable information there is (inputs, version, patches) with little effort.
... but no information about the options. Since options are the hardest part of an expression, it seems easier to me to maintain a parallel set of expressions that "borrow" src, patches and meta from master's expressions and do the rest themselves.
I have been in a discussion whether some attributes in
metashould not ever exist. A wrapper function which is just a trace-if-desired could be simpler to negotiate into Nixpkgs mainline, and easy to override in the downstream use-case.
I agree. Fuzzing, though.
7c6f434c
commented
6 years ago Well, for example
helperArgNames(up to beginning 2016) Hm. From what I can surmise fromgit log -S helperArgNamesand #4210 the issue withbuilderDefsPackagewas that it was more copy-pastey than plainmkDerivations for packages with small number of flags
helperArgNames were criticised separately (and later iterations builderDefsPackage were non-copy-pastey)
I wonder is a bot could maintain
nixpkgs-atticrepository If nix could partially evaluate and dump code as data (Guix has this for free because they do it in LISP, btw)
Well, Lisp gives you dumping the code, but not proper partial evaluation…
With Nix2.0 nix edit, though, it is possible to just grab the leaf package file without even parsing Nix code. Of course, there are packages that won't work like that, but they are a minority.
Yes. Passing the null often allows the expression that doesn't care. Your list of supported overrides is in itself longer than many expressions I have written. In which case you could just dump all of it into
requiredand get exactly the same lack of extensibility.
In the current style I can still pass null. Of course, with required I can pass something with an empty outPath…
so your arguments for USE flag — valid arguments — are arguing for an undesirable tradeoff. Well, whatever. I see your point, you see my point.
I think our value disagreement is mostly that I want to have my LibreOffice from a publically mirrored repository, so I won't simply use SLNOS for everything. Instrumentally, I would also prefer to have the package-database maintenance in Nixpkgs a viable shared resource.
Maybe also the distant hope of having static typechecking for Nix one day. knowing how nixpkgs does things now: "a-ha-ha".
Hey, that's the point that you do not need to change Nixpkgs much, you can just maintain a small list of weird places (outside Nixpkgs mainline).
you do need human input for platforms, you cannot calculate the license in pure Nix The reality is that nobody actually cares about the licenses. If you start properly checking them nothing ever evaluates. But the moral of that story is that if people did actually care about the licenses, you could derive them from package deps too.
It is a fun legal exercise just how optional your GPL optional dependency should be before you can say «the source tarball is MIT but once you enable libSomething support the binary is GPLv3+».
Anyway, the dependencies set a bound on the license, but they don't actually guarantee that the output license is exactly that.
In general if the expression is hard enough to get right even for one set of options, I would expect supporting more build variants to be even harder. I think fuzzing changes that.
Sorry, no. If you need to negotiate with build system to get one variant working, you need to negotiate even more about multiple variants working. Fuzzing may help you notice when it breaks, but when it breaks, it needs fixing, and the initial assumption is that this fixing is complicated.
I will gladly and gratefully accept a counterexample in the form of a working configurable LibreOffice package, but
No, thanks.
... but no information about the options. Since options are the hardest part of an expression, it seems easier to me to maintain a parallel set of expressions that "borrow" src, patches and meta from master's expressions and do the rest themselves.
Are you saying that adding options/use-flags everywhere more than doubles the effort?
I have been in a discussion whether some attributes in
metashould not ever exist. A wrapper function which is just a trace-if-desired could be simpler to negotiate into Nixpkgs mainline, and easy to override in the downstream use-case. I agree. Fuzzing, though.
What about fuzzing? You just replace «pick the warning» function with «these kinds of problems are just ignored».
oxij
commented
6 years ago Well, Lisp gives you dumping the code, but not proper partial evaluation…
But eval for LISP is 15 lines of LISP, so you can just write another interpreter right there... It will be slow, though.
In the current style I can still pass
null. Of course, withrequiredI can pass something with an emptyoutPath…
Fine. I see you want things to be dirty sometimes, I get the usefulness of that. mkUFDerivation needs to support nulls somehow. Will do.
I think our value disagreement is mostly that I want to have my LibreOffice from a publically mirrored repository, so I won't simply use SLNOS for everything. Instrumentally, I would also prefer to have the package-database maintenance in Nixpkgs a viable shared resource.
Wait, do I hear you saying that if I port mkUFDerivation just to make LibreOffice build sanely, the upstream will adopt the patchset?
Hey, that's the point that you do not need to change Nixpkgs much, you can just maintain a small list of weird places (outside Nixpkgs mainline).
Fair enough.
Anyway, the dependencies set a bound on the license, but they don't actually guarantee that the output license is exactly that.
True, but if you don't do any legal gymnastics most of the time you get no actual choice except GPL. (And I think it's a good thing.)
Sorry, no.
Well, ok, fair enough.
Are you saying that adding options/use-flags everywhere more than doubles the effort?
For packages with a bunch of options/use-flags? Of course. It can be 10x-100x more effort. Look at ffmpeg on master. It's so bad that master has two completely different expressions for ffmpeg-3.4 (ffmpeg generic, and ffmpeg-full). Options are hard. Hence mkUFDerivation.
What about fuzzing? You just replace «pick the warning» function with «these kinds of problems are just ignored».
It seems to me that this way you would just push currently explicit structures of possible problems like meta.broken, meta.knownVulnerabilities, etc into a function arguments. Some kind of modular handling for those things is due, though, I agree. How about replacing those things with meta.issues of { kind : String, condition = true : Bool, reason : String, ... } type, e.g.
meta.issues = [
{ kind = "broken.won't-build"; condition = x11Support; reason = "x11 support is broken ATM"; }
{ kind = "broken.won't-work"; condition = pulseaudioSupport; reason = "pulseaudio support is buggy"; }
{ kind = "vulnerable"; id = "CVE-XXXX-XX-XX-XXXX"; }
];? "."-separated strings can be made into lists to make it parse less. Common kinds can be made in lib functions like markBroken condition reason or whatnot.
Most of the OP seems to be pretty easy with that.
7c6f434c
commented
6 years ago Well, Lisp gives you dumping the code, but not proper partial evaluation… But
evalfor LISP is 15 lines of LISP, so you can just write another interpreter right there... It will be slow, though.
But Guile isn't LISP 1.5, it's larger and more complicated. But yes, you can go quite far; although it is enough for a parser to be available inside the language — Julia would be fine as well.
In the current style I can still pass
null. Of course, withrequiredI can pass something with an emptyoutPath… Fine. I see you want things to be dirty sometimes, I get the usefulness of that.mkUFDerivationneeds to supportnulls somehow. Will do.
Actually, requiring having meta and just allowing an empty outPath (up to: literally "") could be a better way. I just want global guarantees, after all (and I am not a SLNOS user and unlikely to migrate to SLNOS as the primary package source), so maybe I want tradeoffs incompatible with yours.
I think our value disagreement is mostly that I want to have my LibreOffice from a publically mirrored repository, so I won't simply use SLNOS for everything. Instrumentally, I would also prefer to have the package-database maintenance in Nixpkgs a viable shared resource. Wait, do I hear you saying that if I port
mkUFDerivationjust to make LibreOffice build sanely, the upstream will adopt the patchset?
There is my position and then there is overall reaction. And then there is usage for LibreOffice and usage in general.
If it turns out to be saner than now (not a very high bar, but a nontrivial one — it still has to work with the existing upstream build system) — I guess it has a very high probability to get adopted specifically in LibreOffice. Spreading it will still not be feasible.
I will just say that I am not rejecting a working LO update, whatever the style and merge the PR. I hope that the tolerance level for this specific package is high enough for noone to want to revert.
True, but if you don't do any legal gymnastics most of the time you get no actual choice except GPL. (And I think it's a good thing.)
«Most of the time» is a measure-dependent notion…
Are you saying that adding options/use-flags everywhere more than doubles the effort? For packages with a bunch of options/use-flags? Of course. It can be 10x-100x more effort. Look at
ffmpegon master. It's so bad that master has two completely different expressions forffmpeg-3.4(ffmpeg generic, andffmpeg-full). Options are hard. HencemkUFDerivation.
I think they just default in the opposite direction.
What about fuzzing? You just replace «pick the warning» function with «these kinds of problems are just ignored». It seems to me that this way you would just push currently explicit structures of possible problems like
meta.broken,meta.knownVulnerabilities, etc into a function arguments. Some kind of modular handling for those things is due, though, I agree. How about replacing those things withmeta.issuesof{ kind : String, condition = true : Bool, reason : String, ... }type, e.g.meta.issues = [ { kind = "broken.won't-build"; condition = x11Support; reason = "x11 support is broken ATM"; } { kind = "broken.won't-work"; condition = pulseaudioSupport; reason = "pulseaudio support is buggy"; } { kind = "vulnerable"; id = "CVE-XXXX-XX-XX-XXXX"; } ];? "."-separated strings can be made into lists to make it parse less. Common kinds can be made in lib functions like
markBroken condition reasonor whatnot.Most of the OP seems to be pretty easy with that.
Hm. I like it. Not sure I want to push for such a change before 18.03, though. This «kill all assert» thing has the benefit of clearly not being doable quickly, so nobody will try until the release.
danbst
commented
6 years ago I've proposed a similar PR a while ago https://github.com/NixOS/nixpkgs/pull/32019. This would fix https://github.com/NixOS/nixpkgs/issues/11226#issuecomment-345906810 and https://github.com/NixOS/nixpkgs/issues/31884
It took only 20 files to edit to make nixpkgs evaluatable on non-cross setup. Also, note that some asserts are used as license checks, so can't be treated as "broken"
7c6f434c
commented
6 years ago Also, note that some asserts are used as license checks, so can't be treated as "broken"
Hm. I wonder if creating lib.licenses.unsatisfied (like unfree only
worse) and setting it when the EULA has not been accepted is a good idea
(asserts are annoying for overrides and for checking meta).
Asserts inside stdenv are OK, of course (I started in pkgs to
exclude lib but forgot about stdenv).
7c6f434c
commented
6 years ago Updated the description re: license check and re: stdenv exclusion
Ekleog
commented
5 years ago @7c6f434c Care to turn this into a proper RFC, so that a “definitive” decision could be made?
stale[bot]
commented
4 years ago Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:
siraben
commented
3 years ago Still relevant.
piegamesde
commented
2 years ago Closing this. This issue is so outdated, we'd need somebody motivated to evaluate the current situation and open a new issue. I don't think this will require an RFC though.
Issue description
A lot of Nixpkgs packages currently use
assert. Most of the uses ofasserthad been introduced beforemeta.brokenhas been implemented. Usingmeta.brokenwith an expression condition has benefits, for example, it can be queried viameta.availableand catching assertions requiresbuiltins.tryEval, and it is always better not to add more uses oftryEval. See also #36226. Apparently enthusiastically supported by @Ericson2314Tracking
My model of assertions: an assertion happens in a
.nixfile insidepkgs/in a line not starting with a#in the beginning of the line or after one of(){}, before end of line (with condition on the next line) or one of(){}.Sanity check:
(files without assertions as defined above that still have
assertas a alphanumeric string as agrepword haveasserteither as a part of a dash-separated name, or inside quotes)stdenvis a separate thing, the uses ofassert— like inlib, which is outsidepkgs— do not have a convenientmeta.brokenaround.Re: asserts for licenses: should there be
lib.licenses.unsatisfied, likeunfreebut without a permanent override available (I guess$NIXPKGS_ALLOW_UNSATISFIED_LICENSEis enough for all debugging purposes)? The motivation is the same as with things that go tometa.broken: inspection ofname/meta`.Progress countdown:
→ 528
List of files: