Consider building unfreeRedistributable Packages

[mkg20001]

Per definition those packages are redistributable without any legal problems (or at least should be, otherwise I feel like the license is wrongly described)

So we could build those as well, because there doesn't seem to be a reason not to

[mkg20001]

The required change would be probably this one

nixpkgs.config.allowUnfreePredicate = (x: if x ? meta.license then x.meta.license.shortName == "unfreeRedistributable" else false);

[stale[bot]]

I marked this as stale due to inactivity. → More info

[mkg20001]

Still not resolved

[domenkozar]

As per https://github.com/NixOS/nixpkgs/issues/15162 we might actually also want to whitelist what drivers to install into an ISO.

As per https://discourse.nixos.org/t/pytorch-and-cuda-torch-not-compiled-with-cuda-enabled/11272 we could then support data science by shipping CUDA-enabled binaries by default.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/comparing-nix-and-conda/11366/65

[petrosagg]

I hit the same problem today after getting the package for materialize which is BSL licensed merged. The software license "grants you the right to copy, modify, create derivative works, redistribute, and make non-production use of the Licensed Work." so it should be perfectly fine to build and distribute binaries through Hydra.

I'd like to help resolve this problem, is the direction set forth in this issue and #97789 still the way to go here?

[mkg20001]

I think so, I'm just not sure which way to implement the flag to enable it (ref https://github.com/NixOS/nixpkgs/pull/97789#issuecomment-691194274)

[Atemu]

All Licenses now have a redistributable attr. (See PR above)

The work that remains is:

• [ ] introduce an allowRedistributable config option
• [ ] Allow redist even if unfree in Hydra jobsets

[K900]

Any updates here?

[mkg20001]

Redistributable is now a thing

nix-repl> lib.licenses.free
{ deprecated = false; free = true; fullName = "Unspecified free software license"; redistributable = true; shortName = "free"; }

nix-repl> lib.licenses.free.redistributable
true

Hydra building and caching redistributable unfree things is still pending

[mkg20001]

The quick fix for hydra would be this:

nixpkgs.config.allowUnfreePredicate = (x: if x ? meta.license then x.meta.license.redistributable else false);

It only checks redistributable but this is true automatically for all free licenses and being redistributable is all that matters in the case of hydra builds, I suppose.

[K900]

Is anything preventing us from making that change on Hydra, like, now?

[mkg20001]

Not really, it could just be done

[grahamc]

We'd need sign-off from the NixOS Foundation which is responsible for supporting the infrastructure, and might have ethical objections. One concern they've raised is NixOS should be fully usable without any unfree software, and allowing it in Hydra could make NixOS an inadvertently unfree Linux distribution.

[K900]

How does proprietary firmware fit into this? That's already distributed in a bunch of places, so NixOS isn't strictly fsf!free...

[grahamc]

I don't have a good answer about that.

[mkg20001]

I propose building every redistributable derivation on hydra, but keeping the allowUnfreePredicate on a regular machine to check for whether a package is free, so vanilla nixOS is still mostly free but the user can enjoy the cache benefits for unfree redistributable packages aswell

Note that this would also fix the issue of having to label some drivers as free to get them to build on hydra, so allowUnfree=false would finally do what it says

[K900]

It should also be possible to run at least a subset of tests an extra time without allowing unfree?

[grahamc]

It should also be possible to run at least a subset of tests an extra time without allowing unfree?

Perhaps one option is having Hydra's jobs for nixpkgs allow unfree and leaving NixOS's jobs as-is.

[mkg20001]

Enabling hydra to build redistributable unfree jobs/tests would allow testing for unfree packages aswell, which might not even be such a bad idea. But then some "is vanilla nixOS still fsf free" tests would be need

Edit: Which isn't really the case right now either because of the linux unfree firmware that is shipped in some installation images.

[grahamc]

vanilla nixOS still fsf free

Note that our concept of "Free" is not really documented and doesn't have a well defined philosophy, and already does not mirror FSF or OSI Free. In other words, there are lots of related improvements to be made.

A related issue: https://github.com/NixOS/nixpkgs/issues/83433

[Atemu]

No matter where the discussion of what is considered free and what isn't leads us, we have a status quo on that and don't need to change it in order to redistribute unfree software on hydra.

What we need to do, as @mkg20001 said, is to create an explicit assurement that regular NixOS stays still as "free" as before because hydra wouldn't catch a deviation anymore otherwise.

Other than that, I see no issue from a user perspective. The only issue would be the ethics of distributing unfree software via the cache.

IMO hydra also serving unfree software doesn't impact any user in a negative way. People that only allow free software on their systems have unfree software forbidden in their local eval before anything gets pulled from hydra and will see no change. For them it's just an output path they will never access because of systematic prevention.

A much bigger issue is what happens when a derivation is mistakenly marked as redistributable or becomes unredistributable in the future. Is it possible to remove an output path from the cache after the fact?
This could cause legal trouble for the NixOS foundation.

[grahamc]

hydra also serving unfree software doesn't impact any user in a negative way.

Some additional consideration is building unfree software takes away time from building free software, and the storage costs money over time. These may not be compelling points.

A much bigger issue is what happens when a derivation is mistakenly marked as redistributable or becomes unredistributable in the future. Is it possible to remove an output path from the cache after the fact? This could cause legal trouble for the NixOS foundation.

This is already true. For an example from just today: #145607 / #145766. Typically issues around redistribution are not complicated to deal with: we get a nastygram asking us to take it down and we do, and that settles it.

[domenkozar]

If someone creates a flake that builds only unfree redistributable software from nixpkgs, I'll sponsor and setup linux/macos builders and a big cache.

If foundation decides to do that upstream at some point we can just switch over :)

[mkg20001]

@domenkozar I made one https://github.com/mkg20001/redistributable-nix

I don't know if eval needs all packages successfully evaluating for your CI to work, otherwise I'd have to add some try-catch iteration to it, similar as hydra has it right now.

[domenkozar]

It seems that flakes don't have a --keep-going command.

Maybe we can give https://github.com/nix-community/nix-eval-jobs a try?

[mkg20001]

A not so unrelated side-note: Currently hydra doesn't build things with security vulnerabilities either, which, similar to the idea of just building everything* and letting the user choose, might make sense to enable aswell

[Atemu]

That does not sound like a sensible policy either, Hydra has lots of packages that contain security vulnerabilities because they've only been discovered and/or declared after they've been built.

[domenkozar]

@mkg20001 I'm hoping to get this one done in a few weeks, I'm trying to wrap up a big chunk of work I've been working on for the past two months :)

[domenkozar]

Ordered!

[domenkozar]

I'm currently stuck trying to evaluate all unfree but redistributable derivations: https://github.com/cachix/nixpkgs-unfree-redistributable

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/petition-to-build-and-cache-unfree-packages-on-cache-nixos-org/17440/2

[domenkozar]

This is almost ready, mostly finishing up nix-darwin changes.

[SuperSandro2000]

I'm currently stuck trying to evaluate all unfree but redistributable derivations: cachix/nixpkgs-unfree-redistributable

The issue mentioned there should be fixed with https://github.com/NixOS/nixpkgs/pull/156082

Let me know if there are other tasks/eval errors.

[samuela]

Having unfree packages built and cached somewhere is absolutely essential for doing any kind of machine learning on NixOS, as everything that relates to CUDA is marked unfree. And the builds of these packages are really quite onerous for end-users, eg. tensorflowWithCuda takes 80 CPU-hours to build.

[zimbatm]

I didn't know about Domen's effort and have started another project: https://github.com/numtide/nixpkgs-unfree. It's synching periodically every day, building both nixos-unstable and nixpkgs-unstable, and publishing them to https://nixpkgs-unfree.cachix.org/

[cosmojg]

What's the status of this issue?

[Madouura]

So what's the verdict? Any updates?

[SomeoneSerge]

@Madouura I think it won't be a lie if I said that the neither hydra.nixos.org nor cache.nixos.org are going to touch unfree packages, especially those marked as sourceProvenance=binaryNative.

This doesn't mean "no support" yet. There have been experiments like nixpkgs-unfree{,-redistributable}.cachix.org and cuda-maintainers.cachix.org, and there's clearly interest. For the sake of transparency, one potential direction might be the nixpkgs opencollective and a space in the nix-community github organization.

Say, are you in any of the nixos dev or cuda matrix channels?

[Madouura]

Yes, I'm in the dev channel. I can join the cuda one as well.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/help-understanding-package-paths-in-nix-store/36682/5

[obadz]

zerotier (whose bsl11 license is marked as free = false; redistributable = true) no longer builds on my cloud host (RAM = 1GB, Swap = 1GB) so now I've got to mess around with building locally and nix-copy-closure gimmicks every time I upgrade.

Long story short, :+1: to Hydra building unfree redistributable packages.

[emilazy]

Just throwing in the opinion I’ve stated elsewhere here: We already don’t do a fantastic job at complying with the letter of FOSS licences and rely somewhat on general community goodwill to paper over our failure to retain copyright notices and licence texts. Having Hydra and its administrators essentially agree to arbitrary EULA texts that can run into several pages and that were designed by potentially‐litigious corporations not trying to help create a software commons would not, in my eyes, be an improvement, even separate to any ideological opposition to non‐free software.

A version of this proposal that I would find more acceptable would involve having a specific curated list of unfreeRedistributable licences that have been specifically approved for Hydra, rather than blanket allowance. I think that the curated list should also have lack of use restrictions (as opposed to modification restrictions) as a criterion, both because of the potential increased risk for our infrastructure when it builds and tests packages, and because I don’t think we really have a good way of informing users of use restrictions at present.

All that said, I really think we should make sure we can handle the basic task of retaining and distributing licence notices before we think about dabbling in trying to comply with far more stringent legal obligations.

[SomeoneSerge]

specific curated list of unfreeRedistributable licences

Agreed, if this were to go we'd need a list of licenses we consider "safe enough". That's the direction we tried with an external builder at https://github.com/NixOS/nixpkgs/blob/e6e5ee6216d2761c2ad2146287c91b83b9430037/pkgs/top-level/release-cuda.nix#L17-L28

Having Hydra and its administrators essentially agree to arbitrary EULA texts that can run into several pages and that were designed by potentially‐litigious corporations not trying to help create a software commons would not, in my eyes, be an improvement, even separate to any ideological opposition to non‐free software.

Let's be honest, it'd be a lie to talk of "compliance" or of "agreeing to terms" in this context. These EULAs aren't designed to be "complied with", they're designed to create and maintain a capacity for non-physical violence.

In the specific case of CUDA, it is being redistributed by every imaginable distribution (julia even distributes a patchelfed driver, which the EULA explicitly prohibits to distribute in any form whatsoever) based on some private sign offs from nvidia. I've no doubts that if nvidia wished to sue any of these said distributions, they could and they would, regardless of whatever prior communication they have had.

Rather than talking about "agreeing to EULAs", I think we're rather openly talking about one or another party, providing a cache (the nixos hydra currently owned by the foundation being one candidate), taking the risk of being sued, acting either under the assumption that the corporation would abstain from resorting to violence, or that they'd be able to (financially) sustain a trial. One could consider whether or not to take such a risk depending on whether or not that'd accommodate use-cases that attract the "useful" kind of contributions (e.g., whether it'd be "useful" to attract HPC users). We try to comply to the best of our abilities to reduce the risk for Nixpkgs: e.g., in the specific case, we tried contacting nvidia through several channels, ultimately getting no attention whatever, and we never distributed the drivers in the external/third-party caches, and we try to communicate to the end user that they should acquaint with the EULAs and that it's a risk they are taking.

Also, to keep a perspective, let's remember that the binary cache is only there for efficiency: the Nixpkgs expressions could be consumed by the end user without any substituters (thus without the substituters "redistributing" any artifacts or committing to any imposed contracts), and often enough the result would be equivalent to substitution, except they'd generate more heat. Arguably, the optimization offered by substituters is not a "bad thing" for "the society". If the legal framework of one or another jurisdiction hinders such optimizations, maybe it's the legal framework that needs adjustment, and maybe we should work with the upstream projects towards making peaceful co-existence possible.

[eclairevoyant]

based on some private sign offs from nvidia

If that's really the blocker then let's get that for nixpkgs as well. The rest of this response is off-topic.

EDIT: since you mentioned that nvidia was contacted already and that failed, we should probably drop discussion of nvidia packages unless we can comply with said EULA.

[emilazy]

I think I am basically in agreement with @SomeoneSerge on the fundamentals here even if we see the issue through different lenses, despite my personally being a FOSS licence compliance stickler. Changing copyright law is certainly beyond the scope or means of the NixOS project, though, for better or worse.

If the CUDA EULAs themselves were not considered acceptable but we had a public sign‐off from NVIDIA that seemed acceptable to the community, then certainly I’d consider CUDA packages to be reasonable candidates for any curated list of Hydra‐acceptable non‐free licences. I’m not sure whether this would be necessary or not, depending on what criteria (if any) we could agree on for acceptable non‐free licences or how onerous the EULAs are in the first place (they’re rather tl;dr). But I personally wouldn’t feel good about informal backroom permission even if they would give it to us.

[SomeoneSerge]

EDIT: since you mentioned that nvidia was contacted already and that failed, we should probably drop discussion of nvidia packages unless we can comply with said EULA

I didn't emphasize this enough, but I don't think this is a Nixpkgs issue. This is a discussion to be had be anyone who volunteers to provide a cache. In case of the nixos hydra, it was a question considered a while ago by the foundation (rejecting the idea at that time), and later, perhaps, it'd be a matter for an Association/Assembly/etc to pick up.

[samuela]

If the CUDA EULAs themselves were not considered acceptable but we had a public sign‐off from NVIDIA that seemed acceptable to the community, then certainly I’d consider CUDA packages to be reasonable candidates for any curated list of Hydra‐acceptable non‐free licences.

FWIW I consider the precedent of nearly every other distribution republishing CUDA as a sort of de facto public sign-off from NVIDIA. I genuinely believe that it is in their best interest to maximize distribution channels for their software, and based on their behavior I glean that they see things similarly. It's my personal guess that some lawyer was tasked with writing up a generic EULA a decade ago and since then no one has fussed too much about it, internal or external to the company.