search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
18.32k stars 14.3k forks source link

Unfree software is unintuitively hidden for new users #17126

Open grahamc opened 8 years ago

grahamc commented 8 years ago

Issue description

New users try to install unfree software, search for it with nix-env -qaP, and find it missing. Some more enterprising users search nixpkgs on GitHub and find it is definitely there but don't know how to find it.

Example chat where this happened: https://botbot.me/freenode/nixos/2016-07-20/?msg=69976623&page=9

Perhaps a way to improve this would be show the unfree software in the list, but with a note about it being unfree.

Steps to reproduce

  1. Be a new user who is evaluating NixOS/Nixpkgs
  2. Try to install hipchat
  3. Find it isn't in nix-env's list
  4. Confirm it is in nixpkgs via GitHub
  5. ??? until they find { allowUnfree = true; } or ask in IRC
grahamc commented 8 years ago

re:

I think this should be revisited. I think we're doing users a disservice by making it difficult to find the software they're wanting to choose. Right now we're choosing to appease strict FOSS users over reducing the headache for newcomers.

I'd rather the onus on more advanced users to get rid of it, than new users to get started with the tools they need and are familiar with.

copumpkin commented 8 years ago

Agree. I think this is largely an incidental consequence of not wanting Hydra to do unfree stuff, but I think the mechanism should be a made a bit more fine-grained to be friendlier to users.

vcunat commented 8 years ago

I tried, and I think it's hard to find strong consensus on this point.

Personally I'm comfortable that official nixpkgs accepts significantly wider range of packages than it's usual for most other large distributions. BTW, we're slowly shifting to specifying packages directly by attribute names, and in that case the user gets a useful evaluation error.

grahamc commented 8 years ago

BTW, we're slowly shifting to specifying packages directly by attribute names, and in that case the user gets a useful evaluation error.

Yeah, this error is really helpful if they get far enough to try to install it that way. A bit tough, though, to get that far when a nix-env search tells you it isn't there.

abbradar commented 8 years ago

I think we really do need to show the user somehow that there are other packages. I've taken a liking to this suggestion, option II. Other solution is to show such packages but place big red label "UNFREE" next to them.

EDIT: after some thoughts I like my solution less, because this heavily depends on what do we consider "promotion". The other suggestion, however, is very unobtrusive IMHO and doesn't even name anything, just says that there are matches.

PlasmaShift commented 8 years ago

Put the allow unfree information where people go and look at how to install software

When they generate your Nixos configuration and have kde and wget " I think that is the one package they show as an example to install" commented out put allow unfree in a comment there as well

When they look at the manual on how to install packages put a note somewhere really close so they see it. Probably at two places, one for nix "nix-env -i ... or nix-env -qa ..." and one for Nixos "inside your configuration.nix

when they search on Nixos.org packages in the description make a note, ex nonfree -- rest of description in package name or attribute name it would would get in the way of normal operations. I am not concerned about nonfree when I am greping around nix-env -qa

These are the places I would go/went to find out how to install applications This way would also not interfere with the nix way as describing your os

As a side note it would also be nice if when you generated configuraiton.nix you would get a stateless user example.

peti commented 8 years ago

The real issue is that nix-env -qa searches are fundamentally broken and don't find literally thousands of useful packages that we have. This phenomenon is not limited to unfree software.

domenkozar commented 8 years ago

This came up again today on irc:

22:50:36           tosmi | hi, i'm a nixos newbie and unable to install the citrix-receiver package   
22:51:09           tosmi | nix-env -qa | grep doesn't find citrix-receiver               
22:51:34           tosmi | i suppose because it depends on an EULA protected blob...
domenkozar commented 8 years ago

I think Nix UI redesign is a good opportunity to fix this.

grahamc commented 7 years ago

Today a user on IRC (ToxicFrog) spent four hours trying to package Sublime3 only to find out it was hidden behind unfree.

grahamc commented 7 years ago

Today another user on IRC (riptawr) spent quite some time trying to figure out discrepancies between nixos-rebuild (as root) and nix-env (as his user) due to his profile not having allowUnfree.

vcunat commented 7 years ago

It's very common that people are confused from nix-env not using nixpkgs config from nixos config. I put it into a separate *.nix file and import in both; I haven't seen a better approach yet – maybe it could be suggested in the docs. (But it seems a digression from the original topic.)

Eilie commented 7 years ago

Hello, I'm a new NixOs user and I've spent an hour trying to figure that out and then got help on IRC. Would be great to have this resolved in some intuitive way.

alecmev commented 7 years ago

Just spent 30~60 minutes trying to figure out why I can't find google-chrome via nix-env -qa, even though it's clearly in the repo, until I've been pointed to this issue by kind people in #nixos. Maybe add a quick note about allowUnfree somewhere high up in the manual, at least in Chapter 2 and Chapter 8, until a warning is implemented? ~Also, I don't see allowUnfree documented anywhere, it juts shows up in random issue descriptions and config excerpts~ (e.g. this wasn't obvious, spent extra 15 minutes on that).

vcunat commented 7 years ago

For reference, the documentation is in nixpkgs manual.

vcunat commented 7 years ago

Also note that restriction of unfree stuff is primarily a "political decision" IMHO. We've had lots of discussions, e.g. on https://github.com/NixOS/nixpkgs/pull/5628.

7c6f434c commented 7 years ago

I think the discussion right now is not as much a plea to reverse that decision as to declare it more visibly. Proudly declaring this decision is not against this decision, and those who want the override will know what to look for.

For the record: I support requiring an explicit action for unfree packages, and I do have some unfree packages installed. I also think that mentioning this filter close to the beginning of NixPkgs manual is a good idea.

ToxicFrog commented 7 years ago

My preference would be for package search (both the website and nix-env -qa) to list nonfree packages by default, but display some kind of indication that they're nonfree. nixos-homepage PR 120 does this for the website, but is blocked on license information being included in the search page. (I noted on that PR that this is the approach taken by other free-software distros such as Debian.)

Failing that, there should at least be (by default) a prominent warning that non-free software isn't being included in the search; a large part of the problem here is that it's not at all evident to the user that their query isn't actually finding everything. An option to enable it would also be nice (manually searching the nixpkgs repo is pretty awful UX), but making sure the user knows that they need to do that at all is a necessary first step.

I wasted a lot of hours trying to repackage nonfree software like Steam, Chrome and Sublime3 before someone on IRC pointed out that the package search was lying to me and I needed to instead search the nixpkgs repo directly. Someone else repeats this experience on a regular basis, as evinced by this very thread.

vcunat commented 7 years ago

Suggestions on showing unfree packages in searches by default are consistently disliked by some core devs (Eelco, Rob, 7c6f434c above, maybe more), so I don't feel like pushing that point anymore.

I like the approach of mentioning the fact that they're hidden on some (more) prominent place. By itself that would seem more like propagating "free" SW than "unfree". However, I don't have a good idea about what parts of docs/website are best visible to newbies.

ToxicFrog commented 7 years ago

For my part I tended to lean heavily on https://nixos.org/nixos/packages.html

Even if they aren't shown by default there should be some way to enable them rather than just shrugging and going "use github search then".

7c6f434c commented 7 years ago

I think all the way that allow installation ($NIXPKGS_ALLOW_UNFREE and Nixpkgs config) enable search, too.

ToxicFrog commented 7 years ago

Good point. That works for people using the command line, and in that case I think the suggestion on #5628 of including allowUnfree = false in the autogenerated templates, with comments, will help a lot with making people aware that the option exists.

That said, the website is both a nicer UI and much faster than nix-env -qa even on a fast machine; it shouldn't be a second class citizen in what it allows you to search for.

Perhaps the answer is just to add "include unfree software" and "include broken packages" checkboxes next to the search box, and default both to off?

7c6f434c commented 7 years ago

I wonder if regardless of our current discussion it is a good idea to make it easy to generate a JavaScript-only local-HTML-file package list with search. Of course, it will follow the brokenness/insecurity/lack-of-freedom choices set by the user.

The problem with package list on the site is that it is a list of Hydra jobs. Of course, Hydra won't build things where it is unclear whether they are OK to distribute.

jcrben commented 7 years ago

Perhaps an interactive prompt when installing could be a compromise? Also spent some time trying to figure out what was going on before hopping on IRC. Not all beginners are up to ask on IRC. Note also that this is documented in the nixpkgs manual but not, as far as I can tell, in the nix manual, which is where I started...

7c6f434c commented 7 years ago

Technically, this is not a Nix-level decision, so it is not in the Nix manual — the license attribute lives in NixPkgs, the license check is performed by code in NixPkgs repository without any special support from Nix, etc.

I somewhat like being able to use NixPkgs for free software with no NixPkgs config at all, so I can understand unwillingness to put warnings unless configured.

Maybe enumerate the natural places like NixPkgs package search where adding word «free» and a footnote «unfree packages need to be explicitly enabled and are never built on Hydra» would make sense?

MaikKlein commented 7 years ago

It was really confusing for me.

I know that I enabled nixpkgs.config.allowUnfree = true; in my configuartion.nix but I was unable to find packages with nix-env -qa.

I was almost convinced that nixos was broken. Also allowUnfree is not in https://nixos.org/nixos/options.html

domenkozar commented 7 years ago

Another instance: https://www.reddit.com/r/NixOS/comments/6mtt5d/how_to_install_atom_on_mac/dk4gy3v/

butterflya commented 7 years ago

In this message I discuss the causes of this problem and some possible solutions.

Causes

There are a few causes of this problem.

The Nixpkgs manual does contain this information, but that makes it appear as if Nix and Nixpkgs are written by two independent entities, which have never even heard of each other, but in fact the manual is written largely by the same person.

The various manuals need to be heavily cross-linked and intra-linked to make them more useful. In at least 20 places I would expect the word "available" to be hyperlinked.

Solutions

I recommend to implement all solutions until the number of questions on IRC about this topic goes to zero.

ToxicFrog commented 7 years ago

@butterflya: This should also be fixed on the nixos.org package search (which a lot of people, including me, use in preference to nix-env -q because it is much faster). There's some thoughts on this earlier in the thread, but maybe it should be split out into a different issue -- there's basically three underlying problems there:

I suspect that fixing 2 and 3 on that list are prerequisites for not having fixes to 1 rejected.

E: oh hey, there are existing bugs for this on the nixos-homepage repo that I already mentioned earlier in this thread and then forgot about. Whoops.

butterflya commented 7 years ago

@ToxicFrog Don't you think it's a little bit weird that on NixOS there is this great module system and they could just have implemented the exact same service behind the nixos.org package search on NixOS and even enabled by default such that it would be indexed during idle time?

The only reason nixos.org package search exists, is because they want to spy on you.

copumpkin commented 7 years ago

The only reason nixos.org package search exists, is because they want to spy on you.

Come on man, I agree with you that these packages shouldn't be hidden by default, but stuff like this just makes it harder to take anything else you say seriously. This is a FOSS project full of folks working on it in their free time. We're not a shady cabal with ulterior motives. The search is there because someone felt like implementing it because they thought it would help the project.

butterflya commented 7 years ago

@copumpkin Point me to a privacy policy claiming that all search queries go to /dev/null and I might start to take you seriously (if you like to do ad hominem arguments, two can play that game).

copumpkin commented 7 years ago

Sigh. Look at the source to https://nixos.org/nixos/packages.html and tell me what you see. Note that it's just grabbing a gzipped tarball of JSON and processing it on the client side. The server never sees individual queries. The source to that page is here.

Granted, there's no guarantee that the deployed webpage and the underlying source code will stay in sync, or that the html won't silently change to record your inputs, but that would be work and frankly I doubt anyone in this project gives a damn about what packages folks search for. We should probably get a privacy policy someday, but accusing the project of spying on you when it's trivial to see that it isn't seems silly.

ToxicFrog commented 7 years ago

@butterflya The search queries are never sent to the server, but are executed entirely in the browser using a package database downloaded when the page first loads. You can trivially verify this for yourself with any combination of source inspection, browser debug tools, and packet capture that you care to use. The NixOS option search page works the same way.

U007D commented 7 years ago

This is me. Right now. After a couple of hours I did stumble across nixpkgs.config.allowUnfree = true;, but even after adding it to my configuration.nix, I still cannot get nix-env to show me all packages (eg. google-chrome is nowhere to be found), and I still don't know why (even after reading this thread, unfortunately).

I can't claim to understand why "the FOSS contingent" would object to unfree software being shown under an Unfree heading. I'll just accept that this is a thing.

What would help me right now, and users like me in the future is for the tool to behave as it does, with a note (make it silenceable if the experts don't want to see the reminder) stating that unfree software is not being shown and the step(s) required to show them. No kidding--I have read through most of the chapters of the manual (everything through ch. 12, and browsed the rest) and see no mention of how to list all (including unfree) packages.

Do a find in your browser on https://nixos.org/nixos/manual/index.html right now, and you'll find there is no mention of allowUnfree anywhere on the page!

I'm 3 hours into my quest now, and still without a solution. NixOS is a great idea, but easily fixable issues like this make the initial experience tough to stick with. Lucky I'm stubborn, I guess... ;)

U007D commented 7 years ago

Solved. For anyone else running into this problem:

@butterflya's description of possible solutions mentions a 2nd manual, namely the nixpkgs manual (https://nixos.org/nixpkgs/manual/), which you may not have known about (I did not).

Unlike the nixos manual (https://nixos.org/nixos/manual/index.html), which does not mention allowUnfree, nixpkgs manual does explain how to get this working. If you're like me, you've had a long search just to get here, so here is the relevant part:

Nix comes with certain defaults about what packages can and cannot be installed, based on a package's metadata. By default, Nix will prevent installation if any of the following criteria are true:

The package is thought to be broken, and has had its meta.broken set to true.

The package's meta.license is set to a license which is considered to be unfree.

The package has known security vulnerabilities but has not or can not be updated for some reason, and a list of issues has been entered in to the package's meta.knownVulnerabilities.

Note that all this is checked during evaluation already, and the check includes any package that is evaluated. In particular, all build-time dependencies are checked. nix-env -qa will (attempt to) hide any packages that would be refused.

Each of these criteria can be altered in the nixpkgs configuration.

The nixpkgs configuration for a NixOS system is set in the configuration.nix, as in the following example:

{
 nixpkgs.config = {
   allowUnfree = true;
 };
}

However, this does not allow unfree software for individual users. Their configurations are managed separately.

A user's of nixpkgs configuration is stored in a user-specific configuration file located at ~/.config/nixpkgs/config.nix. For example:

{
 allowUnfree = true;
}

(Note: you may have to create the ~/.config/nixpkgs folder).

After making both of the above changes, listing available packages (nix-env -qaP '*' --description) will now include unfree packages.

I hope that is helpful for someone.

zoosky commented 7 years ago

Great. I was in this journey too a few months ago.

LisannaAtHome commented 6 years ago

Another one: https://botbot.me/freenode/nixos/2017-12-27/?msg=95086544&page=3

lboklin commented 6 years ago

To add to this; for one-offs, before you go enabling unfree packages system- or user-wide, I recommend simply setting the NIXPKGS_ALLOW_UNFREE environment variable to 1, like so: NIXPKGS_ALLOW_UNFREE=1 nix-env -iA nixpkgs.steam (or export NIXPKGS_ALLOW_UNFREE=1 for persistence within the shell session).

bbigras commented 6 years ago

I'm one of the confused new users. This morning I saw "Nix 2.0 Released" on various websites and I wanted to give NixOS a try.

I wanted to install dropbox. It wasn't showing up with nix-env -qaP '*' --description nor on https://nixos.org/nixos/packages.html but weirdly I found https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/networking/dropbox/default.nix using google. I was wondering if it wasn't available on the stable channel or something like that and I finally saw license = licenses.unfree;. I ended up here after another google search.

Thanks to @U007D's comment I was able to figure out how to use { allowUnfree = true; }.

alexfmpe commented 6 years ago

Just wasted some time over this. I don't understand why the "unfree" message only shows when using attribute names. I get the "don't promote unfree" policy, but would we be violating it by switching the message in these cases from error: selector 'whatever' matches no derivations to error: selector 'whatever' matches only unfree derivations ?

copumpkin commented 6 years ago

@alexfmpe I think there are two issues at play here that are easy to confuse in this issue:

  1. Hydra doesn't want to build unfree software (or host it on cache.nixos.org)
  2. Some folks in the community don't want to encourage use of unfree software

I believe the current unfree solution was added initially to address point 1, and then it incidentally also covered point 2 and people didn't care enough to change that. But I still don't think point 2 is a primary goal, and if we can come up with a sensible mechanism to ensure that Hydra doesn't have to worry about distributing unfriendly licenses, I don't imagine most of the community would complain about the lack of point 2 (or making point 2 an opt-in version, since most users don't feel that strongly about licenses and just want software to work).

So in short, the unfree-blocking mechanism for Hydra could certainly be improved, but someone would need to do it.

7c6f434c commented 6 years ago

@copumpkin what you say is implementable by just setting allowUnfree to true by default and overriding it on Hydra. Of course, with multiple ways to set it needed for different operations, it will destroy option 2, not just make it opt-in — and this would be bad.

I hope that the new nix UI wouldn't have any operations that care about the derivation names as opposed to attribute names.

For search reasons — maybe we should permit evaluating name in addition to meta. Will it be enough?

cx405 commented 6 years ago

I am new to nixos (just few months, but using it as main system now), and this is how to search for unfree packages:

nix-env -qaP --arg config '{ allowUnfree = true; }' | grep <package_name>

This shows all occurrences and lists packages in all accessible channels.

cdyson37 commented 6 years ago

Just had exactly the same experience as the other n00bs. Couldn't find it with nix-env, couldn't find it with the online search, but stumbled across it in github. And now I'm here :)

Perhaps a stopgap solution would be to ask the user when running the magic install script whether allowUnfree should be set?

cx405 commented 6 years ago

@cdyson37 If you remove the "--arg config {}" option in my post above, it will work normally. So this feature is already implemented, just not behind an extra trigger. There is new "nix" interface that supports search.

I assume the best way is to create the built-in option, which would in the end - add the option that I provided above. So, in a curtain way, - you can already use it today.

lexleogryfon commented 6 years ago

I`m new user, and prefer to use https://nixos.org/nixos/packages.html over nix-env for package search, because of beautiful UI. I've spent months with thought that there is no good commercial packages packages like vscode, sublime on nixpkgs. That makes impression that nixpkgs less mature and provide less choices than Arch AUR, or Ubuntu. Recently accidentally discovered that mentioned packages are available as unfree. Please let newcomers know about all available packages in https://nixos.org/nixos/packages.html and merge https://github.com/NixOS/nixos-homepage/pull/120

An absence of packages in https://nixos.org/nixos/packages.html may prevent potential user from switching to NixOs.

It is 2018 already, case - example of bad user experience, issue doesn't have technical obstacles, how it can't be solved so long only because of FOSS zealotry?

ThomasTheGerman commented 6 years ago

I just had the same problem trying to install steam, just tried installing steam-run on a whim, and got a beautiful error message that dependancies are unfree and how to enable unfree packages.

This is especially stupid from my side since there's a FAQ-Entry for this...

GlennS commented 6 years ago

I've been using Nix for about a year and a half now, and just encountered this problem while trying to install VSCode. It wasted probably half an hour to an hour of my time.

I'd really appreciate it if it was mentioned in the NixOS manual.

goertzenator commented 6 years ago

This issue keeps biting me even though I've been running Nixos a few months now. It is frustrating to have to deal with this in so many different contexts:

I wish there was a global switch at install time that could cut through all this friction (web search is separate of course).