search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.99k stars 14.01k forks source link

No way to install/use a specific package version? #9682

Closed novaluke closed 9 years ago

novaluke commented 9 years ago

July 2020 update

This issue is coming up on 5 years old at this point, and has gotten quite lengthy. While the issue may still be relevant, not all of the comments in this thread will be. It has been suggested that you may find it more valuable to skip to the bottom of the thread in order to get a read on the current state of things, rather than working through it from start to finish.

Here's the original post:

It provides . . . side-by-side installation of multiple versions of a package . . . (From the Nix page)

I can't seem to find any way to install a specific package version, though - it seems like when installing a package you just get whatever is current in the channel you've subscribed to. If there's no way to request a specific package version, how is it possible to have "side-by-side installation of multiple versions"?

_(Note: I understand that some packages occasionally have more than one version available in the current state of the channel and can be accessed via eg. package-name_1.2.3, but this seems to be the rare exception and still doesn't allow specifying a specific version, just a choice from a few available versions.)_

Potential use cases:

Solving dependency/Cabal hell: I found Nix through Haskell, where a lot of people offer it as a solution to "Cabal hell". However, if I can only install one version of a package via Nix, I can't solve the situation where I am working on two different projects with two different and incompatible dependency requirements.

Testing against multiple versions of dependencies: Say I'm writing a library, for which I originally depended on foo-1.0.0. A year later I'm making updates and I want to use expand the upper bound of that dependency so that users aren't restricted to an old version of the dependency. I install foo-1.5.0, make a bunch of updates, verify that it works, but then want to double-check that everything still works with foo-1.0.0, because if the new changes don't work with it I'll have to change the lower bound of the dependency, potentially causing problems for users. However, not being able to specify to Nix that I want foo-1.0.0, I can't test against foo-1.0.0 and can only state the dependency as foo == 1.5.0, instead of the more permissible (and usable) foo >= 1.0.0 && foo <= 1.5.0.

Sharing build/development environments:

Nix makes it trivial to set up and share build environments for your projects (From the Nix page)

I've seen people say that the ability to share build/development environments across the team solves the "Well it works on my machine" problem. However, if I send the environment (via default.nix/shell.nix etc.) to another developer to get them up to speed with the project, they may end up installing different versions of packages than me, since the packages channel may have progressed since I last installed/updated my packages. Now we're not on the same page any more and those subtle (and painful) differences and bugs can creep back in.

Deploying to servers/CI/etc: In a similar vein to sharing development environments, it would be great to be able to use Nix to standardize the environment across all our develop/build/test/deploy phases/environments. However, since the channel could change at any point between these phases and we're not able to specify specific package versions, this isn't possible. For example, take the case where the project is deployed across multiple servers. Being in the cloud, they're ephemeral, so when one breaks for some reason it's destroyed and replaced with a new, healthy version. Except that the new version has to be brought up so speed and installs the project via Nix. Since the channel has been updated since the other servers were last updated, the new server is now running in a different environment and introduces potentially problematic inconsistencies to the system.

Now, these are all hypothetical - I don't actually have experience in any of these use cases but if my understanding is correct I can see them potentially cropping up. However, I can find no mention anywhere (despite very exhaustive searching) of anyone concerned about locking down package versions or installing specific versions when using Nix. If I'm the only one, perhaps I'm missing something fundamental here and barking up the wrong tree?

I just can't see how Nix can deliver the benefits everyone says it does without giving users a way to specify specific package versions - is there really no way to do this, or am I misunderstanding the whole system?

vcunat commented 9 years ago
Single-version policy

We keep multiple versions in nixpkgs only when there's a good reason to. Nix is able to handle any number of versions/configurations, but on the other hand it's much more convenient when all (or most) use just a single one. It leads to better sharing of the effort in many respects: simplified maintenance, testing, sharing of the binaries, etc. It's what most distros do. (Only gentoo diverges from the big ones I know, and they pay a price for it.)

When we do create more variants, we just name them (attribute paths), e.g. gcc48 and gcc49 or ffmpeg and ffmpeg-full.

Only now, we build on the order of 40 thousand packages on the build farm (counting platform variants, etc.), but the number of active contributors is only in dozens. http://hydra.nixos.org/jobset/nixpkgs/trunk

Identical environments

If you're concerned about identical environments, you need to also have the same nixpkgs version, that's for sure ;-) To be certain, you can just copy those by nix-copy-closure. Note that version numbers of direct dependencies don't contain all the information at all.

peti commented 9 years ago

Installing specific versions works just fine, AFAIC:

$ nix-env -qaP boost
boost155  boost-1.55.0
boost156  boost-1.56.0
boost157  boost-1.57.0
boost     boost-1.58.0

$ nix-env -p /tmp/foo -i boost-1.56.0
installing ‘boost-1.56.0’
these paths will be fetched (0.00 MiB download, 109.64 MiB unpacked):
  /nix/store/65fwmbxkbar04m8qqig1a7vymx0xqpg8-boost-1.56.0
  /nix/store/awb45xm2izxjj0sksrxi785shg6lxvbj-boost-1.56.0-dev
  /nix/store/c4cd7qx21v9pq4qrmj0ysz58hz2l0sxs-boost-1.56.0-lib
[...]
building path(s) ‘/nix/store/n2bwn9a87jxc6a52i2p21y4kd414ba72-user-environment’
created 3 symlinks in user environment

And here is the same thing for a Haskell package:

$ nix-env -f "<nixpkgs>" -qaP -A haskellPackages Cabal
haskellPackages.Cabal_1_18_1_6  Cabal-1.18.1.6
haskellPackages.Cabal_1_22_4_0  Cabal-1.22.4.0
haskellPackages.Cabal_1_23_0_0  Cabal-1.23.0.0

$ nix-env -f "<nixpkgs>" -p /tmp/foo -iA haskellPackages.Cabal_1_18_1_6
installing ‘Cabal-1.18.1.6’
these paths will be fetched (4.27 MiB download, 64.34 MiB unpacked):
  /nix/store/48h9yz58zbjd5rp17s3lyjavrd92l8lq-Cabal-1.18.1.6
fetching path ‘/nix/store/48h9yz58zbjd5rp17s3lyjavrd92l8lq-Cabal-1.18.1.6’...
[...]

building path(s) ‘/nix/store/mhbn0csr0lxnsbvf416ns82qlncnh9vw-user-environment’
created 70 symlinks in user environment

So I'm not sure what problem you see?

novaluke commented 9 years ago

First of all, thank you both for your responses!

@peti Right, but as @vcunat said, "We keep multiple versions in nixpkgs only when there's a good reason to."

$ nix-env -f "<nixpkgs>" -qaP -A haskellPackages aeson
haskellPackages.aeson  aeson-0.9.0.1

So what happens if I have to fix a bug in an old project that was using 0.8.1.1? I clone the project from Github, go to set it up with Nix, and now it doesn't compile because Nix wants to use 0.9.0.1 and doesn't give me any other options. My choices seem to be either 1) Be forced to upgrade the project to 0.9.0.1 just to fix a minor bug or 2) Somehow install the package from source myself and make Nix use that instead of the version in nixpkgs. Neither option makes using Nix seem like a better choice than eg. Stack or NPM or Bundler etc, since they'd handle this version requirement seamlessly.

@vcunat Thanks for the detailed explanation, that helps me understand the bigger picture better. I hadn't been aware of nix-copy-closure - that looks like a really great tool and opens up a lot of possibilities!

I guess I'm coming at this from a specific perspective - I'm coming from front-end web development (and now trying to use Haskell/GHCJS), and all the tools I've used for package management for these projects (Bower, NPM, Bundler, etc.) operate on the basis of specifying version ranges for dependencies and locking them down to versions that you know will work. This then necessitates the ability to install any arbitrary version of a package since multiple projects (or even multiple dependencies of a project) can end up specifying different versions from each other. However, the benefit is that any time you install the project anywhere you know that you'll have a working set of dependencies and thus a working project. This is also nice when it comes to deployment, because you can have confidence that if the CI build passes the code will work on the server, and you can have continuous deployment with confidence (as long as package maintainers follow semver). As I wrote above, you're also able to pull down old projects and do work on them despite depending on old packages since you're able to install the old packages without problems.

I see that there are some workarounds for some of these things, such as using a previous revision of nixpkgs, but nothing that seems easy and seamless like these other package managers. I just can't see how to use Nix if you can't (reasonably) use it to install specific versions of packages, since that's a use case I've seen again and again. For example, just today I found a bug in a package I needed to use - normally I would have tried the previous release while waiting for the maintainer to fix the package or I would have fixed it and used my fork while waiting for the PR to go through, but with Nix I was out of luck. I've also often needed to install a newer version of a package than my distro made available, which with Ubuntu came down to just simply adding a PPA, but with Nix I'm tied down to whatever's in the channel.

I do love the idea of Nix (and NixOS!), and I really want it to be something that can work for me. Perhaps I'm just missing the point or philosophy of Nix? Given that I seem to be the only person with these concerns maybe there's a reason they're a moot point and the approach I'm trying to take to deal with them is unnecessary with Nix?

peti commented 9 years ago

So what happens if I have to fix a bug in an old project that was using 0.8.1.1?

In that particular case, Nix won't help you and you are better off using a cabal-install sandbox or a stack build. Hackage contains ~60,000 package versions, the vast majority of which is broken and/or obsolete. IMHO, it makes no sense to put all that stuff into the Nixpkgs database, because the vast majority of it will never be used by anyone. Furthermore, adding that number of build expressions to our distribution would impact a lot of people in terms of performance and memory consumption of tools like nix-env and nix-build and Hydra. It's just not worthwhile for us.

You always have the option of adding missing versions of certain packages to your local database by means of an override as described in http://nixos.org/nixpkgs/manual/#how-to-create-nix-builds-for-your-own-private-haskell-packages. You can register aeson-0.8.1.1 in your copy of Nixpkgs without needing to change the Nixpkgs git repository at all.

If you feel that there are certain older packages that Nixpkgs should contain, then you can also add them to the list at https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/configuration-hackage2nix.yaml#L35, and they'll show up in our package database within the next 2-3 days.

Anyhow, if you want a tool that can easily build any version of any package from Hackage, then cabal-install is your best choice. Nix isn't trying to accomplish that.

peti commented 9 years ago

Sorry, I didn't mean to close the issue. I just hit the wrong button.

novaluke commented 9 years ago

Sorry, I didn't mean to close the issue. I just hit the wrong button.

Thanks for clarifying :)

I've been doing a lot more research in light of what you said and I think I'm starting to understand how the pieces fit together. In the interest of brevity I won't explain all the changes in my understanding but there is still one thing I'm struggling to understand: Nix expressions have no control over the set of packages that they receive.

(tl;dr section at the bottom)

From the perspective of having pure functions and having composability/overridability etc. I see the benefit here. However, from the perspective of a web developer where getting our projects up and running needs to able to be automated across any environment (developer machines, CI, production servers, etc.) this seems like a problem, since the project is no longer "self-contained" in terms of containing all the configuration/commands/etc. needed to bootstrap itself.

For example, let's say I create a project using the nixos-15.09 channel. If any machine (eg. a co-worker's machine, or the server, etc.) is using a different channel, the project will likely fail to work on that machine because the set of packages it receives are all at different versions from those of the channel the package was created with. To make matters worse, there would be no indication anywhere in the project as to what that original channel was, so there's no way to know which is the right channel to use without trial-and-error.

Of course, that information could be documented somewhere, but having to follow specific steps outlined in the readme hurts the "reliable and reproducible" aspect of using Nix (and detracts from the benefits of having a default.nix in the first place). When writing a project with end-users in mind the ability to choose an arbitrary channel may be beneficial, but when the intended use of the project is to be easily deployed via automation it seems detrimental.

As an example of a counterpoint to Nix in this regard, Stack allows an LTS Haskell version (similar to a Nix channel if I understand correctly) to be specified in the project's stack.yaml file. The concerns and considerations of each of these approaches is beyond my capacity to understand, so I won't pretend to know what is "best" (if there even is such a thing as a global "best"), but this difference does make Nix seem lacking in regard to being "reliable and reproducible".

TL;DR

Essentially, as I understand it, in order for Nix to enable a project to be reproducible in a reliable and automatable way it would need to be able to specify what channel it needs to get its packages from. This way it 1) Does not need to be (potentially incorrectly) specified by the user and 2) Is consistent across each invocation.

Is there anything I'm missing here that Nix already has that addresses this issue? I know that a default could be specified in the arguments to the Nix expression. That approach would be good in that it still allows the argument to specified manually if necessary, but it still falls short in that the default argument could only reference a channel if it was already registered with the host system.

jb55 commented 9 years ago

@mayhewluke yup that's the general idea. In our organization we have our own branch of nixpkgs that we occasionally update and run a binary cache for. This way employees can easily install dependencies on the local network and know that the builds will work, with the added benefit that it's super fast! Alternatively if you're working in a less centralized environment you could do this trick where you pin the nixpkgs version in the shell.nix itself:

https://garbas.si/2015/reproducible-development-environments.html

nix is flexible enough to cater to these different workflows. I think eventually we'll be able to target stackage from nixpkgs as well if you're into that sort of thing.

peti commented 9 years ago

Nix expressions have no control over the set of packages that they receive.

But they do. Things that are no parameter to your functions cannot vary, i.e. https://github.com/peti/ghc-library-id-bug/blob/master/default.nix#L5.

Stack allows an LTS Haskell version to be specified in the project's stack.yaml file.

Sure, but note that this doesn't mean that the version of all packages in that release are fixed. LTS releases can and do change: https://github.com/fpco/lts-haskell/commits/master/lts-3.2.yaml.

novaluke commented 9 years ago

Ah, fantastic! @jb55 that blog post and @peti your code in ghc-library-id-bug answer that one last missing link for me. I now see how it is possible to 1) Pin the revision of nixpkgs while still allowing overrides and 2) Similarly "pin" to the head of a specific channel via fetchTarball. And because Nix is brilliant it won't fetch these things over and over unless the inputs to them have changed, so performance isn't really a concern. Excellent!

@peti thanks a ton for your patience in helping me understand all this. I now have the answers to every question/concern I've raised in this thread. I'll post a summary here of my questions/the answers once I've had the chance to write it up. Hopefully this will help anyone else who may have similar questions and might also help Nix maintainers to be able to save time on answering questions :) Thanks again!

novaluke commented 9 years ago

I'll post a summary here of my questions/the answers once I've had the chance to write it up. Hopefully this will help anyone else who may have similar questions and might also help Nix maintainers to be able to save time on answering questions :)

Here it is!

chris-martin commented 8 years ago

Some problems with that approach:

I would love to be able to do something like import <channels.nixos-15.09> {} to pull in whatever expression nix-channel has last obtained for that channel name.

vcunat commented 8 years ago

I find one often has to create a fork/branch of nixpkgs for a particular dev project anyway, e.g. because of a need to update something in nixpkgs and use it now without waiting to push to official master or waiting on channels. I think that simple step also "magically" solves all of the problems you mention.

chris-martin commented 8 years ago

That just moves the problem, doesn't it? Instead of needing some way to point a shell.nix to a channel, now you need some way to point it to your fork.

vcunat commented 8 years ago

But you have lots of ways to do that: $NIX_PATH, various command-line parameters, etc.

vcunat commented 8 years ago

BTW, recently nix got support to easily specify exactly the commit hash that you want to use: http://nixos.org/nix/manual/#sec-common-env

chris-martin commented 8 years ago

Don't you want to be able to specify a packages source in a config file, to avoid requiring each dev to set up their own environment variables or command-line parameters? I don't understand how I'm failing to communicate the motivation here.

The current situation is awkward because there exists a multiplicity of nixpkgss, and it could be solved by unifying them under a single root. Suppose nix-channel would set up an expression like this:

with rec {
  channelPath = name: file:
    "${builtins.getEnv "HOME"}/.nix-defexpr/channels/${name}/pkgs/top-level/${file}";
  channel = name: file: import (channelPath name file) { config = config; };
};
pkgs = rec {
  nixos-15-09 = channel "nixos-15.09" "all-packages.nix";
  nixos-unstable = channel "nixos-unstable" "default.nix";
  default = nixos-unstable; # or whatever the user picks as their default
}

(I've done something like this in my own config to write expressions that use multiple channels.)

Then anything that wants to be able to select channels by name could do so from that expression, with no command-line parameters or env vars necessary.

vcunat commented 8 years ago

I personally want my dev projects to use versions independent of each other and independent of channels or the version my system uses, so that nix-channel --update won't break them, but obviously different people want different things.

Still, it would seem nice if nix-channel got better in this, e.g. the way you propose. Currently nix-env does respond in a similar way, but it seems a bit ad-hoc. Different commands react in different ways with respect to channel selection.

thomastjeffery commented 7 years ago

I don't completely understand why this issue is closed.

All I see are workarounds.

To summarize what I have read here (correct me if I am wrong)

@mayhewluke described some real drawbacks of how derivations are currently implemented:

@peti brought up some concerns as reasons for the current behavior:

and some workarounds:

I still have some serious issues with this.

My proposed solution

Here are my thoughts about how Nix should deal with package versions. Please tell me your thoughts. How hard would this really be to implement? Can we make a branch of nixpkgs, and make these changes?

Let a derivation specify a range of versions for each dependency.

Different versions are provided by different derivations.

As far as I know, this is already true. The only difference would be that the names are always the same. If you tell nix to install a package, it should just grab the latest version that channel has. If you want a specific version, specify (that's the part you can't do currently).

TL;DR The versions are there! Use them!

Most, if not all derivations specify a version string, and append it to the name. Let's just stop appending the version to the name, and treat it like the discrete data point it is.

CMCDragonkai commented 7 years ago

@thomastjeffery there are projects that try to automate the conversion of other package manager indexes to Nix expressions. It still requires someone to commit in the new expressions in nixpkgs. Although the I think the haskage integration is fully automated (not sure). But the basic idea is that version numbers don't make sense in Nix because it's meant to be content addressed. There needs to be a translation from the version specification to a content addressed expression in nixpkgs to get what you want. For now most of this is done manually. Also every previous iteration of a channel is still accessible.

CMCDragonkai commented 7 years ago

Oh and some packages do support version as a parameter. Then you need to write a package override and call that package with your desired version specifier. But this parameterisation of the version is not standardised and may not even work if different versions build differently!

thomastjeffery commented 7 years ago

For a clearer example, I just spent a few hours learning (#28248) that I need to add thirteen package overrides, and use the master channel, just to build yi.

This is a real problem.

there are projects that try to automate the conversion of other package manager indexes to Nix expressions

I understand that. Until I found this issue, I didn't know you could search haskellPackages directly. Until then, I was searching the generated hackage.nix

To be clear, I don't advocate providing more versions. We should, however, be clear about what versions are available without this mess of package names. It's confusing for relatively new users like myself, and makes the system more complex.

I just want to be able to specify more clearly, and to work with a cleaner, more functional codebase. After all, Nix is the pure functional package manager.

the basic idea is that version numbers don't make sense in Nix because it's meant to be content addressed.

What do you mean by that? There are packages, and those packages have versions. Sure, Nix addresses the store with hashes, but versions are still important.

CMCDragonkai commented 7 years ago

If you give nix a version specification, nix doesn't know what hash this corresponds to, nor where to get it. Something needs to bridge that gap. That's why versions would have to be a parameter in nix that gets mapped to a hash somewhere, however because it's meant to be pure, we cannot call into other/remote package indexes to dynamically fetch the hash. That's why the translation from version to hash has to happen at "compile time". If we can standardise an infrastructure for doing this and then writing adapters to this infrastructure for every package index that exists, then it would be alot simpler!

CMCDragonkai commented 7 years ago

I've been debating with myself of whether to pin the nix-shell to a commit hash of nixpkgs or to leave it to the OS using <nixpkgs> variable. My OS itself still pins the <nixpkgs>. The advantage of this is that every nix-shell uses the same set of nixpkg expressions, reducing the need to hold many sets of nixpkgs, and also reducing the "build" time that occurs every time you run nix-shell (since they all sharing the same set). However this means the dev environments aren't a fully specified closed expression. If I pin each shell.nix to a particular hash of nixpkgs, what happens when GC runs after I update my OS pinned nixpkgs? Will it wipe out all the dependencies of shell.nix forcing it to rebuild/redownload stuff when I next run nix-shell?

jb55 commented 7 years ago

@CMCDragonkai not pinning hashes has bitten me too many times when building old projects. now I always pin. pin2win. as long as you have a nix-build generated symlink lying around it shouldn't get gc'd.

thomastjeffery commented 7 years ago

So what I want is to have a package's default.nix provide one explicit version. This is practically current behavior, I just don't like the current naming scheme.

The real change I want, is for a derivation to be able to specify a range of compatible versions, or even just one specific version for any given dependency.

nix doesn't know what hash this corresponds to, nor where to get it.

Nix currently knows what hash a package name corresponds to, because it is constrained to the one package that is defined by the channel it is using.

What am I missing?

thomastjeffery commented 7 years ago

The main thing I want to accomplish is to distinguish between the versions of packages, and the versions of derivations.

A nix-channel is a tree of .nix files that define how to get packages. The .nix derivations themselves can be, and are updated independent of package versions. You can generally consider a nix-channel to correspond with a version of those derivations.

...but a nix-channel should not explicitly mean package version. You should be able to have multiple package versions defined by a channel.

One of the great strengths of Nix is the ability to use multiple versions of packages in the same system without renaming that package's files! So why are we renaming the packages themselves?

vyp commented 7 years ago
  • I then want my derivation to pattern match a range of versions.
    • This is not current behavior.
    • The current channel will provide a package with the greatest version number matched by the range provided that it has defined.

What am I missing?

I might be misunderstanding or missing the point here (sorry), but what's the point of a range if it always returns the greatest version number?

If you mean like the greatest compatible version in the version range (whatever compatible means), I might be wrong, but from my understanding, nix doesn't really do any 'dependency management' like this where to attempts to choose a correct version. Instead, everything is declaratively specified/mapped, if that makes any sense.

without renaming that package's files!

What do you mean package's files?

thomastjeffery commented 7 years ago

I might be misunderstanding or missing the point here (sorry), but what's the point of a range if it always returns the greatest version number?

If I am writing a derivation, I am probably not explicitly choosing a channel to build it against. Since different channels will provide different versions, I just want whatever is both compatible with my derivation, and available in that channel. Hence, pattern matching.

What do you mean package's files?

In Debian, ArchLinux, etc, libraries are put in the same standard path. If libstdc has more than one version installed, there is a naming conflict. This is worked around by having libstdc.so.1.

In Nix, every derivation is installed into its own folder in /nix/store, so there is never a naming conflict. You can have several versions of a package installed to the same path and name, since it really isn't the same path.

Since we don't use version numbers in derivations, we now have several packages providing the same library with a different version, but instead of having the same name, they are renamed package-name_1_2_9.

We have moved file name conflicts to package name conflicts. We aren't taking advantage of the functional nature of Nix. Avoiding name conflicts is a core feature. We brag about it on the front page! But it isn't real.

thomastjeffery commented 7 years ago

My biggest issue with renaming packages is that the normal-named package is usually older.

For example, haskellPackages.sdl2-ttf is version 1.0.0, but haskellPackages.sdl2-ttf_2_0_1 is version 2.0.1. Version 1.0.0 has literally none of the high-level API sdl2-ttf is designed to provide, and is therefore practically useless, but it is the default version.

Rather than spending hours learning that, I could have just started using the latest version of the library, like I thought I was. Derivations that wanted 1.0.0 could have said so.

vyp commented 7 years ago

If I am writing a derivation, I am probably not explicitly choosing a channel to build it against. Since different channels will provide different versions, I just want whatever is both compatible with my derivation, and available in that channel. Hence, pattern matching.

Right, but then the build becomes dependent on the 'channel', not on <nixpkgs>? Or..? (Also I'm assuming you mean writing a nix expression, I don't think it's common to write derivations by hand?)

Since we don't use version numbers in derivations, we now have several packages providing the same library with a different version, but instead of having the same name, they are renamed package-name_1_2_9.

I don't understand what's being renamed if the package's files are the same name? I think you mean the attributes? But if so, what's the problem with attributes being renamed though? Isn't that avoiding name conflicts by specifying the version number in the attribute, how else would you do it?

vyp commented 7 years ago

My biggest issue with renaming packages is that the normal-named package is usually older.

Oh right missed your latest comment! Yep I agree, but this is usually the case in my experience though, the behaviour that you want. But if in your experience it's the opposite, then yes it should probably ideally be changed. I guess it might be dependent/related to the packages themselves and what sort of infrastructure they use, what the most used/popular versions of them currently are, and maybe even the maintainers of the package(s). But so, is there a problem apart from that though?

thomastjeffery commented 7 years ago

I don't understand what's being renamed if the package's files are the same name?

Right now, I can type nix-shell -p haskellPackages.sdl2-ttf, and have an environment with sdl2-ttf version 1.0.0, which is outdated, and unusable.

I expected to get the latest package, since I didn't explicitly provide a version, but instead I got an old version, and had to learn that the hard way.

then yes it should probably ideally be changed.

What you are saying is that sdl2-ttf should be changed to be version 2.0.1. But how did we get here in the first place? We can't specify versions for dependencies, so all the packages that already existed that depended on version 1.0.0 made 1.0.0 the default version, even though it's practically a different library from 2.0.1. The reason sdl2-ttf_2_0_1 exists is that we can't specify a version for dependencies. If we could, we would never have gotten into this mess.

I guess it might be dependent/related to the packages themselves and what sort of infrastructure they use

haskellPackages are automagically generated from hackage.

But so, is there a problem apart from that though?

At the beginning of this thread, @mayhewluke described several.

The most interesting to me was that he wanted to use an older version on purpose. Since that version has no expression, he can't, but the larger problem is that he couldn't even try.

Right now, expressions have a version, but that version can't be referenced by buildDependencies. If they can, it's an exception to the rule.

CMCDragonkai commented 7 years ago

You can always use older expressions. Use the pinning method.

thomastjeffery commented 7 years ago

Yes. That is an available and usable workaround.

The problem is that we have a mess, and we are working around it instead of fixing it.

CMCDragonkai commented 7 years ago

I don't consider that a workaround, that's one of the great things of content addressed package managers.

thomastjeffery commented 7 years ago

That's not my point. What I am trying to say is that pinning is a great feature that happens to be used as a workaround for this specific problem. Having a great feature doesn't diminish the problem.

CMCDragonkai commented 7 years ago

@jb55 How do you deal with the situation when your default.nix and shell.nix won't match and cannot match because the "building" of something is separate from the development environment of the same thing. For example some tools are only needed during dev, and not during build. And in other cases, I only need a shell.nix and not a default.nix because the thing I'm developing isn't intended to be a nix deployable package?

I think what's needed is there needs to be a way to make shell.nix integrated into the GC root system. Maybe the new Nix utilities will provide this.

thomastjeffery commented 7 years ago

How do you deal with the situation when your default.nix and shell.nix won't match and cannot match because the "building" of something is separate from the development environment of the same thing.

I'm not sure I follow. What do you mean "won't match"?

CMCDragonkai commented 7 years ago

As in they are not the same file due to bringing in different dependencies, or having different environment variables.

thomastjeffery commented 7 years ago

So what you are saying is that you don't want to require a version for expressions?

That sounds fine with me. I just want to use versions where they already exist. More specifically, I want version strings to pattern-match derivations that specify a version, so I can get a derivation that specifies a version without giving that derivation a unique name.

vyp commented 7 years ago

What you are saying is that sdl2-ttf should be changed to be version 2.0.1. But how did we get here in the first place? We can't specify versions for dependencies, so all the packages that already existed that depended on version 1.0.0 made 1.0.0 the default version, even though it's practically a different library from 2.0.1. The reason sdl2-ttf_2_0_1 exists is that we can't specify a version for dependencies. If we could, we would never have gotten into this mess.

We got here because when updating sdl2-ttf, someone made a new sdl2-ttf_2_0_1 attribute instead of just using the existing sdl2-ttf attribute. They should have made a new sdl2-ttf_1_0_0 attribute instead for all packages that were still using version 1.0.0.

So in a way, specifying the version number in the attribute is a way of specifying a version for dependencies. However, if I understand what you want correctly now, I think this distinction between the package name and the version (instead of having them together) could help in giving a guarantee that you are getting the latest available version of a package in a channel.

As in, attributes cannot do that because you have no guarantee that the default (i.e. version-less) attribute is really the latest available version (e.g. sdl2-ttf was really only 1.0.0, because imo someone didn't do it right). And names, as they are currently, cannot do that because they always have the version in them, so you are always forced to specify a version (or are you.. maybe there's like some 'glob' syntax or something??). So if we split the name into the 'package name' and the version, then we can just specify a package name, and then nix can look at all the available nix expressions for a particular package name and simply pick the one with the biggest version number. And that comparison that nix would do between the versions would give us this guarantee that we got the latest available version.

Is something like that what you are looking for?

The most interesting to me was that he wanted to use an older version on purpose. Since that version has no expression, he can't, but the larger problem is that he couldn't even try.

Unless I'm missing something, he could though, he just didn't know about pinning.

I think the bigger issue with that though is that I don't really think 'channels' are all that useful for development environments. They're probably OK for users, but you're right channels are too limiting because you're always stuck on what's on your channel. In fact, iirc there was some talk about removing the entire concept of channels from nix, but I'm not sure if it was for the same/similar reasons (and can't find it atm unfortunately).

jb55 commented 7 years ago

@jb55 How do you deal with the situation when your default.nix and shell.nix won't match and cannot match because the "building" of something is separate from the development environment of the same thing. For example some tools are only needed during dev, and not during build. And in other cases, I only need a shell.nix and not a default.nix because the thing I'm developing isn't intended to be a nix deployable package?

My default.nix just takes a single variable: nixpkgs. That way everything remains consistent in the build and in the shell. Here's an example from one of my projects:

https://gist.github.com/jb55/6e93156ca7fe90a36bb08df0408446a3

I'm being kind of sloppy with the buildInputs here, could be cleaned up a bit.

thomastjeffery commented 7 years ago

@vyp It sounds like we are on the same page regarding package names.

Unless I'm missing something, he could though, he just didn't know about pinning.

If I understand correctly, pinning allows a .nix to specify what channel it gets, not what dependency versions it gets. Package versions are still just whatever the channel provides. That may solve the "reproducible" issue, but not the "specific dependency version".

If you are dependent on the channel for reproducible builds, what happens when the channel updates a package?

Channels are effectively nix versions. Using the 17.09 channel is akin to using Debian 9, and using the master channel is akin to using Debian sid/unstable. Neither guarantees a specific package version unless you specify a specific channel revision. If I understand correctly, that is what you mean by "pinning". There are several obvious issues with that:

  1. What's "pinning"? Is this ever explained to a new user? Where?
    • (Yes, I understand what it is now, but new users probably will not, meaning it doesn't exist to them.)
  2. Your packages never get bug-fix updates
  3. Your packages never get security updates
  4. Broken package expressions never get fixed.
  5. If you need to update one package, you must update them all.
    • This can be seen as either consistency, or limitation. It depends on the use case.

I think the bigger issue with that though is that I don't really think 'channels' are all that useful for development environments. They're probably OK for users, but you're right channels are too limiting because you're always stuck on what's on your channel.

That's the heart of the issue. The problem is not the limitation of channels, per se, but the fact that we are trying to shove them into a use-case they aren't designed for.

Like I said earlier, channels are like OS versions.

CMCDragonkai commented 7 years ago

No pinning isn't about which channels you get. Pinning pins to a content addressed commit hash of nixpkgs or any nixpkgs even your own fork. Channels is orthogonal and probably a mistake and should be removed.

On 27 Sep 2017 06:24, "Thomas Jeffery" notifications@github.com wrote:

@vyp https://github.com/vyp It sounds like we are on the same page regarding package names.

Unless I'm missing something, he could though, he just didn't know about pinning.

If I understand correctly, pinning allows a .nix to specify what channel it gets, not what dependency versions it gets. Package versions are still just whatever the channel provides. That may solve the "reproducible" issue, but not the "specific dependency version".

If you are dependent on the channel for reproducible builds, what happens when the channel updates a package?

Channels are effectively nix versions. Using the 17.09 channel is akin to using Debian 9, and using the master channel is akin to using Debian sid/unstable. Neither guarantees a specific package version unless you specify a specific channel revision. If I understand correctly, that is what you mean by "pinning". There are several obvious issues with that:

  1. What's "pinning"? Is this ever explained to a new user? Where?
    • (Yes, I understand what it is now, but new users probably will not, meaning it doesn't exist to them.)
  2. Your packages never get bug-fix updates
  3. Your packages never get security updates
  4. Broken package expressions never get fixed.
  5. If you need to update one package, you must update them all.
    • This can be seen as either consistency, or limitation. It depends on the use case.

I think the bigger issue with that though is that I don't really think 'channels' are all that useful for development environments. They're probably OK for users, but you're right channels are too limiting because you're always stuck on what's on your channel.

That's the heart of the issue. The problem is not the limitation of channels, per se, but the fact that we are trying to shove them into a use-case they aren't designed for.

Like I said earlier, channels are like OS versions.

  • They provide the latest version of a package that they want to provide.
  • They may provide multiple versions of a package, to fill dependencies.
    • This introduces a naming conflict that we are working around instead of fixing.
  • They do not provide older package versions that aren't used by other packages.
    • That's reasonable, since users don't want them, but developers do.
    • How about we give developers a clear way to do that?
      • I don't think channels are the solution here.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NixOS/nixpkgs/issues/9682#issuecomment-332324361, or mute the thread https://github.com/notifications/unsubscribe-auth/AAnHHdLZ5YWX2XKQ5Ohtf3GeJY_wItiNks5smV2LgaJpZM4F4oZ9 .

thomastjeffery commented 7 years ago

My mistake.

From my understanding, though, channels simply represent nixpkgs git branches. I suppose pinning, then, is more like git tags. Thanks for the explanation.

Channels is orthogonal and probably a mistake and should be removed.

That's an interesting point, but since, as you said, channels are orthogonal, it would probably be best to have that discussion elsewhere.

Still, that doesn't change the limitations I mentioned, or the current situation with package names sometimes containing versions.

CMCDragonkai commented 7 years ago

I think pinning is more than just git tags. Git tags is like versions (one can change a git tag to point to something else after it is first published, and there's no computational relationship between the git tag and the data/code that it points to). Pinning is making use of a content addressed hash, the result is a cryptographically guaranteed snapshot of the software you're expecting. Because git is a content addressed store. You can then pick and choose any commit hash through out the entire history of nixpkgs to use including even the PRs and other people's forks.

One effect of this is that you get reproducibility. Note that this is not binary reproducibility, since it's still possible for the compilation of code to give different resulting binaries. But it is reproducibility within the context of Nix universe.

Oh and with regards to pinning, take a look at this: http://matrix.ai/2017/03/13/intro-to-nix-channels-and-reproducible-nixos-environment/ I wrote that a while ago regarding the pinning of the OS. And yes you don't get automatic updates. This is by design. Auto updating in nix would not be automatic like the way Chrome updates, it would be a more considered approach, where you choose when you want to update. Also you'll find that reproducibility is something that you will prefer when you're developing things!

If channels were to be removed, it would be more clearer to everybody including new users, what the real use and power of nix is, and all documentation would go straight to pinning. Then only after understanding this, can people write higher level abstract tools to provide an auto-updating "channel" like interface to the underlying pinning concept.

Note that it is not an all-or-nothing concept. Pinning allows you pin just one package. You can actually compose multiple nixpkgs expression sets together.

vyp commented 7 years ago

That may solve the "reproducible" issue, but not the "specific dependency version".

But wouldn't reproducibility involve specific dependency versions though? Otherwise in my opinion it's not reproducible.

Neither guarantees a specific package version unless you specify a specific channel revision. If I understand correctly, that is what you mean by "pinning".

  1. What's "pinning"? [...]

Yep, this is mentioned in a link earlier in this thread: https://github.com/NixOS/nixpkgs/issues/9682#issuecomment-139831114

I'm not sure you can pin channels though because if a channel gets updated, it's sha256 would also change. However, I do know that you can do use the -I command line option to specify a nix expression search path for nix commands:

nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs-channels/archive/nixos-17.03.tar.gz

And thus retrieve a channel like that.

  1. [...] Is this ever explained to a new user? Where?

Don't think so, but it definitely should! See https://github.com/NixOS/nixpkgs/issues/27994 for this.

  1. Your packages never get bug-fix updates
  2. Your packages never get security updates
  3. Broken package expressions never get fixed.

Well yeah, if they would then it wouldn't be the same stuff, so not reproducible.

  1. If you need to update one package, you must update them all.

Not sure what exactly you mean here?

This introduces a naming conflict [...]

Actually I'm not sure what naming conflict you're talking about still? In my previous comment I described a use case for separating the package name from the version, so that we can just specify a package name and have a guarantee we'll get latest version available for it in nixpkgs. But I don't think that had anything to do with naming conflicts?

  • They do not provide older package versions that aren't used by other packages.
    • That's reasonable, since users don't want them, but developers do.
    • How about we give developers a clear way to do that?

I don't see how separating the package name and version makes it any different for providing older versions of packages, than just using attributes like we are now?

From my understanding, though, channels simply represent nixpkgs git branches.

I mean conceptually, pretty much, I think! In fact, there's https://github.com/NixOS/nixpkgs-channels which has branches named after channels which track the nixpkgs revisions that the respective channels are using.

So for actually separating out package names and versions though, it might be because of my limited nix expression language knowledge, but I have no idea how you would implement this though, because attributes pretty much always used to refer to packages, in configs, package expressions, modules, everywhere I think. Maybe you could write your own tool or nix function or something that parses out the name and the version for all attributes that begin with the same characters, and then returns the one with the biggest version number.

If however you want to make more substantial changes than writing your own function/tool to deal with this (again, I wouldn't know what it would look like), and you feel strongly enough about this issue, I'd suggest first opening a new issue for this, or sending an email to nix-devel, and then eventually going through the RFC process to do it.

michaelpj commented 7 years ago
  1. If you need to update one package, you must update them all.

Just wanted to point out that this isn't true - you can take some packages from a pinned revision and some from a channel, or anywhere else. You can see an example of using multiple package sets at the bottom of CMCDragonkai's post. So in practice if you only need to pin one package you can pin just that one package. That package will indeed not get updated (and it will have its own, old, copy of its dependencies), but that's the point of pinning.

thomastjeffery commented 7 years ago

So now I completely understand pinning. Sorry it took so long. Maybe I need to read more, maybe documentation is not up to par. I'll let you come to your own conclusion on that front.

  1. If you need to update one package, you must update them all.

Not sure what exactly you mean here?

I meant that if you are dependent on channels for dependency versions (which is not pinning like I thought), then you would not be able to select specific package versions, so as soon as you update the channel, you end up updating all dependencies. I was confused, and conflated pinning packages (reality) with pinning channels (nonsense).

I suppose pinning packages is a good way to work, but I don't necessarily see it as the only way. Sure, it's more reliable, but some features can still be seen as limitations in certain context.

Pinning is a great workflow, and content-addressed packages are a core feature of Nix. It's not an easy workflow to jump into, though. We can make it easier.

If I want a package that is a certain version, I need to find out what content-address to get it from.

Right now, that involves scouring Github for the name of a package that corresponds to the version I want. Let alone the fact that there are multiple nonstandard ways to do this, I shouldn't be doing it in the first place. I'd rather add the version string to my name search.

The version string is there, but it's hard to find. It's either ignored, appended to the package name with underscores instead of dots, or something else entirely. Because the version string isn't explicitly used by Nix, it isn't required, and there is no standard way to use or provide it.

All I want is a standard way for packages to provide version strings, and a standard way to reference them. That way I can search for version strings the same way I search for package names.

Think of it as "loose pinning". Instead of pinning a hash, you are pinning a range of hashes. For convenience, you reference the hashes using a single, list, or range of version strings. If a version string references no hashes, you know immediately. If a version string references several hashes (because an expression was updated, but the version did not change), you grab the newest hash.

novoxd commented 5 years ago

You know, as a user I really want this feature, as a developer I can leave with out it (solutions posted upper).

For example I want to install pidgin (solving it right now). What I get - Pidgin 2.13.0 has segfaulted and attempted to dump a core file.. Nice, ok no problem. And the first thing I want to do - try older version, cause in op world it is quite common to have bugs in newer versions. Why I want to do this - fast determine (or just get piece of the info) is this problem in my system or in package. Cause I know that there is good chance that old version will work. And I want it simple without bothering about any of nix config or getting too deep. And there not so much versions needed for example just 2.10.0, 2.0.0, 1.7.0, If they are tested and stable. And what I want - just to get working software right now. I can use old version and maybe invest some of my time with fixing new one, and of course I will switch to new one if it will be fixed. (The cool thing could be - if such problem finds out new version moves to unstable and last working one to stable channels.)

Its all about comfort of each day using and entry level threshold.

I thought a little - I have a simple solution. Lest have additional option for nix-env and inner build staff or package name like packagename#version. And a map file in nixpkgs repo what will map package with name and version to repo sha. So then on install we need to simply checkout that sha and build package. Additionally there will be need of some sugar - support of multiply versions in nix store and db, in bin path resolver. But this can be also solved with naming packagename for last package and packagename#version for versioned packages and last version what name is packagename has an alias with it packagename#version so there will be no need for additional fixes.

@vcunat , @peti, @edolstra what do you think about it?