There isn't a clear canonical way to refer to a specific package version.

[thomastjeffery]

Context

The overwhelming majority of packages in the Nix ecosystem did not come from the Nix ecosystem. As such, the projects these packages are made to provide tend to use incompatible organization philosophies like version numbering.

Even so, users expect Nix to provide multiple instances of packages that are delineated by version number.

Nix Design

In Nix, a package is described in two places:

1. The derivation: Somewhere in nixpkgs/pkgs is a default.nix that describes where to get the source, how to build it, and what files to keep.

2. The global attribute path: The package name is defined in pkgs/top-level/all-packages.nix. The name evaluates a nix expression using the derivation defined in nixpkgs/pkgs/somewhere/package-name/default.nix.

It is assumed that a derivation will be built from the latest source available. To update a package, the derivation is edited to build from a newer source version, tested, and the changes to the derivation are added to a git PR to be merged with nixpkgs/master.

It isn't uncommon for a derivation to depend on a different version of a package than what is provided by its default.nix. For example, many C/C++ programs do not build with the latest GCC.

When different versions of the same project are provided, they each need a unique derivation file and attribute path name.

It is recommended in the manual that extra package versions are created as separate derivations alongside the default derivation, and the version name is appended to the package name:

If there is only one version of a package, its Nix expression should be named e2fsprogs/default.nix. If there are multiple versions, this should be reflected in the filename, e.g. e2fsprogs/1.41.8.nix and e2fsprogs/1.41.9.nix. The version in the filename should leave out unnecessary detail.

All versions of a package must be included in all-packages.nix to make sure that they evaluate correctly.

For example, there currently exist the attribute paths gcc6, gcc7, gcc8, gcc9, and gcc10 alongside gcc, which provides gcc 9.2.0.

Nix Usage

When installing a package, a user will generally refer to its attribute path name, as defined in all-packages.nix.

When a user updates their system with nixos-rebuild --upgrade [switch|boot] or nix-env -u [packagename|*], nix gets whatever derivations are defined in the latest nixpkgs commit.

NOTE: This is made a little more complicated by channels. Just remember that channels refer to git branch names.

Problem

It is fairly common for a user to want to use a different package version than the default one provided in their nix channel. In order to use it, the user must find it.

There are three ways this can be accomplished:

1. If there is already a derivation provided by the user's channel, the user need only find its attribute path name. This has gotten easier with nix search, but isn't always reliable. Usually, packages follow the naming convention of appending the version to the package name, but this is not always the case.

2. If there is not a derivation provided in the user's channel, the user must find a hash that points to the desired derivation version, and use pinning to refer to the package by hash. The hash can be found using this third-party tool, and used as described here.

3. If there is not a derivation that provides the desired package version, the user much create their own. Suddenly, a user who just wanted to use a simple and elegant package manager needs to learn how to write a nix derivation, or at least how (and where) to modify an existing one and use that. This is usually done by checking out nixpkgs/master, and either altering the existing derivation (or a copy) to one that builds the desired package version. This method is described in the manual here.

None of these options is obvious, clear, canonical, and easy to use.

Discussion has been happening on this topic for about five years (off and on) here: #9682

[vcunat]

Perhaps it's just me, but I'm not sure where you aim at. I think the intended canonical way will be Flakes, but it was linked in the other thread already, so perhaps I'm missing something you want? That is, apart from more standardization of attribute naming, for the relatively uncommon cases where we do keep multiple versions in nixpkgs.

---

Generally, I believe that in practice the main work is in maintenance of those versions – ensuring that the combinations you'd like keep working together (up to exponential number), producing binaries for those combinations (machine time & space), patching important (security) bugs, etc. I'd rather try pushing the community towards using relatively few combinations of versions and configurations, as it just appears to make life easier (through sharing that work, basically).

For instance, what do you do after you find your desired version in some older nixpkgs commit? I believe typically you don't really want to just use the older nixpkgs version. If that was OK, why would any distro bother with stable releases? (And I omit the complication of having more of such dependencies.) I expect you typically want something like forward-porting that version to current nixpkgs stable branch (or master) – which might just work trivially or it might require further changes.

[markuskowa]

Is this maybe what you are looking for: https://matthewbauer.us/blog/all-the-versions.html

[spinus]

It is assumed that a derivation will be built from the latest source available.

where did you get this assumption from? is it your assumption? I didn't see this assumption anywhere else.

I think nix ecosystem would be solving different problems than ones you express and desire. Let's look at nix website: Reproducible Declarative Reliable Those are problems nix is trying to solve, not to deliver as many packages in all of different versions. It's giving you a tool to do it in a stable manner, but that's not the goal. How much work would be to support so much software.

[jeff-hykin]

@vcunat

I'm not sure where you aim at

Trivial Example

I want to install Lua 5.0.x globally (assume I don't care about ANYTHING else)

1. How can I list every package/channel/derivation that contains Lua 5.0.x
2. What exact command do I run to install that version globally

(Now repeat the process for Python 3.6.x and Ruby 2.5.x)

Problem

If your answer is "well there's no straightforward way of doing that" then that is what this issue is about: we want a straightforward way of doing it because it is a frequent basic need

If there is a straightforward way, my understanding is that will be the end of this issue

Discussion

I believe @thomastjeffery's post is long because he was diving into the details of how this process could potentially be made simple and the current state of why it is difficult

[thomastjeffery]

As far as I am concerned, until a solution does not need to be searched for by users, this is an open issue.

There needs to be a straightforward way for users to install different versions of packages.

There needs to be a straightforward way to create derivations that depend on specific versions of packages.

Right now, there may be ways of doing those things, but those methods are not obvious or straightforward.

As I stated in the original issue, I do not have enough expertise to lead this discussion. I have done my best to outline the problem, and will be happy to answer any questions about it, and try to understand proposed solutions.

[vcunat]

My point of view is that if there still are good use cases for Lua 5.0, Python 3.6, etc. we should have them as attributes in the current nixpkgs repo. That way they will be comfortable to use, security problems will be easier to fix on those packages, etc. (Say, we probably won't build all pythonPackages with 3.6 on Hydra, but that's a tiny detail.) Those versions were typically removed because no known person used them.

[8573]

I want to install Lua 5.0.x globally (assume I don't care about ANYTHING else)

I'm afraid I don't remember the prior art on this from when I used other distros: How is this done with apt-get and other common tools for installing programs globally?

[jeff-hykin]

My point of view is that if there still are good use cases for Lua 5.0, Python 3.6, etc. we should have them as attributes in the current nixpkgs repo. That way they will be comfortable to use, security problems will be easier to fix on those packages, etc. (Say, we probably won't build all pythonPackages with 3.6 on Hydra, but that's a tiny detail.) Those versions were typically removed because no known person used them.

That's good in theory, but just because someone/some group doesn't think it's a good use case doesn't somehow change that I need a particular version. If I'm a security researcher studying vulnerabilities, or a someone working in a government environment with versions that take decades to be approved, or any number of other cases: my "good use case" and your "good use case" are going to be different.

If nix can't meet that need, then I'm just going to use something like brew, apt, or pacman that has a larger package base, and then I'll use luavm, rbenv, and asdf to manage versions on an individual basis like before.

Maybe I'm missing something but what's even the point of having a functional packaging system when you can't reproduce the inputs? There's no use in guaranting an output if you can't reproduce the inputs, which is the same as any other package manager. They all supply "good enough" packages.

[jeff-hykin]

I want to install Lua 5.0.x globally (assume I don't care about ANYTHING else)

I'm afraid I don't remember the prior art on this from when I used other distros: How is this done with apt-get and other common tools for installing programs globally?

In the current state of things without nix, you have to get a version manager specific to that binary: rbenv or rvm for Ruby, nvm for node, etc. Then use those tools to install multiple versions of ruby/node/etc.

Why not just do that?

1. It seems like that would be against the nix philosophy to install a version manager within nix (aka dynamic version mutation instead of functional versioning), since nix itself is supposed to manage versions functionally.
2. In that case, if I was going to do that, I just wouldn't use nix at all and I'd use a normal package manager. But even then its far from ideal having 10 different version managers, each of which are managing versions of language-specific-package-managers (e.g. pip) which manage versioning of packages.

[vcunat]

Maybe I'm missing something but what's even the point of having a functional packaging system when you can't reproduce the inputs? [...]

You can. Just check out the historic nixpkgs commit you want and you will reproduce the build, including all the old dependencies, etc. I've done repo-wide git bisect over long history periods, multiple times, it's really nice. (Most tarballs should get mirrored on cache.nixos.org IIRC and old binary builds are there as well.) Note that typically those packages will depend on vulnerable OpenSSL, for example. If that's what you want... I expect you mainly need a tool that will find you a suitable nixpkgs revision (some of the linked ones perhaps).

[jeff-hykin]

Just check out the historic nixpkgs commit you want

How can I do that when I can't find the nixpkgs commit that has the version I need??? (literally the title of this issue)

[vcunat]

As I wrote you'd need some tool for that, e.g. this one (which I only know thanks to this GitHub issue).

[jeff-hykin]

I expect you mainly need a tool

I'm sorry we are still failing to communicate this: we expect nix to be that tool / incorporate that functionality. Only nix can create/enforce an official versioning standard, and until that happens no tool will have a canonical way search versions.

some tool for that, e.g. this one

The tool is a work-around and exists because a community member had the need so frequently, and the existing method fell so short, that they took the time to build and host an entire website as a make-shift solution which: doesn't search all packages, breaks on non-normal version methods, and breaks when versions are stored as derivations. Lazamar isn't the only one who took the time to create their own workaround, Matthew Bauer wrote a command line one thats more convenient, but it misses more versions than lazamar's tool. Having these tools as the pseudo-official solution really undermines the reliable development cycle I believe nix is trying to create.

[jeff-hykin]

I just don't understand how nix has the best command line installer I've ever seen, a one-liner complete with colors, transparent sudo usage, and explicit mention of all changes to my system. But then immediately falls on its face when I try to complete step 1: install my first package with a version that matches the version needed for my pre-existing project. I shouldn't need to use a community-member-made self-hosted website just to complete step 1 of using a package manager to install a very popular version of a package.

Its like getting a car, but not the keys to the car, and then having to search the internet for where to find the keys, and then having someone refer you to their friend that knows where to get lots (but not all) keys, and that friend can probably get a key that's good-enough if you jiggle it to get it to fit in the ignition. You could search through every possible key in the world yourself, or you could just decide to take the car to the shop and pay to have them change the ignition, either way is less than ideal.

[jeff-hykin]

Is this maybe what you are looking for: https://matthewbauer.us/blog/all-the-versions.html

Thanks a lot for this @markuskowa, that is much closer to what I've been looking for. Sadly it lists less versions than lazamar's but it's much easier to use. You might want to add the example to your comment so people don't need to read most/all of the blog post to get the answer. Maybe something like:

To get versions for emacs run

nix eval "(builtins.attrNames (import (builtins.fetchurl https://matthewbauer.us/generate-versions.nix) {}).emacs)"

(it will take awhile)

To get versions for firefox run

nix eval "(builtins.attrNames (import (builtins.fetchurl https://matthewbauer.us/generate-versions.nix) {}).firefox)"

To install version 25.0.1 of firefox run

LC_ALL=C nix run "(import (builtins.fetchurl https://matthewbauer.us/generate-versions.nix) {}).firefox.\"25.0.1\"" -c firefox

[vcunat]

Having these tools as the pseudo-official solution really undermines the reliable development cycle I believe nix is trying to create.

I believe that stuff discussed in this thread so far has never been meant among the main applications of NixPkgs, but I don't see that as a problem. Don't get me wrong, I'm (personally) not at all opposed to having some "more official" way of finding the versions you need.

But the process starts the same – people from the community implement stuff and then it gets merged, documented, etc. For big changes it may be advisable to go through an RFC beforehand (if you want to avoid working on something that gets refused in the end).

[jeff-hykin]

But the process starts the same – people from the community implement stuff

Fair enough, thats a really good point 👍

it may be advisable to go through an RFC

Thanks for the advice, I didn't know of nix's RFC system. I think this issue is hoping to create a community discussion to develop an actionable of a plan of how to best integrate versions with nix. Basically: listen to people's use-cases, hear from users who understand nix's feature set and goals, talk to design a system that is ideal both to use and to implement, then, if it requires major changes to nix, go the route of an RFC and develop a pull request.

[jeff-hykin]

so far has never been meant among the main applications of NixPkgs

I think I am genuinely misunderstanding some aspect about nix. Maybe this thread isn't the place for this question: but what are the other main applications? I've read the Nix Package Manager Guide, but everything in there about reproducibility seems to point to installing (and therefore finding) specific versions. I mean the first title in chapter 1 is multiple versions

Any other multiple-simultaneous-version managers: asdf, kiex, gvm, rvm, rbenv, pyenv, jenv, nvm, Renv, luavm, all have a list-available-versions as either their 1st or top-5 most-used commands. But nix has 5.3K stars, so there must be something I'm not understanding if nix is missing something so vital

[vcunat]

OK, I'll try to keep explaining :thinking:

Command that lists versions... has always been there:

$ nix-env -qa llvm | cat
llvm-10.0.0
llvm-5.0.2
llvm-6.0.1
llvm-7.1.0
llvm-7.1.0
llvm-8.0.1
llvm-9.0.1

but in NixPkgs we choose to restrict the number of maintained versions (which has been discussed at length in these threads), so there are few packages that have multiple versions and having this many is very rare. That approach is take by most distros, because QA is best that way. I understand that this combination can seem confusing.

But even that way the multi-versioning gets utilized a lot, e.g. for rollbacks. Perhaps it's worth noting that with Nix the "installed versions" bring all transitive dependencies down to libc and they can contain configuration (or whole OS), contrary to typical multi-version management.

So instead we focused this discussion on installing versions from other/"historic" NixPkgs commits, and there's no built-in tool to search in there... well I tried to explain I personally see limited use for that. I think usually people choose the NixPkgs version they want (say, latest release-20.03) and not some particular package version (apart from cases like LLVM) – or alternatively they already had some package(s) and they keep them "unupdated" while replacing others with different versions. Also Flakes are now being introduced as another way of "managing versions".

[jeff-hykin]

Okay, that was helpful and better explains https://github.com/NixOS/nixpkgs/issues/9682#issuecomment-138105076

In that case though, why would Nix opt to concurrently maintain multiple versions when historic versions can be depended upon? In other words, why do this (my current understanding of the packages):

current
    llvm-10.0.0
    llvm-5.0.2
    llvm-6.0.1
    llvm-7.1.0
    llvm-7.1.0
    llvm-8.0.1
    llvm-9.0.1

historic commit#a35lk6djfi3
    llvm-5.0.2
    llvm-6.0.1
    llvm-7.1.0
    llvm-7.1.0
    llvm-8.0.1
    llvm-9.0.1

historic commit#fjo350395
    llvm-5.0.2
    llvm-6.0.1
    llvm-7.1.0
    llvm-7.1.0
    llvm-8.0.0

Instead of doing this:

current
    llvm-10.0.0

historic commit#a35lk6djfi3
    llvm-9.0.1

historic commit#a35lk6djfi3
    llvm-9.0.0

historic commit#G98olk6djf
    llvm-8.0.1

historic commit#C35lk6djfi3
    llvm-8.0.0

historic commit#B252335lk6
    llvm-7.1.0

historic commit#Hfh593fjo35
    llvm-7.0.0

If the answer has to do with this:

bring all transitive dependencies down to libc

Why can't current-packages depend on packages in historic commits?

[vcunat]

They could. Well, it's not as simple due to many "packages" being involved in the dependency graph, but some variants of this are not difficult. If there's a good use case for some old versions, we just bring them to current NixPkgs, because it's easier to manage. EDIT: well, Flakes will be a comfortable way, too, IIRC.

[jeff-hykin]

because it's easier to manage

is there a resource I could read to explain what makes it easier? Closest thing I have found is a video clip here

Without detailed knowledge, it seems to me that it would be better to create a consistent flat system (that depends on history) rather than some kind of mixed system with an arbitrary "good-enough" use case as a decision point. Why have a decision point instead of creating tooling that would make it easy to use any historic version, regardless of who thinks a version is good enough or not.

[vcunat]

That way it's easier e.g. to fix serious (security) problems in the packages, including all their dependencies. Also, packages don't live in isolation – one NixPkgs commit is a set of packages that should work together well (for example, the build farm runs VM tests on full systems). A different combination (of historic versions) might work well or it might not, especially for packages that are more tightly coupled to each other.

[thomastjeffery]

Don't forget that there are two timelines happening here:

1. The timeline for versions of a package.
2. The timeline for the derivation implementing a specific package version.

Sometimes, you want to change something in a derivation that implements an older package version. For example, the source for llvm-8.0.1 might be moved tomorrow, which would break any new builds of your llvm-8.0.1 derivation.

[jeff-hykin]

I believe I have a relatively simple back-end solution; a database with mappings for every package e.g. (package version) => (commit hash). Every commit to master would trigger an update to the database using a CI hook.

This would also optimize nix-env -qA since it's currently significantly slower than doing a curl call to lazamars site. It should also partly address your recently mentioned concerns @thomastjeffery

I believe currently the channel(s) are being scanned for versions of a package, which is pretty inefficient. Lazamar's tool is doing 5 week snapshots which is missing some versions.

The "database" need not be an actual database, although it certainly could be. In theory it could be a single JSON file although that would be impractical. To get around having one giant file, there could be a json file for each package, with versions as the keys and commit hashes as values E.g.
nameOfPackage.json (package version) => (commit hash)
The data structure could also be made more complex to show which versions of the package were available on a stable channel.

Note the commit hash would not be the first (aka oldest) commit with the version but instead the chronologically most-recent commit with the version. This is intentional to address the issue @thomastjeffery mentioned where the source of the version changed, even though the version did not.

I've built similar versions of all the components needed to implement this, the only thing slowing me down is learning nix, which I have sunk a considerable amount of time into between reading the manual and trying to understand nix --help.

Examples for the following would be really helpful for making progress on this

1. A command for searching for versions of a package within a specific commit
2. [completed] A command for listing all package names from specific commit
3. [completed/discovered] A command for installing a package derivation from a pinned commit
4. [completed/discovered] The code for a shell.nix that would install a specific deviation from a pinned package

If I can get those working then there will be a clear path towards an RFC

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/re-using-nixpkgs-derivations-for-different-package-versions/9236/2

[vivook]

I'm glad I found this thread and the one it came from. I just installed nix and have just uninstalled it. There are 2 reasons:

1 - not possible to search for which package contains a binary (e.g. where is sed?) 2 - not possible to pin to arbitrary versions without jumping through the hoops described here

It's a shame. I thought this could be a good solution to building reproducible developer environments like the web site says, but I guess not. It's beyond me why @jeff-hykin had to write so many posts to explain the simple concept of point 2 above.

Farewell nix :-1:

[rsynnest]

1 - not possible to search for which package contains a binary (e.g. where is sed?)

@vivook this is totally doable! You can install nix-index which provides the nix-locate command which does exactly what you need. I discovered this via the very handy Nix to Debian phrasebook

curl -L https://nixos.org/nix/install | sh  # reinstall nix :)
nix-env -iA nixpkgs.nix-index           # install nix-index
nix-index                   # build the index, takes a while (~5 minutes)
nix-locate --top-level --minimal --at-root --whole-name "/bin/sed"
# RESULTS:
# toybox.out
# gnused_422.out
# gnused.out
# busybox.out

Another cool approach is comma, which transparently runs a nixpkgs binary without actually installing anything to the system. Great for trying stuff out. So in your case you would install comma then run , sed --help

You can also search packages the traditional way with nix search. You may have to get creative with such a short common name like "sed", but remember nix search also searches package descriptions. nix search " sed" (note the leading space) works great, returns 5 packages with the 2nd one being sed. Most things are much easier to find.

[rsynnest]

Back to the topic of discussion, to me, this version history feature seems like a worthwhile addition. Communities like Ruby, Python, etc. provide dozens of versions going back decades, despite the fact they are no longer maintained, so why not Nix? The popularity of tools like nvm, pyenv, etc shows there is a clear user demand, and I think Nix is uniquely capable of offering a clean solution to this common user problem.

@jeff-hykin 's proposal of a database tracking versions seems like a good approach to an up-to-date metadata service like this. This seems like it would be a good fit for integrating into NixOS Search.

Guix has some similar tooling setup for browsing past versions of packages and even comparing derivations across versions. Perhaps this upcoming RFC could borrow from there or reach out for tips? Judging by their README they are doing something very similar to what @jeff-hykin is proposing.

[jeff-hykin]

@vivook even though this isn't really the place for this discussion: I think nix is the foundation of the solution you've been looking for (for reproducible environments). With great great great difficulty, I've managed to start using it in my projects.

However, it's really too young to really use right now. This issue (and really searching for packages in general) is the largest problem IMO, but there are handfuls of other small issues that make it near impossible to use nix. The problems are simple, and often the answers are simple, but finding them is like being a blind man searching for a needle in a haystack. I can't even setup a python project without creating multiple forum posts after days of reading documentation, blogs, and performing trial-and-error tests.

You may want to check back in 5 years to see if massive wrapper tool has been created. Right now there are tons of good tools (like lazamars site) but they're as hard to find as package versions, and you need like 12 of them to have a manageable project.

[jeff-hykin]

Thanks for the info and support @rsynnest.

And as an update for the issue on hand. I think I do have the pieces to build a search tool, but currently not the time to build a finished product.

For others interested the pseudo code would be something like:

• git clone nixpkgs
• use git to list every commit git log --format="%H" | cat
• use this command to list all packages + versions for a particular commit nix-env -f nixpkgs=https://github.com/NixOS/nixpkgs/archive/${COMMIT_HASH}.tar.gz -q --attr-path --available
• iterate through all commits

Ideally the --json --meta would be used in that nix-env command but it fails on mac with

error: assertion (stdenv).isLinux failed at /nix/store/x0q87hvyab6431g84iswgr92qz5wngaw-nixpkgs-20.09pre240426.f9567594d5a/nixpkgs/pkgs/os-specific/linux/kernel/generic.nix:61:1

That part is easy, (slow and huge, but easy). I made a node.js script here that works.

The harder part will be the github action hook and creating a central place to search this information from.

[jeff-hykin]

This issue is now being blocked by https://github.com/NixOS/nixpkgs/issues/101094

I ran the script (45 hour runtime) and built the 20gb version index (uncompressed), only to realize I must use the --json option because there is no way to extract the pname from the version name. The pname can contain dashes and the version can contain dashes and they're joined by a dash. And, there some packages both sides use dashes in the name, meaning theres no way to know what part is the name and what part is the version (I should've remembered the title of issue) . There's no way (AFAIK) to get the version and pname separately without the --json option.

The revised script is ready, but fails because of #101094. The script is somewhat more complicated now because it needs to enumerate the commit hashes and pnames in order to keep the output size a few orders of magnitude below the 20Gb level.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/clarification-on-package-names-and-versions/9819/3

[yakman2020]

I want to install Lua 5.0.x globally (assume I don't care about ANYTHING else)

I'm afraid I don't remember the prior art on this from when I used other distros: How is this done with apt-get and other common tools for installing programs globally?

apt-cache search python

: : (gets a bunch of packages)

apt install python-3.6.3

'
Of course this is only completely robust in a container, since other projects may want python-3.7.8 and barf with 3.6.3.....

[acolinisi]

IIUC, currently the answer to these threads is essentially: obtain a recipe for whatever version you need (by scouring the VCS history of Nixpkgs repo (yuck!), finding the recipe in some other recipe store / channel, or by writing the recipe yourself), then add the recipe to the recipe store (e.g. fork+edit Nixpkgs repo, or point into some other repo/channel which has the recipe you're after), and refer to it by plain name in shell.nix -- version identifiers are thus unnecessary. I'm attempting to suggest a small functionality patch to enable a slightly improved answer good for FAQ, to help people landing on these threads (as I have) after realizing "Nix is just what I need, it's better than Gentoo (Prefix)! oh wait... shell.nix can't have version specifiers? and the Nix recipe store has recipes for fewer versions than in Gentoo?".

The goal is to be able to answer: yes, Nix supports specifying versions (not range constraints!) in shell.nix (e.g, via cmake=3.3) and Nix will build that version of the package as long as there is a recipe for that version in your recipe store; but don't expect a binary package because only one version is maintained and built; and Nix doesn't resolve versions from version constraints (a la Portage). This seems simpler to grasp and precludes getting lost in discussions about mining VCS history and channels and pinning recipe stores.

Introducing a dependency between Nix and the version control system of the recipe store does not seem right. VCS history mining (and tools to do it) enters this discussion only due to the choice to do version bumps in-place via an edit instead of via a copy of the build recipe file. Example: updating pkg from 1.1 to 1.2 involves editing the tarball hash in the build recipe in existing pkg/1.1.nix file and renaming it to pkg/1.2.nix. Instead, if you left pkg/1.1.nix in the repo, you would eliminate the dirty proposition of searching VCS history for that pkg/1.1.nix.

Communities like Ruby, Python, etc. provide dozens of versions going back decades, despite the fact they are no longer maintained, so why not Nix?

This. Upstreams host heavy tarballs for all versions, language-specific package repos (pip, cargo, etc) host all versions, on their infrastructure. Nixpkgs hosts light build recipes (on free Github). Would it be doable to let Nixpkgs host recipes for any versions that anyone cared to contribute a recipe for, whether older or newer than the current curated version? The advantage of hosting in the centralized Nixpkgs repo rather than in my private repo is to reduce duplication of effort -- to save the next person the trouble of writing the recipe of a version, i.e. hunting down the deps, figuring out which build config flags changed, and working around compilation/linking issues. Obviously there's no guarantee that what built against package set I used will build against your package set, but one is a very good starting point for the other.

but in NixPkgs we choose to restrict the number of maintained versions (which has been discussed at length in these threads),

In contrast to infeasible cost of maintaining/building/testing of packages in infinite combinations, the cost of hosting recipes is negligible, isn't it? Hosting .nix files doesn't involve building them at all. Nobody expects that someone will build+test+maintain every random combination of versions I happen to need and provide binary packages for me! What's asked of Nix is a way to easily build and use any version, for which there is a recipe, against a package set I define. The only maintenance cost added by more .nix files is the refactoring cost when something in Nix language or in shared .nix module changes that requires all .nix files to be updated; adding more versions in each package does increase this cost, but such changes are already costly undertakings, doesn't seem like situation would worsen prohibitively, no?

Gentoo follows a policy that's half-way there: minor version bumps are edit+rename (like in Nix), major ones are copy(+edit). So in Gentoo like in Nix, recipe being in repo implies the software is maintained against rest of recipes in repo. (I'm arguing against this here.) Also, Gentoo cannot keep .ebuilds, because more versions increase complexity of the constraint satisfaction problem that Portage has to solve (figure out a collection of versions that satisfy version constraints in dependencies declared by build recipes). This problem does not arise with Nix by design, and I am not trying to introduce it with making versions first-class.

If we decouple hosting of recipes from the curating a coherent set of packages, then what is missing is hardly much:

1. Policy change to keep .nix for all versions in the repo instead of edit+rename. To reduce bloat, add some facilities in the recipe language/API for associating the same recipe with multiple versions (i.e. define tarball urls+hashes for more than one version, so similar effect to creating multiple symlinks to same .ebuild file in Gentoo). Also, factoring common logic shared across recipes for related versions into modules would reduce duplication.

2. A mechanism for marking one of the versions as the one that's part of the curated package set to be built for the binary cache (doesn't have to be the latest one).

3. First-class status for version specifications so that build recipes can declare what version they build (they already do), and so that shell.nix and such can force a version, like cmake==3.3 (no ranges!), and so that Nix can resolve such a versioned pkg identifier to the .nix build recipe. IIUC, the implementation for this is virtually identical to implementation of version as part of name, like cmake_3_3, the benefit is just that the information is now structured, so that search queries can use name and version in intuitive ways (e.g. search my store for recipes for cmake <3.0, instead of some regex relying on naming convention), and so that identifiers in shell.nix are more consistent (instead of, e.g., gcc6 and foo_3_1). Identifiers without a version resolve to the ONE curated version as they do now. Unlike in Portage (and Pip?), this does NOT create a constraint satisfaction problem of automatically choosing versions given constraints. There are no constraints, just a way to identify build recipes by (name, version) instead of by (astringwithnameandversion).

4. [optional bonus] Allow recipes to specify version ranges for documentation and checking purposes only (not for automatic dependency resolution!). A recipe can declare cmake>=3.3. This constraint does NOT affect resolution into a version (resolved version would be either the curated version as it is now, or the version set in shell.nix with versioned identifiers, like cmake=3.4). However, Nix may check the resolved version against these constraints, and a failure can be reported if you're trying to select cmake=2.0 but something you're pulling in declares cmake>=3.3. This implies that the versions in the curated set will always satisfy all such constraints present within that same set.

Would this superficial improvement not resolve these threads?

[jeff-hykin]

@acolinisi, so something like this?

Bob's repo of Bob's Lang

// packageInfo.json (or a .nix equivlent)

"package": {
    "uuid": "bob1234", 
    "name": "bob_lang",
    "hostUrl": "http://github/bob1234/bobs_lang/master", // for issues
    "defaultVersion": "1.0.1-stable",
    "versions": {
        "1.0.1-beta": { /* package info (hash/url/dependences) */ },
        "1.0.1-stable": { /* package info (hash/url/dependences) */ },
        "1.0.0-stable": { /* package info (hash/url/dependences) */ },
        "0.7.1-stable": { /* package info (hash/url/dependences) */ },
    },
}

Then in the console/terminal

$ nix publish "$BOBS_REPO_PATH"
>  error no auth information, can't confirm you're allowed to use "uuid": "bob1234", 

# adds auth information from nix-discourse into a .config
$ nix publish "$BOBS_REPO_PATH"
> Thanks, other people will now be able to find your package when searching

$ nix seach bob_lang
> [searching database of 3rd party packages]
> found 
>     bob1234/bob_lang

$ nix seach bob_lang --verions
> [searching database of 3rd party packages]
> found 
>     bob1234/bob_lang  1.0.1-beta
>     bob1234/bob_lang  1.0.1-stable
>     bob1234/bob_lang  1.0.0-stable
>     bob1234/bob_lang  0.7.1-stable

$ nix install bob1234/bob_lang@1.0.1-beta
> getting the hashsum
> add the following to your shell.nix
>
>    bobs_lang = nixpkg.something { hashsum = "FN24...467K5"; gitFetch="git/exact/commit-hash"; }
>

[jeff-hykin]

I've resigned myself to using lazamar's tool (TLDR link, also his full explaination) until I can find a replacement for nixpkg. If that^ (the "bob's repo" example above) kind of a system was implmented I'd be interested.

But until then, my recommendation to others is
Turn Back; Here be Dragons
2022 update: Come back in 5 years, when Flake-Packages are stable

What changed my opinion? I read the nixpkg source code
I spent several weeks, but it only takes 30 seconds to find the 3+ year old FIXME's, TODO's, and commented out code because they're all over the place. I didn't think functional code could be this convoluted, but there's enough spaghetti execution to choke a whale--there's literally intentionally-invalid code acting as a means of control flow. There's dynamic OS-specific execution thats about as pure as Somilia's drinking water, no naming standards, no formatting standards, no calling conventions, no seperation of builtin/essential packages from misc junk that doesn't even build on the systems it says it supports.

By itself, all of that is fine; I've got repos messy enough to choke whales too. The problem starts with nixpkg portraying itself as "stable pure reliable" when its not. And I can tolerate that too; this is open source software, and effectively all of us contribute to open source code out of good intentions, even if we fall short. The root reason for me is; the maintainers seem indifferent to disscussions around fixing foundational problems that totally undermine reliability/stability/purity. How can I possibly care about the new flakes system when there isn't even a canonical name/version for packages? this (original) issue was opened in 2015, its not fixable without a major PR/rewrite. I thought if I learned the system I could create a wrapper to make it usable & stable, but at this point I don't even think a fork of nixpkg could save it.

The issue with a particular package ? => just create an issue on the package manager (nixpkg) repo and want to publish a tiny update to your package? => you need a PR against nixpkg directly were just nails in the coffin below the flashing "this is un-maintenable/un-scaleable" sign 😕

It would be easier for me to start from scratch than it would be for me to contribute a makeshift solution better than Lazamar's.

[jeff-hykin]

(Thanks @milahu. Comment updated, and reworded to be at the top 👍)

[axb21]

Maybe this is a naive question but: how does nix-env -q do it, and why is that not a satisficing solution?

When I run that command on my machine, I see:

me@machine:~$ nix-env -q
coursier-2.0.16
joe-4.6
nix-2.3.10
openjdk-15.0.1-ga-jre
sbt-1.4.9
scala-2.13.5

i.e., packages with the sorts of version numbers you expect to see for those packages. If those version numbers "violate the nix philosophy" (lol, come on), why are they being displayed by one of the workhorse tools of nix? Why is there so much angst and tribulation about finding these numbers and using them to control which packages are installed, when nix-env is supplying them? Why is it not possible for me to express, using nix, "hey, I know you really want to install scala-2.13.5 for some reason, but I prefer scala-2.12.11" without forcing me to search around for hashes that nobody but a computer wants to know about? Humans should not be doing things, like searching for and copy-pasting hash strings, that computers are much better, faster, and more reliable at doing.

This is such a common and important use case it's remarkable it hasn't been addressed somehow. How does nix-env -q do it?

[06kellyjac]

nix-env --query is showing the version that you have installed but it doesn't have capabilities to search through git/channel history for previously existing versions to try install

kinda like on a rolling distro like arch you can see your package versions but you can't natively roll back, you need to manually install the package from Arch Linux Archive or an external tool that searches ALA and does it for you.

[dnr]

A lot of this thread has been talking about finding easier ways to identify historic nixpkgs commits that contain certain versions, and then installing packages from those. It's been mentioned but maybe not emphasized: you probably don't want to do that. Because of how Nix works, when you install packages from a commit, you get the package built with all its dependencies all the way down to libc at that point in time (which probably have more security vulnerabilities, etc.). If the target package is really old, then maybe that's what you want, because it won't build with more recent dependencies, but I imagine more often you want to just build an older version as it would be built if it was still part of the current system, or if you downloaded the source and build it manually on the current system.

The best way to do that, in most cases, would be overriding the version and/or src attribute of the package you want. You can do that easily in an overlay, or in shell.nix, or wherever you like. The package will get built from source, because you're building it against a different set of dependencies.

I just did a git log in my nixpkgs and picked a package at random: squid, currently at 4.14. Let's say I wanted to use 4.6. I'd do this (copying version and src from pkgs/servers/squid/default.nix and then changing them:

with import <nixpkgs> {
  overlays = [
    (self: super: {
      squid_4_6 = super.squid.overrideAttrs (old: rec {
        version = "4.6";
        src = self.fetchurl {
          url = "http://www.squid-cache.org/Versions/v4/${old.pname}-${version}.tar.xz";
          sha256 = "0h9x63h0ljz4w1arj1rqhcyiwiqn89fdy1f6qi153454sgjssnq1";
        };
      });
    })
  ];
};
mkShell {
  buildInputs = [ squid_4_6 ];
}

If I had to go back to 3.5, that's a little harder. I'd do git log pkgs/servers/squid and find the commit that removed 3.x (948b3e34a51e), and copy the derivation out of that and stick it in my overlay.

I tried to do the version/src thing with @jeff-hykin's example of Lua 5.0, but the Lua build is too complicated (it has a bunch of system- and version-dependent patches). I ended up using git log to find the commit that removed 5.0 (544eaaa52b59) and copying the derivation out of that, and it worked fine.

Should that derivation be available in nixpkgs or a user-contributed library of derivations? Well, the commit message says that Lua 5.0 has open security issues that are not fixed... it doesn't seem unreasonable to me that someone who needs such an old version might have to spend a few minutes getting it working.

Btw, the example given above to install Python 3.6 on Ubuntu/Debian doesn't work anymore even on Ubuntu 20.04, so I don't think it's accurate to say that other distros have solved this problem. When I used Ubuntu (and Redhat long before that) I remember trying to install older versions by downloading .debs/.rpms for previous OS releases and installing them manually, and then fighting through various library incompatibilities... with Nix it might take more work up front to get what you're looking for but at least you know it'll work once you build it.

---

A final point worth noting: fundamentally, nix works with derivation hashes and not version numbers because version numbers don't really mean all that much. Upstream might re-release a last-minute bug fix with the same number, nixpkgs (and other distros) apply patches on top of upstream, build flags turn on and off parts of the code, etc. A version number isn't enough to fully specify what you want to be installed/built, so having to be very explicit about what you want (either copying an old derivation, overriding attributes of one, or pulling one from an older commit) seems natural and expected in the nix world.

[jeff-hykin]

@dnr You've got good points. (I'd don't mean this comment as a rejection, just a clarification)

the example given above to install Python 3.6 on Ubuntu/Debian doesn't work anymore even on Ubuntu 20.04, so I don't think it's accurate to say that other distros have solved this problem.

... and that's exactly why I'm still using nixpkgs :/

you probably don't want to do that. Because of how Nix works, when you install packages from a commit, you get the package built with all its dependencies all the way down to libc at that point in time (which probably have more security vulnerabilities, etc.).

Actually that's exactly what I want to do. I'm running these environments in isolated containers. I don't care if it takes 30Gb of storage, 13 hours of compiling from source, has massive 0-day vulnerabilities, and charges my bank account $5 every time I use it. I just want the code to be 100% immune to bit-rot. Its not just spending a few minutes getting old things to work, I have teams of people spending weeks, sometime months trying to solve the "well it works on my machine" problem. I had an assignment just last week where the professor required us to use python 2.6.7 (!). Once the environment is working (vulnerabilities and all) then it becomes practical to roll dependencies forward one at a time by rolling back and debugging if there is a problem.

The best way to do that, in most cases, would be overriding the version and/or src attribute of the package you want. You can do that easily in an overlay, or in shell.nix, or wherever you like. The package will get built from source, because you're building it against a different set of dependencies.

This is really really good to know. And thank you for the example. I would not have understood without the code and the explanation of your strategy of how you found the info.

A final point worth noting: fundamentally, nix works with derivation hashes and not version numbers

I agree. The hash is the ID, I just want the version and name to be metadata for searchability. Actually, I wish packages couldn't be installed by name, I wish we could only search by name but had to run the install with their hash identifier.

[howdoicomputer]

Sorry, I'm new to NixOS and am trying to build a vmware appliance that contains a working development environment. This is my first exposure to Nix and is a side project that is intended to help me learn as well as potentially solve a real business need at work. I've read through this thread and through the other thread (well, not a full read because I don't have hours) and I'm still confused.

If I want to install Terraform 0.11.14 and Terraform 0.14.X and then switch between them... how do I do that? Do I continue to use tooling like asdf? Do I curl binaries and put them in my $PATH like the caveman days? The Terraform package in the NixOS packages doesn't even have a 0.14. Do I write my own modules? Does NixOS not fit my use case?

I'm not an intelligent man. What is the simplest answer to this question for Nix/NixOS?

[alexeicolin]

I don't care if it takes 30Gb of storage, 13 hours of compiling from source, I just want the code to be 100% immune to bit-rot.

Exactly my use case, too. Gentoo Prefix has been fulfilling this use case well for me successfully. Especially, coupled with git-lfs where I store the source tarballs. A complete self-contained system with versions of everything clear and frozen and everything built from source, just add a kernel (even macOS Darwin is supported). Portage supports whatever set of versions you want to leave exposed as options without any VCS archeology. Excuse the marketing.

[dnr]

If I want to install Terraform 0.11.14 and Terraform 0.14.X and then switch between them... how do I do that?

This sounds like a perfect case for Nix (you don't need NixOS). Multiple versions of terraform are maintained in nixpkgs, except it looks like 0.11 was dropped recently, so you'll have to reach back for that.

Install nix, install direnv, and then put this in a shell.nix in the directory with your terraform 0.14:

{ pkgs ? import (fetchTarball {
    # first commit with 0.14.11, but this could be any commit on nixpkgs-unstable
    url = "https://github.com/NixOS/nixpkgs/archive/3dd11611cfb92fb24f9ec65b8c16927893e3a81b.tar.gz";
    sha256 = "01xra6zv4lsjynv67cvwds9w4i1gbnvy9sab3yv1v0ip6zyc46mn";
  }) { } }:
with pkgs;
mkShell {
  buildInputs = [
    (terraform_0_14.withPlugins (tp: [
      tp.aws
    ]))
  ];
}

(add whatever plugins you need). Also put use nix in .envrc for direnv.

Then put this in your directory with terraform 0.11:

{ pkgs ? import (fetchTarball {
    # last commit with 0.11
    url = "https://github.com/NixOS/nixpkgs/archive/209bdc8ddf9693c1ce67a643da4075a3a12eb427.tar.gz";
    sha256 = "0jirbavwdh4hq04bfw6fi7mx6fv4al27lksbmb16sq90dksk3wh0";
  }) { } }:
with pkgs;
mkShell {
  buildInputs = [
    (terraform_0_11.withPlugins (tp: [
      tp.aws
    ]))
  ];
}

I found those commits by looking here: https://github.com/NixOS/nixpkgs/commits/nixpkgs-unstable/pkgs/applications/networking/cluster/terraform/default.nix (and then using nix-prefetch-url to get the hashes)

When using terraform_0_x.withPlugins, that will provide a single packaged version of all the plugins you list (built as of the same commit as terraform itself), and terraform will use that one preferentially, as long as your provider version specifier in your terraform matches that one. If it doesn't, it'll try to download the provider and nix will no longer be able to provide any guarantees about reproducibility (but terraform may provide some).

---

I haven't used asdf so I can't compare to that. It might be simpler for this case. I'd say the benefit of Nix is that you can manage literally anything on your system in the same way, once you learn how (which is admittedly not easy).

[howdoicomputer]

Thank you @dnr. Super helpful. I had a feeling that Nix would be able to accomplish this since the documentation literally yells it at you as a selling point but, boy howdy, I could not figure out how to do it. Like at all.

Now that I read your post I can see why this issue exists; this is not at all ergonomic or straightforward. Shouldn't this be in the documentation somewhere? I'm brand new to Nix so I don't know if documentation for this exists (maybe in a "piece it together from several sources" kind of way).

Oh, and I picked NixOS because the build tool chain for producing an artifact is literally fantastic. I was originally looking at archiso but the method for producing an OVA or an ISO was easy breezy. For example, the generators project is just amazing. It's like producing static binaries in Go. Sure, every language can do it but it felt really easy using the generator project.

One last question, when you say Then put this in your directory with terraform 0.11: do you mean in the same directory as a module/project? Or in your configuration.nix? This looks like build instructions but I'm more interested in "use this version globally but use this version locally when needed." That's a pretty common use case for programming languages. Especially pertinent in this use case because Terraform binary versions can be incompatible with the state that's created by other versions. Can I put both blocks in configuration.nix, have them installed into the OVA, and then use nix-env to load a specific version?

[dnr]

@howdoicomputer This is getting a little off-topic from this issue and I don't want to clutter it up, so we could take it to email? I'm happy to help though, I struggled through this stuff ~6 months ago myself.

Quick answers:

The best documentation for stuff like this is https://nix.dev, specifically https://nix.dev/tutorials/declarative-and-reproducible-developer-environments.html

Those snippets are intended for a shell.nix in the directory with your terraform module, so that it'll activate (with direnv) when you cd to that directory.

Sure, you can put just the (terraform_0_14.withPlugins ...) bit in your environment.systemPackages to get a system-wide default. If you're doing that you might want to put them both in overlays and refer to the overlay packages in shell.nix. Then it's also easier to make sure they end up in the vm image instead of getting fetched on-demand.

Don't use nix-env for anything.

[axb21]

A final point worth noting: fundamentally, nix works with derivation hashes and not version numbers because version numbers don't really mean all that much.

I mean, I get the points being made in this thread, but this above simply is not true. It's just not, and I wish people would stop saying it. Human beings can read and remember version numbers, but they cannot read and remember derivation hashes. Most of us can't, anyway. And since that's largely true, that is a meaning that version numbers have that derivation hashes lack. You may personally choose to ignore or downplay that meaning, but your choice does not mean it isn't there.

When I run this:

me@machine:~$ ls -1 /nix/store/*drv | perl -ne '($h,$p,$v) = m|/nix/store/([^\-]+?)\-(.*?)\-([0-9\.]+).drv|; print "$p version $v: $h\n" if $h' | head -5

I get this output:

hxt-unicode version 9.0.2.4: 000406sq23a5grwyaz2aiyk8n1jj73fv
python3.8-idna version 2.10: 00126z5icidi5fsla0lpb5clxmy30jr7
valgrind version 3.16.1: 0022ahvxd56v75xby9ygxvyrfiwfpgvm
autoconf version 2.69: 002dywnjxm4m857ni8wmvrz7d2xy4lx4
python3.8-pylama version 7.7.1: 007x7siimzjaf01wajn624sa0xbki4s7

How is that not already partway there? Run some variation of this in a big /nix/store to build up a lookup table and write a tool that uses it to do the required mapping? There are 11394 lines output from this when I run it on my own computer without head limiting it.

I also poked through approximately 100 packages in nixpkgs, ranging from dash to calibre-web to a bunch of fonts, and every single one of them had a version string in default.nix. Not the nix way? Come on, it's right there!

[milahu]

Run some variation of this in a big /nix/store to build up a lookup table and write a tool that uses it to do the required mapping?

such a big /nix/store does not exist only some versions are in public binary caches build declarations can be impure so the hash would change

probably wont get better than 1. try pinning the version with overrideAttrs (wiki) 2. if the old version requires a different build script, use lazamar's tool at https://lazamar.co.uk/nix-versions/ to find the corresponding nixpkgs version, and use the build script from there 2.a. or, to save some disk space, download only the package's subfolder from the old nixpkgs, and use let some-package = pkgs.callPackage ./extra-pkgs/some-package {}; in { ... config ... }

[stale[bot]]

I marked this as stale due to inactivity. → More info