NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
18.13k stars 14.17k forks source link

tarball: packages.json is missing a couple thousand packages #102508

Closed Atemu closed 3 years ago

Atemu commented 4 years ago

Describe the bug A clear and concise description of what the bug is.

The packages.json inside the tarball contains a list of packages in Nixpkgs.

However, this JSON is missing a bunch of packages only contained in sub-package-sets; most notably ~5000 emacsPackages, ~4000 texlive packages and a whole bunch of smaller package sets.
Also see: https://github.com/repology/repology-rules/issues/396

Since, countrary to most other repositories, we're not a flat list but rather a nested set and also contain functions and other non-packages, generating a flat list of packages is a non-trivial task. We have a script for this task but it only recurses into haskellPackages and rPackages currently.

The new NixOS search have adapted this script and made it recurse into a few more sets which should be nearly all packages users might want to search for.
We should at least port this back to Nixpkgs.

As this JSON isn't used for our package search anymore but rather academic purposes like Repology, I think including all named packages available to the user in Nixpkgs might not be a bad idea.
So, I went at it the other way around and started with all attrsets at the top-level and stripped away all the things that aren't package sets, are a copy of the top-level set or can't be evaluated:

https://github.com/NixOS/nixpkgs/issues/102509

Notify maintainers

cc Marketing Team as the list in my local JSON went from 63341 to 71370 and 98640 packages long respectively and I'm sure such large numbers would look nice on Repology and the homepage. ;)

This also means that we're now larger than the AUR, possibly even larger than AUR + Arch repos combined. :)

@edolstra @garbas @samueldr

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:
Atemu commented 3 years ago

Unstable has advanced aaand:

image

;)

jonringer commented 3 years ago

we should probably adjust the repology parser, as almost all of these new packages got added to unique packages.

Atemu commented 3 years ago

Seems like it's the emacsPackages, the numbers match up (~5000).

https://github.com/repology/repology-rules/issues/447