search

OleMussmann / Nix-Package-Search

Cache the nix package list, query and sort by relevance
MIT License
16 stars 1 forks source link

nps stops at eval failure #5

Open OleMussmann opened 1 year ago

OleMussmann commented 1 year ago

nps tries to evaluate all packages including overlays and stops at the first eval failure.

Config, logs and console output needed for debugging.

SuperSandro2000 commented 1 year ago

I ran nix run github:OleMussmann/Nix-Package-Search -- overlay and I basically have an overlay that does:

  (self: super: {
    unused = super.does-not-exist;
  })

but I currently can't reproduce this because the output and cache file is just empty.

OleMussmann commented 1 year ago

Hm, I don't know enough about your setup to figure this out yet. I probably also don't know enough about nix. Could you share your setup files?

To narrow it down: if you have an empty cache file it means that the caching process failed somehow. Subsequent calls to nps will produce empty output, because it's (rip)grepping an empty file. Not ideal, I should catch that somehow. Anyways, to reproduce the error you can force caching anew with:

nix run github:OleMussmann/Nix-Package-Search -- -r

Since the caching process is borked, could you check for me what the command

nix-env -qaP --description

does for you? Some more questions:

SuperSandro2000 commented 1 year ago

Could you share your setup files?

My setup is rather complex and without my cache you would need to compile a lot.

nix run github:OleMussmann/Nix-Package-Search -- -r

 ➜ nix run github:OleMussmann/Nix-Package-Search -- -r
Querying and caching packages...
error: attribute 'nextcloud26' missing

       at /home/sandro/.config/nixpkgs/overlays/default.nix:445:17:

          444|
          445|   nextcloud26 = prev.nextcloud26.overrideAttrs (oldAttrs: {
             |                 ^
          446|     nativeBuildInputs = oldAttrs.nativeBuildInputs or [ ] ++ [
       Did you mean one of nextcloud23, nextcloud24 or nextcloud25?

On that line is indeed a package that does not exist and nps shouldn't trip over this IMO.

...
  nextcloud24 = prev.nextcloud24.overrideAttrs (oldAttrs: {
    nativeBuildInputs = oldAttrs.nativeBuildInputs or [ ] ++ [
      compress-website
    ];
  });

  nextcloud25 = prev.nextcloud25.overrideAttrs (oldAttrs: {
    nativeBuildInputs = oldAttrs.nativeBuildInputs or [ ] ++ [
      # compress-website
    ];
  });

  nextcloud26 = prev.nextcloud26.overrideAttrs (oldAttrs: {
    nativeBuildInputs = oldAttrs.nativeBuildInputs or [ ] ++ [
      compress-website
    ];
  });
...

nix-env -qaP --description outputs the same, as the error seems to be from its execution or something similar.

On another of my machines nix-env -qaP --description does not output anything.

On all systems I don't have any channels and only flakes.

 ➜ nix-channel --list

 ➜ nix-env -qaP --description -vvvvvvv
evaluating attribute '_combineChannels'

Do you have a 'normal' /tmp folder or does your setup something funky with it?

Nope, it has more inodes than normal.

 ➜ mount | rg /tmp
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=8032412k,nr_inodes=819200)

My NIX_PATH looks like

 ➜ echo $NIX_PATH
nixpkgs=/nix/store/1l84sifmjy4ygzfk8c6h7rz88ahbkda8-source:nixos=/nix/store/1l84sifmjy4ygzfk8c6h7rz88ahbkda8-source:nixos-config=/etc/nixos/configuration.nix

If I remove the root channel that was still accidentally on of my machines, nix-env also does not output anything there. So it seems to be that the tool requires channels.

OleMussmann commented 1 year ago

Thanks a lot @SuperSandro2000 , this is very helpful. The current "stable" version of nps relies indeed on channels for scraping package information. Without channels it usually fails silently (which I need to fix), so I need to do some digging to see why it trips over your nextcloud (which I also need to fix).

The current development branch addresses this (as well as #2 and #3 ) by allowing package infos being pulled via nix search on your registry flakes instead. This is switched on with the --experimental / -e flag. You can test it here:

nix run github:OleMussmann/Nix-Package-Search/development -- -e true rustup

It not only searches your system flake, but all the other registry flakes as well. I don't think there's a way to remove global flakes from the registry, so you will often find some duplicated packages for example in the nimble# registry flake. That will be addressed in the future. Maybe by allowlist/blocklist certain flakes? We'll see. But for now, I'm curious if the experimental flag works for your complex setup.

SuperSandro2000 commented 1 year ago

Right now it fetches all flakes from the default registry. This was a bit unexpected for me. I would have expected that it only fetches nixpkgs by default.

For me it is also not working on your development branch

 ➜ nix run github:OleMussmann/Nix-Package-Search/development -- -e true rustup
Fetching registries, this might take a while...
Heads-up: not all registry flakes contain searchable packages!
Fetching agda ...
Fetching arion ...
Fetching blender-bin ...
Fetching dreampkgs ...
Fetching dwarffs ...
Fetching emacs-overlay ...
Fetching fenix ...
Fetching flake-parts ...
Fetching flake-utils ...
Fetching gemini ...
Fetching hercules-ci-effects ...
Fetching hercules-ci-agent ...
Fetching home-manager ...
Fetching hydra ...
Fetching mach-nix ...
Fetching nimble ...
Fetching nix ...
Fetching nix-darwin ...
Fetching nixops ...
Fetching nixos-hardware ...
Fetching nixos-homepage ...
Fetching nixos-search ...
Fetching nur ...
Fetching nixpkgs ...
Fetching templates ...
Fetching patchelf ...
Fetching poetry2nix ...
Fetching nix-serve ...
Fetching nickel ...
Fetching bundlers ...
Fetching pridefetch ...
Fetching helix ...
Fetching sops-nix ...
Filter registry flakes...
/nix/store/bs8rhqi1j2gmyww4wk8f61j83v2q1vap-nps/bin/.nps-wrapped: line 363: /tmp/nps/raw_caches/nixpkgs: No such file or directory
 ➜ nix registry list | rg nixpkgs
user   flake:nixpkgs github:SuperSandro2000/nixpkgs/nixpkgs-unstable
system flake:nixpkgs path:/nix/store/4gh56d4dglg1hj4pbp7m60m70vwf0d1g-source?lastModified=1673536221&narHash=sha256-uoL6LhnwWLr9OgiT5CjzltLJrd86a7+kaeauDfTIWG4=&rev=c9c073af869b95a7e7165beb0c0961d7689a375c
global flake:nixpkgs github:NixOS/nixpkgs/nixpkgs-unstable
OleMussmann commented 1 year ago

Channel searching errors, as well as having empty channels should be caught now on the development branch. I also removed caching non-system flakes and fixed some other bugs. Could you give it a try again?

nix run github:OleMussmann/Nix-Package-Search/development -- -e true rustup

If that still does not work, could you give me the output of this command?

echo "nix registry list | rg '^system *flake' | sed -e 's/^system *flake:\([^ ]*\) path.*/\1/'" | nix-shell -p gnused ripgrep --run bash

And does this one actually search your nixpkgs?

nix search $(echo "nix registry list | rg '^system *flake' | sed -e 's/^system *flake:\([^ ]*\) path.*/\1/'" | nix-shell -p gnused ripgrep --run bash)
SuperSandro2000 commented 1 year ago

Could you give it a try again?

Nothing again :(

echo "nix registry list | rg '^system *flake' | sed -e 's/^system *flake:\([^ ]*\) path.*/\1/'" | nix-shell -p gnused ripgrep --run bash

nixpkgs

nix search $(echo "nix registry list | rg '^system *flake' | sed -e 's/^system *flake:\([^ ]*\) path.*/\1/'" | nix-shell -p gnused ripgrep --run bash)

Yes, it does.

About the overlay problem see https://github.com/trofi/nix-olde/issues/2#issuecomment-1399088459 for maybe a solution.

OleMussmann commented 1 year ago

First of all, thanks a lot for your patience and help. 🙏 I created a debug branch to test a few things.

If you have the time, could you please post the terminal output of the following two commands?

This one checks if the 'overlay' error is caught. It runs the search via 'channels' and will fail, but I'd like to see if it fails gracefully this time. The possible improvement from https://github.com/trofi/nix-olde/issues/2#issuecomment-1399214691 is incorporated, but it should still fail for you since your channels are basically empty.

nix run github:OleMussmann/Nix-Package-Search/debug -- -e false -r rustup

The next one is running the search via 'nix search' and has a more robust regex filtering, hardcoded nixpkgs as search target, and much more debug output. If it fails, it should at least give more infos about what is going wrong.

nix run github:OleMussmann/Nix-Package-Search/debug -- -e true -r rustup
SuperSandro2000 commented 1 year ago

First of all, thanks a lot for your patience and help. pray I created a debug branch to test a few things.

No problem.

nix run github:OleMussmann/Nix-Package-Search/debug -- -e false -r rustup

 ➜ nix run github:OleMussmann/Nix-Package-Search/debug -- -e false -r rustup

[DEBUG] $SEARCH_FOLDER: /home/sandro/.nix-package-search
[DEBUG] $CACHE_FILE: nps.cache
[DEBUG] $EXPERIMENTAL_LOG_FILE: nps.caching.log
[DEBUG] $EXPERIMENTAL_CACHE_FILE: nps.experimental.cache
[DEBUG] $EXPERIMENTAL: false

Fetch packages from channels...

[DEBUG] head $CACHE:

[DEBUG] content of /tmp/nps/:
total 4.0K
drwxr-xr-x  3 sandro users  80 Jan 21 17:36 .
drwxrwxrwt 23 root   root  600 Jan 21 17:36 ..
-rw-r--r--  1 sandro users   1 Jan 21 17:36 nps.cache
drwxr-xr-x  2 sandro users  40 Jan 21 17:36 raw_caches

Move cache to /home/sandro/.nix-package-search
Done

nix run github:OleMussmann/Nix-Package-Search/debug -- -e true -r rustup

 ➜ nix run github:OleMussmann/Nix-Package-Search/debug -- -e true -r rustup

[DEBUG] $SEARCH_FOLDER: /home/sandro/.nix-package-search
[DEBUG] $CACHE_FILE: nps.cache
[DEBUG] $EXPERIMENTAL_LOG_FILE: nps.caching.log
[DEBUG] $EXPERIMENTAL_CACHE_FILE: nps.experimental.cache
[DEBUG] $EXPERIMENTAL: true

[DEBUG] filtered $TEST_STRING:
AMB-plugins 0.8.1 A set of ambisonics ladspa plugins
ArchiSteamFarm 5.4.0.3
AusweisApp2 - Authentication software for the German ID card

Fetch packages from nixpkgs...
OleMussmann commented 1 year ago

Man, this one is a tough nut. I still don't know why the second command would fail without any further output. Did you wait for it to finish, or did you kill it with CTRL+C?

I built in a spinner now, so it's clearer when stuff is still happening in the background. nps should now also catch the lack of channels better. If you have the time, could you try these two again?

nix run github:OleMussmann/Nix-Package-Search/283e45a5bcf7a0547fd55026a489482f0ab916eb -- -e false -r rustup && echo $?
nix run github:OleMussmann/Nix-Package-Search/283e45a5bcf7a0547fd55026a489482f0ab916eb -- -e true -r rustup && echo $?
SuperSandro2000 commented 1 year ago

Did you wait for it to finish, or did you kill it with CTRL+C?

I think I waited for it to finish. I rerun it again:

 ➜ nix run github:OleMussmann/Nix-Package-Search/debug -- -e true -r rustup

[DEBUG] $SEARCH_FOLDER: /home/sandro/.nix-package-search
[DEBUG] $CACHE_FILE: nps.cache
[DEBUG] $EXPERIMENTAL_LOG_FILE: nps.caching.log
[DEBUG] $EXPERIMENTAL_CACHE_FILE: nps.experimental.cache
[DEBUG] $EXPERIMENTAL: true

Fetch packages from nixpkgs, this might take a while.
. Number of flakes evaluated: 0

[DEBUG] Package search failed.

[DEBUG] head $SEARCH_FOLDER/$EXPERIMENTAL_LOG_FILE:
error: search term cannot be empty

nix run github:OleMussmann/Nix-Package-Search/283e45a5bcf7a0547fd55026a489482f0ab916eb -- -e false -r rustup && echo $?

 ➜ nix run github:OleMussmann/Nix-Package-Search/283e45a5bcf7a0547fd55026a489482f0ab916eb -- -e false -r rustup && echo $?

[DEBUG] $SEARCH_FOLDER: /home/sandro/.nix-package-search
[DEBUG] $CACHE_FILE: nps.cache
[DEBUG] $EXPERIMENTAL_LOG_FILE: nps.caching.log
[DEBUG] $EXPERIMENTAL_CACHE_FILE: nps.experimental.cache
[DEBUG] $EXPERIMENTAL: false

  Fetch packages from channels, this might take a while... Done.

[DEBUG] head $CACHE:

[DEBUG] content of /tmp/nps/:
total 4.0K
drwxr-xr-x  3 sandro users  80 Feb 18 16:44 .
drwxrwxrwt 26 root   root  720 Feb 18 16:44 ..
-rw-r--r--  1 sandro users   1 Feb 18 16:44 nps.cache
drwxr-xr-x  2 sandro users  40 Feb 18 16:44 raw_caches

[DEBUG] cache file is too empty
[DEBUG] wc -c: 1
[DEBUG] wc -l: 1

Channel error. Please use the experimental flag `--experimental true` to
query registry flakes instead. See `nps -l` for extended help on how to set
flags permanently.

nix run github:OleMussmann/Nix-Package-Search/283e45a5bcf7a0547fd55026a489482f0ab916eb -- -e true -r rustup && echo $?

 ➜ nix run github:OleMussmann/Nix-Package-Search/283e45a5bcf7a0547fd55026a489482f0ab916eb -- -e true -r rustup && echo $?

[DEBUG] $SEARCH_FOLDER: /home/sandro/.nix-package-search
[DEBUG] $CACHE_FILE: nps.cache
[DEBUG] $EXPERIMENTAL_LOG_FILE: nps.caching.log
[DEBUG] $EXPERIMENTAL_CACHE_FILE: nps.experimental.cache
[DEBUG] $EXPERIMENTAL: true

Fetch packages from nixpkgs, this might take a while.
. Number of flakes evaluated: 0

[DEBUG] Package search failed.

[DEBUG] head $SEARCH_FOLDER/$EXPERIMENTAL_LOG_FILE:
error: search term cannot be empty

Also you should not cache things in /tmp/nps since it is publicly writable. Either use mktemp -d to get a random directory or write it into $XDG_RUNTIME_DIRECTORY.