Add support for the Nix package manager

[coolGi69]

Similar to https://github.com/Macchina-CLI/libmacchina/pull/150 except updated to the latest version of libmacchina. (closes https://github.com/Macchina-CLI/libmacchina/issues/120) Unlike that commit which attempts to estimate the total amount of nix pkgs by counting the files in /nix/store, this uses nix's sqlite database (/nix/var/nix/db/db.sqlite).\ The sqlite library has also been updated (0.31.1 -> 0.36.0) due to it having the ability to read the path as a uri (so that I can tell sqlite that the database is immutable).

This queries the database using SELECT COUNT(path) FROM ValidPaths WHERE ultimate=1 (which is equivalent to nix path-info --all --sigs | grep ultimate | wc -l) which gives a time of around 10ms, while the equivalent nix command gives around 40ms.

This lookup is grabbing a slightly different metric which gives 739 on my system (nix store giving 1306). So I think nix store is including extra things that don't count as packages (as the /nix/store folder stores everything from fetches, sources, man pages, binaries, etc).

(all packages built with cargo build --release, and tested with hyperfine -w 4 -m 500 -N)	Type	Speed
Original Macchina	5.9 ms ± 0.2 ms
SQLite Lookup	14.2 ms ± 1.7 ms

[Gobidev]

This looks very interesting. From my experience, invoking nix-store was significantly slower than every other readout of libmacchina. I am currently using pretty much the same method as in this PR in pfetch-rs, but added an extra option to skip it because it is comparatively very slow. Maybe a recent Nix update made this method faster, I will do some testing myself later.

Just out of curiosity, what is your macchina runtime if you skip Nix package count?

P.S. What are the runtime differences when you compile macchina with optimizations (cargo b --release)?

[Gobidev]

I have done a bit of testing in a NixOS Live ISO environment with about 1100 packages installed, and this method is still very slow, with the package count taking about 200ms. pfetch-rs currently has a total runtime below 10ms in the same environment when skipping the package count, so this is still a big hit to the runtime. This also only gets worse with more packages installed and on older hardware.

So as much as I would like Nix support to be added, this method just is not fast enough for libmacchina in my opinion.

If you are interested, you could try to get the package count in other ways, as I tried in #150, but this probably requires deeper knowledge of the nix source code.

I also had a look at what fastfetch does, they seem to use the command invocation and cache the result for future queries, which is something we could also think about for libmacchina.

[coolGi69]

I think that command is the standard for getting the amount of packages. It's also whats used in fastfetch (which is which is what I compared it to), as well as most stack overflow answers.

(all packages built with cargo build --release, and tested with hyperfine -w 4 -m 500 -N)

Type	Speed
Default	2.2 ms ± 0.4 ms
This pull	116.5 ms ± 22.1 ms
This pull, but removing nix-user (as pr 150 only gives one value)	50.3 ms ± 9.6 ms
This pull, but replacing the method with the one in pr 150	28.9 ms ± 4.8 ms

I think my previous test were wrong (I'm not even sure how I got the original results). These numbers seem to be more realistic.

In reply to your latest comment, as well as the more realistic results, it is probably best to cache the result somewhere. I'll look into it later tomorrow to see if that's the best way to go.

[coolGi69]

(I just changed it to a draft until I find a faster, but still reliable method)

[grtcdr]

Hi, thanks for the contribution. I personally want to see more progress in #150 due to the performance penalty of dishing out to a command-line program.

To respond to @Gobidev on the caching proposal, I believe it shouldn't be libmacchina's responsibility to cache things. I'd rather keep the library as stateless as possible (despite the obvious oxymoron), downstream programs should implement this if they so wish.

[coolGi69]

Quick update:\ Nix stores all its data for stuff in /nix/var/nix/db/db.sqlite (defined in the libstore/local-store)\ In it, it contains several tables, but the one we are interested in is the DerivationOutputs table, and its id column. The quickest way is to just count the amount of items with the id of dist giving an extremely close approximation of the value gotten by the original nix-store command (nix-store giving 1306, and the amount of dists 1033) dist is not the correct thing to use.\ I think I'll just need to look into the code a bit more to see where the extra 300 packages come from so it can be more accurate.

Hopefully this method works and can give good speeds! (well, as good as you'll get with an sqlite database)

[Gobidev]

Nice, that sounds really promising, thank you for putting in the work! I will close #150 in favor of this PR.

[coolGi69]

I think the best sqlite query is SELECT COUNT(path) FROM ValidPaths WHERE ultimate=1 (which is equivalent to nix path-info --all --sigs | grep ultimate | wc -l) which gives a time of around 10ms, while that nix command gives around 40ms.

Here are the results of hyperfine:	Type	Speed
Original Macchina	5.9 ms ± 0.2 ms
SQLite Lookup	14.2 ms ± 1.7 ms

This lookup is grabbing a slightly different metric which gives 739 on my system (nix store giving 1306). So I think nix store is including extra things that don't count as packages (as the /nix/store folder stores everything from fetches, sources, man pages, binaries, etc).

Also, I've updated sqlite from 0.31.1 to 0.36.0.

[grtcdr]

Big thanks for the contribution (TIL about breaking out of loops with labels, thanks for that too!).

[Gobidev]

Thanks again for your work. I didn't have much time to test this, yet, but at least in a NixOS live CD environment, it doesn't really seem to work right now. Your method returns 3 installed packages, while nix-store -q --requisites /run/current-system/sw | wc -l returns 1098 and nix-store -q --requisites ~/.nix-profile | wc -l returns 15.

This lookup is grabbing a slightly different metric which gives 739 on my system (nix store giving 1306). So I think nix store is including extra things that don't count as packages (as the /nix/store folder stores everything from fetches, sources, man pages, binaries, etc).

If the lower count is more accurate than the one returned by nix-store, this is of course fine, though I expect that there will be issues reported because of inconsistent package counts across different fetch tools.

I will check your method on a system with NixOS actually installed and on a different distro with Nix installed to see if that gives different results, maybe there is just something weird about the NixOS live CD.

[coolGi69]

It only lists the main installed packages. I've uploaded a video of me adding cargo (to a Mint system that I freshly installed Nix onto), and despite it having several dependencies, it only recognizes it as a single one.\ The result is the same on NixOS, but I didn't have my system with it installed on hand to record.

https://github.com/Macchina-CLI/libmacchina/assets/57488297/84278267-71f7-4959-bcd2-c88cf803728b

(I haven't tested it on the NixOS live env, but my guess is that it only shows 3 as it is registering basically everything under a single meta package)

[coolGi69]

The other option for an sqlite search is doing SELECT COUNT(DISTINCT path) FROM DerivationOutputs WHERE id="out", but that lists everything and I just prefer the current lookup format.

[Gobidev]

The other option for an sqlite search is doing SELECT COUNT(DISTINCT path) FROM DerivationOutputs WHERE id="out", but that lists everything

How does that count compare to the other methods?

[coolGi69]

Wrote my thought process, as well as some notes. Hopefully you'd agree that the current method in the pr is the best one (well, at least that I could find):

Type	Speed	Packages	Note
Original	2.9 ms ± 0.1 ms	N/A
SELECT COUNT(path) FROM ValidPaths WHERE ultimate=1 (this pr)	5.3 ms ± 0.9 ms	832	Remember, this only includes the packages in the available generations. So stuff from nix-shell -p wont count towards this number (older generations that haven't been deleted are included here, remember this as it'll come up at the end)
SELECT COUNT(DISTINCT path) FROM DerivationOutputs WHERE id="out"	15.1 ms ± 3.0 ms	8521	This includes every output (so that includes generations, patches, and different entries for the tar.gz vs extracted stuff, so on and so on. Check it yourself if you wanna see just how many other things are included)

If I remove the DISTINCT to make the other lookup go faster gets quicker times, but the amount of packages also go way up.	Type	Speed	Packages	Note
SELECT COUNT(path) FROM DerivationOutputs WHERE id="out"	6.4 ms ± 0.3 ms	16333	Despite this being nearly on-par with the current method (maybe the same as its within a small enough error), this also includes duplicates making the number nearly double!

nix-store -q --requisites /run/current-system/sw | wc -l: 1306\ nix-store -q --requisites ~/.nix-profile | wc -l: 1171\ (note, the packages in current system and nix profile will have overlap)

Getting the overlap by doing the following:

nix-store -q --requisites /run/current-system/sw > /tmp/pkgs_system.txt
nix-store -q --requisites ~/.nix-profile > /tmp/pkgs_profile.txt
awk 'FNR==NR {a[$0]++; next} !($0 in a)' /tmp/pkgs_profile.txt /tmp/pkgs_system.txt | wc -l # Gets the count of different lines between the 2 files

(on my system being 651)\ Then subtracting that from the sum gives 1826 on my system. (again, remember that this number includes more than just packages, and only counts it for the current running generation (so nix-shell -p will still be disregarded, as well as older generations))

[Gobidev]

Thanks a lot for the extensive comparison :)

Based on your numbers, your current method seems to be the one to go with. If there was a way to include dependencies in the count, that would be preferable, but that might not be possible with this approach.

I tested your method on a stock NixOS system, and it returns 313 installed packages, which is a believable number. I will test some more scenarios in which Nix might be used tomorrow.

Thanks again for the amount of work you put into this, it is nice to finally see Nix support being added to libmacchina^^

[Gobidev]

I tested this PR on a single- and multi-user Nix installation on Arch Linux and on both it returns the package count without dependencies as expected.

@grtcdr what is your opinion on the dependency count not being included? AFAIK it is included for all other package managers, but the count without it is not inherently wrong.

[grtcdr]

I'd include the dependencies to avoid the inevitable, unwanted reports we'd undoubtedly receive from users. I'm familiar with the basic concepts of Nix, but the intricacies of it escape me, so if the majority agrees that including just the base packages is a better index then I'm all for it.

[coolGi69]

How about something like SELECT COUNT(path) FROM ValidPaths WHERE sigs IS NOT NULL\ It seems to give the correct results with dependacies.

On a fresh install of nix on another distro, it outputs 0. Then once I install cargo (which has 63 dependacies, that it says at the top of the output when its downloading), it outputs 64 as expected.\ I think this is what we've been looking for!

Sorry, not sure how I didnt notice that before. I guess doing my tests on a non-nix system is easier to read as there isn't soo many packages installed.

[coolGi69]

Yeah, this is wayy better!\ Doing the same thing to get the amount of packages on both the current generations (printing the output of the nix-store thingie and getting the amount of non-duplicates) gives 1826. Then with this new method it gives 1870.

This slightly larger number is understandable as the new sqlite lookup includes every generation (which won't add any on my system due to me cleaning old generations recently), as well as packages on your system but not on the current generation (ie from nix-shell)

[Gobidev]

I have tested this new method and found the following:

On an Arch installation with Nix installed in multi-user mode, I get 67 installed packages after installing cargo and pfetch-rs with nix, which looks good.

On an Arch installation with Nix installed in single-user mode, I get 28 installed packages after installing wget and cargo, but it stays at 28 after uninstalling wget.

On a NixOS installation, i get 331 installed packages (it was 314 with the previous method), while nix-store -q --requisites /run/current-system/sw | wc -l alone returns 929.

This method is still not perfect but imo the best we've had so far.

[coolGi69]

Remember to clear your cache ((sudo) nix-collect-garbage -d). Nix never removes stuff unless you tell it to, hence why it is still counting it even if you "uninstall" it.

[grtcdr]

I'm happy with what we've got - thank you so much for the quality work - shall we get this merged?

[coolGi69]

Yeah, sounds good!