Table Request: NixOS Installed Packages

FritzX6 commented 2 years ago

What is being requested?

Given existing osquery support for NixOS devices, it would be ideal if we could have the same level of visibility into the installed packages on a device that we possess for other Debian and RPM-based Linux distributions.

How can this be accomplished?

Using the nix-env --query command, we can retrieve the packages installed in the current user profile.

Furthermore, this command supports the output of both JSON and XML structured data, which means we should have a fast path towards pushing the data through the dataflatten approach.

https://manpages.ubuntu.com/manpages/impish/man1/nix-env.1.html#operation%20--query

Additional considerations:

Because we are limited to the current user-profile we will likely want to employ a strategy similar to the macOS screenlock table which can instantiate an osquery process as various users.

directionless commented 2 years ago

osquery does not have good NixOS support. People are tinkering with it, but it's a little rough. (Arch is much better).

I wonder if we can do this table in core osquery.

directionless commented 2 years ago

https://www.reddit.com/r/NixOS/comments/fsummx/how_to_list_all_installed_packages_on_nixos/ implies several ways to get system level info, and not just user level

FritzX6 commented 2 years ago

@directionless the suggested methods in that thread look somewhat promising, the most being maybe this one:

# installed via configuration.nix

nixos-option environment.systemPackages | head -2 | tail -1 | sed -e 's/ /\n/g' | cut -d- -f2- | sort | uniq

# + dependencies

nix-store --query --requisites /run/current-system

nix-store --query --requisites /run/current-system | cut -d- -f2- | sort | uniq

# list user packages

nix-env --query

When reading about this yesterday, it sounded to me like in order to get the system packages we would need to run the command as root:

sudo nix-env --query

and that if we ran it without sudo it would only retrieve the packages installed by the "current user"

This stackoverflow answer seems to agree with the commands suggested above: https://stackoverflow.com/a/47957142/12609425

To figure out what is installed on your system, you may run the following commands:

nix-env -q to figure out what is installed in an imperative user environment.

nixos-option environment.systemPackages to query the set of packages that will appear in /run/current-system/sw when your configuration is built and activated. To quote its documentation, "These packages are automatically available to all users."

nix-store -q --requisites /run/current-system ~/.nix-profile the combined closure of dependencies of the current system and your user profile

nix-store -q --references /run/current-system direct dependencies of the current system

See --query section or nix-store --help for more options.

znewman01 commented 1 year ago

You've opened a bit of a can of worms. NixOS doesn't have a coherent notion of installed packages. There's:

/nix/store: every package you've ever built, unless it gets garbage collected
"Profiles": a subset of /nix/store that gets symlinked into your filesystem and put in your path. This includes a "system" profile and default user profiles. But you can also keep extra profiles around and switch between them.
Local development environments (distinct from profiles). Basically, think of these as Python virtualenvs on steroids: when I cd into a project directory, I might load that project's compiler
Also, sometimes you just run things one-off (e.g., nix run nixpkgs#jq will download (if necessary) and run jq even if I don't have it in any of my profiles. So the software is on my machine and I can run it, but it's not really "installed." I may never run it again.

There are two approaches that seem coherent to me:

All packages in the store.

You can get something like this with

$ nix path-info --all --json > /tmp/system.json   # takes ~10s, 81 MB
$ jq -r '.[:2] | map(.path)[]' < /tmp/system.json
/nix/store/0006yk8jxi0nmbz09fq86zl037c1wx9b-automake-1.16.5.tar.xz.drv
/nix/store/0009f92b97nzvx3mn7s9vdjnrb0wj929-extract.r52117.tar.xz.drv
$ nix show-derivation /nix/store/0006yk8jxi0nmbz09fq86zl037c1wx9b-automake-1.16.5.tar.xz.drv
{
  "/nix/store/0006yk8jxi0nmbz09fq86zl037c1wx9b-automake-1.16.5.tar.xz.drv": {
    "outputs": {
      "out": {
        "path": "/nix/store/gmaq49vzfrkvr714y4fhfxv100ijihin-automake-1.16.5.tar.xz",
        "hashAlgo": "sha256",
        "hash": "f01d58cd6d9d77fbdca9eb4bbd5ead1988228fdb73d6f7a201f5f8d6b118b469"
      }
    },
    "inputSrcs": [
      "/nix/store/lphxcbw5wqsjskipaw1fb8lcf6pm6ri6-builder.sh"
    ],
    "inputDrvs": {
      "/nix/store/6pj63b323pn53gpw3l5kdh1rly55aj15-bash-5.1-p16.drv": [
        "out"
      ],
      "/nix/store/8kd1la3xqfzdcb3gsgpp3k98m7g3hw9d-curl-7.84.0.drv": [
        "dev"
      ],
      "/nix/store/g3m3mdgfsix265c945ncaxyyvx4cnx14-mirrors-list.drv": [
        "out"
      ],
      "/nix/store/zq638s1j77mxzc52ql21l9ncl3qsjb2h-stdenv-linux.drv": [
        "out"
      ]
    },
    "system": "x86_64-linux",
    "builder": "/nix/store/1b9p07z77phvv2hf6gm9f28syp39f1ag-bash-5.1-p16/bin/bash",
    "args": [
      "-e",
      "/nix/store/lphxcbw5wqsjskipaw1fb8lcf6pm6ri6-builder.sh"
    ],
    "env": {
      "SSL_CERT_FILE": "/no-cert-file.crt",
      "buildInputs": "",
      "builder": "/nix/store/1b9p07z77phvv2hf6gm9f28syp39f1ag-bash-5.1-p16/bin/bash",
      "cmakeFlags": "",
      "configureFlags": "",
      "curlOpts": "",
      "curlOptsList": "",
      "depsBuildBuild": "",
      "depsBuildBuildPropagated": "",
      "depsBuildTarget": "",
      "depsBuildTargetPropagated": "",
      "depsHostHost": "",
      "depsHostHostPropagated": "",
      "depsTargetTarget": "",
      "depsTargetTargetPropagated": "",
      "doCheck": "",
      "doInstallCheck": "",
      "downloadToTemp": "",
      "executable": "",
      "impureEnvVars": "http_proxy https_proxy ftp_proxy all_proxy no_proxy NIX_CURL_FLAGS NIX_HASHED_MIRRORS NIX_CONNECT_TIMEOUT NIX_MIRRORS_alsa NIX_MIRRORS_apache NIX_MIRRORS_bioc NIX_MIRRORS_bitlbee NIX_MIRRORS_centos NIX_MIRRORS_cpan NIX_MIRRORS_debian NIX_MIRRORS_fedora NIX_MIRRORS_gcc NIX_MIRRORS_gentoo NIX_MIRRORS_gnome NIX_MIRRORS_gnu NIX_MIRRORS_gnupg NIX_MIRRORS_hackage NIX_MIRRORS_hashedMirrors NIX_MIRRORS_ibiblioPubLinux NIX_MIRRORS_imagemagick NIX_MIRRORS_kde NIX_MIRRORS_kernel NIX_MIRRORS_luarocks NIX_MIRRORS_maven NIX_MIRRORS_mozilla NIX_MIRRORS_mysql NIX_MIRRORS_openbsd NIX_MIRRORS_opensuse NIX_MIRRORS_osdn NIX_MIRRORS_postgresql NIX_MIRRORS_pypi NIX_MIRRORS_qt NIX_MIRRORS_roy NIX_MIRRORS_sageupstream NIX_MIRRORS_samba NIX_MIRRORS_savannah NIX_MIRRORS_sourceforge NIX_MIRRORS_steamrt NIX_MIRRORS_tcsh NIX_MIRRORS_testpypi NIX_MIRRORS_ubuntu NIX_MIRRORS_xfce NIX_MIRRORS_xorg",
      "mesonFlags": "",
      "mirrorsFile": "/nix/store/2pm0lfi03anfdvrn5vb2n0jv4jfs7nb6-mirrors-list",
      "name": "automake-1.16.5.tar.xz",
      "nativeBuildInputs": "/nix/store/jkp0ww7d1b62lkb4xc8nwhxx0iga9nqq-curl-7.84.0-dev",
      "nixpkgsVersion": "22.11",
      "out": "/nix/store/gmaq49vzfrkvr714y4fhfxv100ijihin-automake-1.16.5.tar.xz",
      "outputHash": "0sdl32qxdy7m06iggmkkvf7j520rmmgbsjzbm7fgnxwxdp6mh7gh",
      "outputHashAlgo": "sha256",
      "outputHashMode": "flat",
      "outputs": "out",
      "patches": "",
      "postFetch": "",
      "preferHashedMirrors": "1",
      "preferLocalBuild": "1",
      "propagatedBuildInputs": "",
      "propagatedNativeBuildInputs": "",
      "showURLs": "",
      "stdenv": "/nix/store/p93ivxvrf3c2w02la2c6nppmkgdh08y3-stdenv-linux",
      "strictDeps": "",
      "system": "x86_64-linux",
      "urls": "mirror://gnu/automake/automake-1.16.5.tar.xz"
    }
  }
}

What metadata do you need about the packages? Can you show me an ideal input JSON document or similar?

Unfortunately, it's pretty tricky to go back and find the metadata that was available at build time (like package name/version/license). You can make a pretty good guess from the "name" field though.

directionless commented 1 year ago

Hrm... Looking at that, reading a bit, and installing nix on my mac. I see what you mean about it not really making sense.

At some level, we can make tables for whatever makes sense -- If nix doesn't have the same need, then it doesn't. The underlying driver for these sort of things is that administrators usually want to understand (a) what's installed and (b) what might have security updates waiting to be installed.

as for what's installed, I think this probably fits as 2 tables. One that lists everything in the nix store. And one that enumerates the profiles. I suspect the depth of info in those json files isn't needed, I'd start with some filesystem iteration. ls looks like it's about 65% of the way there....

I kinda wish there was an API for some of this? I'm generally not a fan of shelling out and parsing. We do it, but it's least favorite.

znewman01 commented 1 year ago

as for what's installed, I think this probably fits as 2 tables. One that lists everything in the nix store. And one that enumerates the profiles.

That makes sense to me.

I suspect the depth of info in those json files isn't needed, I'd start with some filesystem iteration. ls looks like it's about 65% of the way there....

Which 65%? Name + version? What else would you need?

I kinda wish there was an API for some of this? I'm generally not a fan of shelling out and parsing. We do it, but it's least favorite.

Agreed. Under the hood the nix CLI commands do just parse files. For instance, a .drv is an unusual-but-ASCII format (example; with prettyprinting).

Also, there is a sqlite database which can trace outputs back to their derivations:

$ cp /nix/var/nix/db/db.sqlite /tmp/nix.sqlite
$ sqlite3 /tmp/nix.sqlite
SQLite version 3.39.3 2022-09-05 11:02:23
Enter ".help" for usage hints.
sqlite> .mode column
sqlite> .tables
DerivationOutputs  Refs               ValidPaths

Some examples of the schema here. Not sure what the stability guarantees are, I can ask around if you'd like.

It's not quite as nice as select name, version from installedpackages but you can do it without the CLI.

directionless commented 1 year ago

I suspect the depth of info in those json files isn't needed, I'd start with some filesystem iteration. ls looks like it's about 65% of the way there....

Which 65%? Name + version? What else would you need?

If I imagine a nix_store stable, I see how to get path, name, and sha. Though the last of those may not be needed. But I think it would be interesting to get type as well. Nix stores everything in the store, right? Some are packages, some are derivations, and maybe something else.

The profiles seem easier. Those are look like straight forward symlink collections.

Also, there is a sqlite database which can trace outputs back to their derivations:

osquery can read sqlite databases directly. Just having this as something we could expose seems good.

kolide / launcher