NixOS support

[kpcyrd]

It would be cool to import data from github.com/NixOS/nixpkgs.

I'm not sure what's the best way to get the relevant data. I know i can do

nix-instantiate --eval --json --strict -E 'with (import <nixpkgs> {}); {name = zlib.pname; version = zlib.version; hash = zlib.src.outputHash; urls = zlib.src.urls; }'

To get something like

{"hash":"sha256-/wukwpIBPbwnUws6geH5qBPNOd4Byl4Pi/NVcC76WT4=","name":"zlib","urls":["https://github.com/madler/zlib/releases/download/v1.3/zlib-1.3.tar.gz"],"version":"1.3"}

But I'm not sure if that's the proper way of doing it, and how to do this for all of nixpkgs.

[winterqt]

The main issue with this approach is that the hash won't be equal to what everyone else is using, as more often than not, we unpack the tarball and hash the contents. 😕

[drupol]

Hi!

Why would this be a problem with this approach (we unpack the tarball and hash the contents), this is still a hash, no?

[winterqt]

The main issue with this approach is that the hash won't be equal to what everyone else is using

Unless that's not really an issue for the project, since it may be able to confirm that at least the contents' hashes match (and then calculate/compare the NAR hash based on that).

[drupol]

Yes that make sense, but I don't see an issue there, as long as it is explained somewhere on the website?

[kpcyrd]

See my response on the reproducible-builds email list, assuming the .tar -> .nar conversion is deterministic I can register the .nar hash as an alternative representation, the same way I consider a .tar.gz an alternative representation of .tar.

[winterqt]

.tar -> .nar is deterministic, by way of unpacking it, then calculating the NAR hash on the directory. (Though IIRC the directory's name might also have something to do with it?)

[drupol]

Would it be possible to mention multiple hashes pointing to the same src or it must be a one-to-one mapping only?

[kpcyrd]

As far as I know there can be multiple .tar files that get normalized into the same .nar, the database table does not enforce constraints on this, so this state is possible in the database:

• nar (sha256:abc...xyz) -> tar 1 (sha256:1234...)
• nar (sha256:abc...xyz) -> tar 2 (sha256:5678...)

When opening /artifact/sha256:abc...xyz the webserver is going to render the content of either tar 1 or 2.

I think this is fine though, the database is not open submission and only documents upstream -> downstream interactions of selected distributions (although the sql query should be checked to be deterministic in this case). Since the database is able to handle this I can check every once in a while if this has ever happened (and investigate wtf is going on if the distros disagree about source code xattr flags or something).

[lilyinstarlight]

.tar -> .nar is deterministic, by way of unpacking it, then calculating the NAR hash on the directory.

(To be clear, the unpacking is mostly in nixpkgs fetchers (and some nix builtin fetchers), not anything inherent to nix or nar format)

(Though IIRC the directory's name might also have something to do with it?)

(The nar's storepath is not included in the nar serialization and so should not affect the recursive outputHash (which is just a narHash). Changing the name does change storepath hash part though, because that is determined by an opaque "type", outputHash, nix's configured storeDir, and storepath name. The storepath (including hash part) just shouldn't matter here)

[kpcyrd]

I've added quite a few distros now, but I still haven't figured out how to parse nixpkgs data.

The whatsrc database has a concept of db::Ref with the following fields:

#[derive(sqlx::FromRow, Debug, Serialize)]
pub struct Ref {
    pub chksum: String,
    pub vendor: String,
    pub package: String,
    pub version: String,
    pub filename: Option<String>,
}

Given this Nix package:

{ lib, stdenv, fetchurl, alsa-lib, fixDarwinDylibNames }:

stdenv.mkDerivation rec {
  pname = "sndio";
  version = "1.9.0";

  src = fetchurl {
    url = "https://www.sndio.org/sndio-${version}.tar.gz";
    sha256 = "sha256-8wgm/JwH42nTkk1fzt9qClPA30rh9atQ/pzygFQPaZo=";
  };

  nativeBuildInputs = lib.optional stdenv.hostPlatform.isDarwin fixDarwinDylibNames;
  buildInputs = lib.optional stdenv.hostPlatform.isLinux alsa-lib;
  configurePlatforms = [];

  postInstall = ''
    install -Dm644 contrib/sndiod.service $out/lib/systemd/system/sndiod.service
  '';

  enableParallelBuilding = true;

  meta = with lib; {
    homepage = "https://www.sndio.org";
    description = "Small audio and MIDI framework part of the OpenBSD project";
    license = licenses.isc;
    maintainers = with maintainers; [ Madouura ];
    platforms = platforms.all;
  };
}

I want to generate this data:

[
    {
        "chksum": "sha256:f30826fc9c07e369d3924d5fcedf6a0a53c0df4ae1f5ab50fe9cf280540f699a",
        "vendor": "nixpkgs",
        "package": "sndio",
        "version": "1.9.0",
        "filename": "https://www.sndio.org/sndio-1.9.0.tar.gz"
    }
]

The base64 sha256-8wgm/JwH42nTkk1fzt9qClPA30rh9atQ/pzygFQPaZo= is simply converted to hex, in this case the page already exists: sha256:f30826fc9c07e369d3924d5fcedf6a0a53c0df4ae1f5ab50fe9cf280540f699a.

---

Given this Nix package:

{ lib
, fetchFromGitHub
, libpcap
, libseccomp
, pkg-config
, rustPlatform
, stdenv
}:

rustPlatform.buildRustPackage rec {
  pname = "sniffglue";
  version = "0.16.0";

  src = fetchFromGitHub {
    owner = "kpcyrd";
    repo = pname;
    rev = "v${version}";
    hash = "sha256-MOw0WBdpo6dYXsjbUrqoIJl/sjQ4wSAcm4dPxDgTYgY=";
  };

  cargoHash = "sha256-vnfviiXJ4L/j5M3N+LegOIvLuD6vYJB1QeBgZJVfDnI=";

  nativeBuildInputs = [ pkg-config ];

  buildInputs = [
    libpcap
  ] ++ lib.optionals stdenv.isLinux [
    libseccomp
  ];

  meta = with lib; {
    description = "Secure multithreaded packet sniffer";
    homepage = "https://github.com/kpcyrd/sniffglue";
    license = licenses.gpl3Plus;
    maintainers = with maintainers; [ xrelkd ];
    platforms = platforms.linux ++ platforms.darwin;
    mainProgram = "sniffglue";
  };
}

I want to generate the following data:

[
    {
        "chksum": "sha256:30ec34581769a3a7585ec8db52baa820997fb23438c1201c9b874fc438136206",
        "vendor": "nixpkgs",
        "package": "sniffglue",
        "version": "0.16.0",
        "filename": null
    }
]

I also want to generate a "task" struct that looks like this:

[
    {
        "nixpkgs-fetch-from-github" : {
            "owner": "kpcyrd",
            "repo": "sniffglue",
            "rev": "v0.16.0",
            "hash": "sha256-MOw0WBdpo6dYXsjbUrqoIJl/sjQ4wSAcm4dPxDgTYgY="
        }
    }
]

I want to be able to do this on a nixpkgs git checkout.

[kpcyrd]

I've discovered this gives me a list of all packages:

nix-instantiate --json --eval -E 'with import <nixpkgs> {}; lib.attrNames pkgs' | jq .

But I also discovered trying to query data from them fails with this error:

$ nix-instantiate --json --eval -E 'with import <nixpkgs> {}; lib.mapAttrs (name: pkg: { inherit name; version = pkg.version or "unknown"; }) pkgs'
error:
       … while evaluating attribute 'AAAAAASomeThingsFailToEvaluate'

         at «none»:0: (source not available)

       … message for the trace

         at «none»:0: (source not available)

       error: cannot convert a function application to JSON

       at «none»:0: (source not available)

I'm not sure if nixpkgs data is supposed to be queried like this (accessing all data instead of just those I want to build my system out of), however search.nixos.org seems to be able to as well.

I rewrote the original query to this (I'm still very new to Nix language):

nix-instantiate --eval --json --strict -E 'with import <nixpkgs> {}; { name = pkgs.zlib.name; version = pkgs.zlib.version; urls = pkgs.zlib.src.urls;}'

I discovered I can get a list of all attributes like this:

$ nix-instantiate --eval --json --strict -E 'with import <nixpkgs> {}; lib.attrNames pkgs.zlib' | jq .
[
  "NIX_CFLAGS_COMPILE",
  "__ignoreNulls",
  "__structuredAttrs",
  "all",
  "args",
  "buildInputs",
  "builder",
  "cmakeFlags",
  "configureFlags",
  "configurePlatforms",
  "depsBuildBuild",
  "depsBuildBuildPropagated",
  "depsBuildTarget",
  "depsBuildTargetPropagated",
  "depsHostHost",
  "depsHostHostPropagated",
  "depsTargetTarget",
  "depsTargetTargetPropagated",
  "dev",
  "doCheck",
  "doInstallCheck",
  "dontAddStaticConfigureFlags",
  "dontConfigure",
  "dontDisableStatic",
  "dontStrip",
  "drvAttrs",
  "drvPath",
  "enableParallelBuilding",
  "enableParallelChecking",
  "enableParallelInstalling",
  "inputDerivation",
  "installFlags",
  "makeFlags",
  "mesonFlags",
  "meta",
  "name",
  "nativeBuildInputs",
  "out",
  "outPath",
  "outputDoc",
  "outputName",
  "outputs",
  "override",
  "overrideAttrs",
  "overrideDerivation",
  "passthru",
  "patches",
  "pname",
  "postInstall",
  "postPatch",
  "preConfigure",
  "propagatedBuildInputs",
  "propagatedNativeBuildInputs",
  "setOutputFlags",
  "src",
  "static",
  "stdenv",
  "strictDeps",
  "system",
  "tests",
  "type",
  "userHook",
  "version"
]

I tried to access nativeBuildInputs and propagatedBuildInputs but they were both empty, I also discovered this:

$ nix-instantiate --eval --json --strict -E 'with import <nixpkgs> {}; pkgs.zlib.src'
"/nix/store/n22iqgfwr3jr08f9hl4140y3sqmrzf2z-zlib-1.3.1.tar.gz"

But this path does not exist for me.

[raboof]

I tried to access nativeBuildInputs and propagatedBuildInputs but they were both empty, I also discovered this:

$ nix-instantiate --eval --json --strict -E 'with import <nixpkgs> {}; pkgs.zlib.src'
"/nix/store/n22iqgfwr3jr08f9hl4140y3sqmrzf2z-zlib-1.3.1.tar.gz"

But this path does not exist for me.

Right: nix-instantiate will show you the path but not actually build it - you can use nix-build to build it:

nix-build -E 'with import <nixpkgs> {}; pkgs.zlib.src'

Or slightly shorter:

nix-build '<nixpkgs>' -A zlib.src