Additions and overrides

[paulyoung]

Thanks for putting this together! I'm trying to figure out how to use this and keep some local additions, and wonder if I'm missing something on the best way to go about it.

I tried importing the generated packages.nix and extending the attribute set but it didn't work out because pp2n expects to call packages.nix a certain way so I resorted to just appending things to packages.nix in a way that overwrites attributes.

However, my project still fails to compile since modules from my additional packages weren't found, and even using when using the "set": "local", "source": "" trick I'm told that the packages I need aren't part of the attribute set.

Any suggestions?

[paulyoung]

To clarify; the additional packages appear to be fetched and then written to disk as expected in the .psc-package directory.

[justinwoo]

I've been overriding using Dhall, but admittedly I don't really like that solution either: https://github.com/justinwoo/vidtracker/blob/d9a89277e1e648c0b998cb5d6bc50ad349242258/packages.dhall https://github.com/justinwoo/vidtracker/blob/d9a89277e1e648c0b998cb5d6bc50ad349242258/Makefile

Can you show me some examples of what you're doing?

[justinwoo]

Also it depends what you're actually using. Psc-Package will always use the package set file to figure out the dependencies. pp2n will use packages.nix only.

[paulyoung]

I was already using the regular psc-package and Dhall approach (thanks!) but had problems getting it to work with Nix. I was using a fixed output derivation that worked on macOS but failed on Linux, so figured I'd give psc-package2nix a try.

My current hack is to have an overrides.nix file that I just cat and append to packages.nix using >> so that pp2n picks up my customizations. overrides.nix isn't a valid nix file because of this. I'd prefer to be able to import packages.nix into overrides.nix and override the fields I want, but there appears to be no way to feed this to pp2n.

I also tried using psc-package instead, using jq to modify psc-package.json on the fly first but setting source to "" results in fatal: repository '' does not exist. I was using an additions field in the json as a quick hack to add the packages that aren't in the package set along with their dependencies (think the contents of packages.dhall as a flat array of package names).

PureScript overlay:

{}:

self: super:

{
  psc-package2nix = import (self.fetchFromGitHub {
    owner  = "justinwoo";
    repo   = "psc-package2nix";
    rev    = "4135235e9577bb130e23b33931a2e228a34fcd34";
    sha256 = "087641lyvpwvd51qc3zgfk8nb4n7fhp0cmg4f2f7rvm9k0ha456v";
  }) { inherit (self) pkgs; };

  psc-package-json2nix = { projectPath }: self.stdenv.mkDerivation {
    name = "purescript-package-set-nix";
    src = super.lib.sourceByRegex projectPath [
      "^psc-package.json$"
      "^overrides.nix$"
    ];
    buildInputs = [
      self.pkgs.cacert # For psc-package2nix
      self.psc-package2nix
    ];
    NIX_PATH = "nixpkgs=${(import ../nixpkgs.nix).src}"; # For psc-package2nix
    buildPhase = ''
      psc-package2nix

      # HACK
      echo ' // {' >> packages.nix
      echo '  set = "local";' >> packages.nix
      echo '  source = "";' >> packages.nix
      echo '  inputs = inputs // '$(cat overrides.nix)';' >> packages.nix
      echo '}' >> packages.nix
    '';
    installPhase = ''
      mkdir -p $out
      cp packages.nix $out
    '';
  };

  purescript-packages = { projectPath }: (
    let
      utils = import (self.psc-package2nix.src + "/utils.nix");
      pscPackageJson2Nix = self.psc-package-json2nix { inherit projectPath; };
      packages = import (pscPackageJson2Nix + "/packages.nix") { inherit (self) pkgs; };
      packageDrvs = builtins.attrValues packages.inputs;
    in
      self.stdenv.mkDerivation {
        name = "purescript-install-package-set-nix";
        unpackPhase = "true"; # This derivation has no src
        buildPhase = ''
          ${utils.mkDefaultShellHook packages packageDrvs}
        '';
        installPhase = ''
          mkdir -p $out
          cp -r .psc-package $out
          cp -r ${pscPackageJson2Nix.out}/. $out # For pp2n
        '';
      }
    );

  purescript = (super.haskell.packages.ghc844.override {
    overrides = hself: hsuper: {
      purescript = hself.callCabal2nix "purescript" (
        super.fetchFromGitHub {
          owner  = "purescript";
          repo   = "purescript";
          rev    = "v0.12.3";
          sha256 = "0y18hvinq1n08zg2aa1zyyj9inny89p0n2pjjimm72l3rn90l4b3";
        }) {};
    };
  }).purescript;

  # TODO: purescript-psa

  purescript-output = { doCheck ? true, projectPath }: (
    let
      purescriptPackages = self.purescript-packages { inherit projectPath; };
    in
      self.stdenv.mkDerivation {
        name = "purescript-output";
        src = super.lib.sourceByRegex projectPath [
          "^psc-package.json$"
          "^((src|test)(\/[A-Z]\.+)*)(\.purs)?$"
        ];
        buildInputs = (with self.pkgs; [
          cacert
          git
          jq
          nodejs
          psc-package
        ]) ++ [
          self.psc-package2nix
          self.purescript
        ];
        NIX_PATH = "nixpkgs=${(import ../nixpkgs.nix).src}"; # For pp2n
        buildPhase = ''
          cp -r ${purescriptPackages.out}/. .
          # cat <<< "$(jq ".set = \"local\" | .source = \"\" | .depends = .depends + .additions" < psc-package.json)" > psc-package.json
          # psc-package build
          pp2n build
        '';
        doCheck = doCheck;
        checkPhase = ''
          . ${self.purescript-test}/bin/test.sh
        '';
        installPhase = ''
          cp -r output/. $out
        '';
      }
  );

  purescript-test = self.pkgs.writeTextFile {
    name = "purescript-test";
    text = ''
      psc-package build 'test/**/*.purs'
      NODE_PATH=output node -e "require('Test.Main').main();"
    '';
    executable = true;
    destination = "/bin/test.sh";
  };
}

psc-package.json (including "additions" hack)

{
  "name": "my-project",
  "set": "psc-0.12.3-20190227",
  "source": "https://github.com/purescript/package-sets.git",
  "depends": [
    "affjax",
    "console",
    "effect",
    "node-buffer",
    "prelude",
    "routing",
    "routing-duplex",
    "spec",
    "spec-discovery",
    "variant"
  ],
  "additions": [
    "specular",

    "aff",
    "avar",
    "debug",
    "foreign-object",
    "generics-rep",
    "prelude",
    "random",
    "record",
    "typelevel-prelude",
    "unsafe-reference",

    "uri",

    "arrays",
    "generics-rep",
    "globals",
    "integers",
    "parsing",
    "profunctor-lenses",
    "unfoldable",
    "these"
  ]
}

overrides.nix (currently invalid as a standalone nix file)

{
  specular = pkgs.stdenv.mkDerivation {
    name = "specular";
    version = "v0.2.0";
    src = pkgs.fetchgit {
      url = "https://github.com/restaumatic/purescript-specular.git";
      rev = "v0.2.0";
      sha256 = "0g6cvw2mmpdw7v8sv2plmlp508ca0ncig7sdwxsm0mb5h06vnaas";
    };
    phases = "installPhase";
    installPhase = "ln -s $src $out";
  };

  uri = pkgs.stdenv.mkDerivation {
    name = "uri";
    version = "v6.0.0";
    src = pkgs.fetchgit {
      url = "https://github.com/slamdata/purescript-uri.git";
      rev = "v6.0.0";
      sha256 = "1ws30jn870jgnnjvl5g2n0s9vls0zkfz6162pmvx1qn2i34hiw3j";
    };
    phases = "installPhase";
    installPhase = "ln -s $src $out";
  };
}

Previous/desired overrides.nix (doesn't work with pp2n)

{ packages, pkgs ? import <nixpkgs> {} }:

let
  # TODO
  # mkPackage =

  overrides = {
    specular = pkgs.stdenv.mkDerivation {
      name = "specular";
      version = "v0.2.0";
      src = pkgs.fetchgit {
        url = "https://github.com/restaumatic/purescript-specular.git";
        rev = "v0.2.0";
        sha256 = "0g6cvw2mmpdw7v8sv2plmlp508ca0ncig7sdwxsm0mb5h06vnaas";
      };
      phases = "installPhase";
      installPhase = "ln -s $src $out";
    };

    uri = pkgs.stdenv.mkDerivation {
      name = "uri";
      version = "v6.0.0";
      src = pkgs.fetchgit {
        url = "https://github.com/slamdata/purescript-uri.git";
        rev = "v6.0.0";
        sha256 = "1ws30jn870jgnnjvl5g2n0s9vls0zkfz6162pmvx1qn2i34hiw3j";
      };
      phases = "installPhase";
      installPhase = "ln -s $src $out";
    };
  };
in packages // {
  inputs = packages.inputs // overrides;

  # set = "local";
  # source = "";
}

[paulyoung]

I think I have a better understanding of how the Dhall approach fits in now and I should be able to use Dhall as I was before to deal with overrides first before doing any of the above (as you appear to be doing with vidtracker). I'll try that and report back.

[justinwoo]

I also finally got done with scrapping the Perl code, so you might just want to see how this works and edit it at will https://github.com/justinwoo/psc-package2nix/blob/169847c9a29fbfa298d0453b979acb6da4912e3b/pp2n.hs#L143

[paulyoung]

I got things working locally but ran into this issue on Hydra that I was going to look at today:

building
/nix/store/<hash>-psc-package2nix/bin/psc-package2nix: /nix/store/<hash>-psc-package2nix/bin/.psc-package2nix-wrapped: /usr/bin/env: bad interpreter: No such file or directory

Not sure if the changes you mention above will affect that.

[paulyoung]

I tried updating to use pp2n with the updated info in the README:

let
  pp2n = import (pkgs.fetchFromGitHub {
    owner = "justinwoo";
    repo = "psc-package2nix";
    rev = "cc48ccd64862366a44b4185a79de321f93755782";
    sha256 = "0cvd1v3d376jiwh4rfhlyijxw3j6jp9rkm9hdb7k7sjxqs1dsviv";
  }) { inherit pkgs; };

and ran into this:

[1 of 1] Compiling Main             ( pp2n.hs, pp2n.o )
Linking pp2n ...
clang-5.0: warning: argument unused during compilation: '-nopie' [-Wunused-command-line-argument]
clang-5.0: warning: argument unused during compilation: '-nopie' [-Wunused-command-line-argument]
Undefined symbols for architecture x86_64:
  "_iconv", referenced from:
      _hs_iconv in libHSbase-4.11.1.0.a(iconv.o)
     (maybe you meant: _hs_iconv_close, _base_GHCziIOziEncodingziIconv_iconvEncoding8_closure , _base_GHCziIOziEncodingziIconv_iconvEncoding1_info , _base_GHCziIOziEncodingziIconv_iconvEncoding4_closure , _base_GHCziIOziEncodingziIconv_iconvEncoding15_info , _base_GHCziIOziEncodingziIconv_iconvEncoding4_info , _base_GHCziIOziEncodingziIconv_iconvEncoding6_closure , _hs_iconv_open , _base_GHCziIOziEncodingziIconv_iconvEncoding9_closure , _base_GHCziIOziEncodingziIconv_iconvEncoding12_info , _base_GHCziIOziEncodingziIconv_iconvEncoding13_info , _base_GHCziIOziEncodingziIconv_iconvEncoding6_info , _base_GHCziIOziEncodingziIconv_iconvEncoding12_closure , _base_GHCziIOziEncodingziIconv_iconvEncoding8_info , _base_GHCziIOziEncodingziIconv_iconvEncoding9_info , _base_GHCziIOziEncodingziIconv_iconvEncoding1_closure , _base_GHCziIOziEncodingziIconv_iconvEncoding5_closure , _base_GHCziIOziEncodingziIconv_iconvEncoding11_closure , _base_GHCziIOziEncodingziIconv_iconvEncoding11_info , _base_GHCziIOziEncodingziIconv_iconvEncoding13_closure , _base_GHCziIOziEncodingziIconv_iconvEncoding2_closure , _base_GHCziIOziEncodingziIconv_iconvEncoding3_closure , _base_GHCziIOziEncodingziIconv_iconvEncoding2_info , _base_GHCziIOziEncodingziIconv_iconvEncoding7_info , _base_GHCziIOziEncodingziIconv_iconvEncoding_closure , _base_GHCziIOziEncodingziIconv_iconvEncoding15_closure , _base_GHCziIOziEncodingziIconv_iconvEncoding10_bytes , _base_GHCziIOziEncodingziIconv_iconvEncoding_info , _base_GHCziIOziEncodingziIconv_iconvEncoding14_bytes , _hs_iconv , _base_GHCziIOziEncodingziIconv_iconvEncoding7_closure )
  "_iconv_open", referenced from:
      _hs_iconv_open in libHSbase-4.11.1.0.a(iconv.o)
     (maybe you meant: _hs_iconv_open)
  "_iconv_close", referenced from:
      _hs_iconv_close in libHSbase-4.11.1.0.a(iconv.o)
     (maybe you meant: _hs_iconv_close)
  "_locale_charset", referenced from:
      _localeEncoding in libHSbase-4.11.1.0.a(PrelIOUtils.o)
ld: symbol(s) not found for architecture x86_64
clang-5.0: error: linker command failed with exit code 1 (use -v to see invocation)
`cc' failed in phase `Linker'. (Exit code: 1)

[justinwoo]

The first issue I guess is because coreutils isn't in the PATH of psc-package2nix, which we could fix with wrapProgram PATH, but the second I really don't know. Are you on OSX?

[paulyoung]

Yes, I’m on macOS High Sierra

[justinwoo]

Maybe it needs https://github.com/NixOS/nixpkgs-channels/blob/nixos-unstable/pkgs/misc/vim-plugins/overrides.nix#L54, I really don't know anything about osx nix

[paulyoung]

So, I'm not using the latest version yet but my troubles with the old version were apparently due to trying to run commands that invoke nix-prefetch-git, etc on CI in a nix sandbox.

I'm now just creating packages.nix statically and checking it in to the repository. But still can't run pp2n build on CI because it uses nix-instantiate via getGlobs.

[justinwoo]

How come nix-instantiate doesn't work? Does the nix package not come with it?

[paulyoung]

I'm pretty sure nix commands can't be used inside of nix expressions because they would introduce impurities or non-determinism. @taktoa, is that right?

[taktoa]

The problem is that nix-instantiate attempts to access the nix store, which cannot be done inside of the nix sandbox. The new nix eval command might not suffer from this, but I am not sure.

[justinwoo]

Just a small update, I've updated the pp2n code to save the hash for the package set when generating packages.nix, so you should be able to grab and copy the package set file to the appropriate location for psc-package to use. Furthermore, there's no need to use the pp2n build command, since the globs can be retrieved in pure nix by calling getGlobs directly: https://github.com/justinwoo/psc-package2nix/blob/master/nix/mkCompilePscPackages.nix

[justinwoo]

I forgot about that last link for a long while because I don't actually try to get a build product out of psc-package2nix projects, but hopefully that should do what you need. The overall idea is that you should only need to refer to the nix derivations in ${psc-package2nix.src}/nix and never call pp2n to do anything outside of development.

[paulyoung]

Thanks for following up on this. In the meantime I've been doing:

let
  compile = ''
    purs compile ".psc-package/*/*/*/src/**/*.purs" "src/**/*.purs" "test/**/*.purs"
  '';

[justinwoo]

ah right, this should work too if you can manage to have a clean packages installation too