`intero` is broken: `stack` cannot run `intero`

[deepfire]

Issue description

The emacs / intero integration is broken.

Steps to reproduce

deepfire@andromedae:~/src/tmp$ cat > shell.nix
{ nixpkgs ? import <nixpkgs> {}, pkgs ? nixpkgs.pkgs }:
(pkgs.haskellPackages.callPackage
  ({ mkDerivation
   , emacs
   , intero
   , stack
   , nix
   }:
   mkDerivation {
     pname = "foo";
     version = "0.0.1";
     src = ./.;
     isLibrary = false;
     isExecutable = true;
     executableHaskellDepends = [
          ((pkgs.emacsPackagesNgGen pkgs.emacs).emacsWithPackages
       (epkgs: [ epkgs.melpaPackages.intero ]))
      intero
      stack
      nix
     ];
     description = "Foo";
     license = "foo";
   })
  {}).env

deepfire@andromedae:~/src/tmp$ nix-shell --pure
[nix-shell:~/src/tmp]$ emacs --no-init --batch --eval '(load-library "intero")' --eval '(message "intero: %s" (intero-installed-p))'
Loading /nix/store/yl5mbp2k0gh8fh2l4z9xix4i6kfr9r7p-emacs-25.1/share/emacs/site-lisp/site-start.el (source)...
Loading intero...
intero: not-installed

[nix-shell:~/src/tmp]$ stack exec intero
error: Nix database directory ‘/nix/var/nix/db’ is not writable: Permission denied

[nix-shell:~/src/tmp]$ stack --nix exec intero
error: Nix database directory ‘/nix/var/nix/db’ is not writable: Permission denied

Technical details

• System: 17.03.git.09af945 (Gorilla) (unrelated changes atop de99dc50495e0fb75a039586b882d1cd1b51eb0b)
• Nix version: nix-env (Nix) 1.11.4
• Nixpkgs version: "17.03.git.09af945e3fM" (unrelated changes atop de99dc50495e0fb75a039586b882d1cd1b51eb0b)

[deepfire]

Specifying NIX_REMOTE=daemon doesn't appear to help much, although it changes the error:

[nix-shell:~/src/tmp]$ export NIX_REMOTE=daemon

[nix-shell:~/src/tmp]$ stack exec intero
Executable named intero not found on path: ["/home/deepfire/.stack/global-project/.stack-work/install/x86_64-linux-nix/lts-7.9/8.0.1/bin","/home/deepfire/.stack/snapshots/x86_64-linux-nix/lts-7.9/8.0.1/bin","/nix/store/i7vid8fncc8k521ag0akr92mws6v0vs6-ghc-8.0.1/bin","/nix/store/f1ba8h5f8zwpl0ff32f755y8bp297yq5-patchelf-0.9/bin","/nix/store/9xiq017g22sp5aawphkya2wyfiwslnj9-paxctl-0.9/bin","/nix/store/cymbr3a1s7d1zj4073mb85hb0vdrr4la-coreutils-8.26/bin","/nix/store/r2g04csk4flx3v9kf5g0i3h6j7m5wfq1-findutils-4.6.0/bin","/nix/store/4l6izrwsf5kxjacy4382mlvpqhfb5z2b-diffutils-3.5/bin","/nix/store/64xlllqh7gak9kf7g608ck5axd29r98k-gnused-4.2.2/bin","/nix/store/rff8mpdyx6782qr12y65vc05f7n16qhb-gnugrep-2.27/bin","/nix/store/ls5w3k30k9f7vvg46wv43zhy7nvaam7y-gawk-4.1.3/bin","/nix/store/lsbbm5734yyga96xq4sjkgi3gvmqhwh6-gnutar-1.29/bin","/nix/store/gb1z43qn3qm8ybhw8wqbrwvvcmn160hc-gzip-1.8/bin","/nix/store/2m5r1yjck44b7fccqdif89wz30rxhw0q-bzip2-1.0.6.0.1-bin/bin","/nix/store/f0mycwxbzgj50p6958gd9n7mnx9cxk4i-gnumake-4.2.1/bin","/nix/store/hs9zk8bc1w6hc8s8k7hyl5p0sx02l25w-bash-4.3-p46/bin","/nix/store/f2bv1zcjdab3jhqa2p51vdkp6v03ia11-patch-2.7.5/bin","/nix/store/0bwaf47jwaikrgpnklxa2l9p7blq1yzj-xz-5.2.2-bin/bin"]

..i.e. it devolves to #21494

[deepfire]

In short, the current intero situation on NixOS is extremely fiddly:

1. melpaPackages.intero commit must correspond with haskellPackages.intero, otherwise things break
2. recent stack requires nix: enable: true in ~/.stack/config.yaml
3. stack (with nix support enabled) requires nix-shell in $PATH, which is unavailable inside nix-shell --pure environments -- unless nix is specifically added into the shell.nix recipe
4. when stack is run from inside nix-shell --pure, it tries to access /var/nix/db: error: Nix database directory ‘/nix/var/nix/db’ is not writable: Permission denied ..unless it is provided with NIX_REMOTE=daemon
5. ..in which case, basically, #21494

[deepfire]

Also, I wonder if the recipe can/should be packaged into a NixOS test -- it involves fiddly integration across various components..

[deepfire]

..same goes for fresher nixpkgs, 614ae8f66042dd9337d8fbc1375abc2cea700d39, with stack 3.2.0

[bennofs]

You need to let stack manage your haskell packages if you want to use intero. Don't add the intero haskell package to the dependencies, the intero emacs mode should install a suitable version of it into your stack environment automatically.

[bennofs]

For example, the following shell.nix works fine for me:

with import <nixpkgs> {};

let
  interoEmacs = (emacsPackagesNgGen emacs).emacsWithPackages (epkgs: [
    epkgs.melpaPackages.intero
  ]);
in stdenv.mkDerivation {
  name = "shell-env";
  src = null;
  buildInputs = [ interoEmacs stack nix ];

  shellHook = ''
    export NIX_REMOTE=daemon
    export HOME="${builtins.toString ./env-home}"
    mkdir -p $HOME
  '';
}

(Assuming that you have nix: enable: true in your stack.yaml)

[deepfire]

@bennofs, with nixpkgs commit 96e1220813888f8870de6c3f323db4435f38e849 (2016-12-02):

deepfire@andromedae:~/src/tmp$ cat shell.nix
with import <nixpkgs> {};

let
  interoEmacs = (emacsPackagesNgGen emacs).emacsWithPackages (epkgs: [
    epkgs.melpaPackages.intero
  ]);
in stdenv.mkDerivation {
  name = "shell-env";
  src = null;
  buildInputs = [ interoEmacs stack nix ];

  shellHook = ''
    export NIX_REMOTE=daemon
    export HOME="${builtins.toString ./env-home}"
    mkdir -p $HOME
  '';
}

deepfire@andromedae:~/src/tmp$ stack --version | grep -v -
Version 1.3.0 x86_64
Compiled with:

deepfire@andromedae:~/src/tmp$ intero-installed-p
Loading /nix/store/yl5mbp2k0gh8fh2l4z9xix4i6kfr9r7p-emacs-25.1/share/emacs/site-lisp/site-start.el (source)...
Loading intero...
intero: not-installed
deepfire@andromedae:~/src/tmp$ nix-shell --pure

[nix-shell:/home/deepfire/src/tmp]$ ../core/nix/intero-installed-p
Loading /nix/store/ripfwjzy29h45nl8y8bbdlf1aj0kvyxy-emacs-25.1/share/emacs/site-lisp/site-start.el (source)...
Loading intero...
intero: not-installed

[nix-shell:/home/deepfire/src/tmp]$ cat ../core/nix/intero-installed-p
#!/bin/sh

emacs --batch --no-init \
--eval '(load-library "intero")' \
--eval '(message "intero: %s" (intero-installed-p))'

[deepfire]

with nixpkgs 614ae8f66042dd9337d8fbc1375abc2cea700d39 (2016-12-29):

deepfire@andromedae:~/src/tmp$ stack --version | grep -v -
Version 1.3.2 x86_64
Compiled with:
deepfire@andromedae:~/src/tmp$ cat ~/.stack/config.yaml
# This file contains default non-project-specific settings for 'stack', used
# in all projects.  For more information about stack's configuration, see
# http://docs.haskellstack.org/en/stable/yaml_configuration/
#

#{}
nix:
  enable: true
deepfire@andromedae:~/src/tmp$ nix-shell --pure

[nix-shell:/home/deepfire/src/tmp]$ stack --version | grep -v -
Version 1.3.2 x86_64
Compiled with:

[nix-shell:/home/deepfire/src/tmp]$ ../core/nix/intero-installed-p
Loading /nix/store/jvxwflhzy7vv8k8881ilg8wzb9njyds3-emacs-25.1/share/emacs/site-lisp/site-start.el (source)...
Loading intero...
intero: not-installed

[nix-shell:/home/deepfire/src/tmp]$ echo $HOME
/home/deepfire/src/tmp/env-home

[nix-shell:/home/deepfire/src/tmp]$ cat > env-home/.stack/config.yaml
nix:
  enable: true

[nix-shell:/home/deepfire/src/tmp]$ ../core/nix/intero-installed-p
Loading /nix/store/jvxwflhzy7vv8k8881ilg8wzb9njyds3-emacs-25.1/share/emacs/site-lisp/site-start.el (source)...
Loading intero...
intero: not-installed

[deepfire]

I guess (intero-installed-p) isn't really giving it opportunity to exercise the self-auto-install thing.. investigating further..

[bennofs]

Just start emacs and open a Haskell file (of the stack project), and turn on intero-mode. Then everything should just work™.

[deepfire]

@bennofs, I see.. the nix-managed way used to work.. : -/

The stack-managed way has its complications -- now I need to thread some sort of a global $HOME through all my shell.nix files..

[bennofs]

@deepfire why would you need a globally managed home? You'd only install it once per project, no? You can also just leave home as-is and stack will work fine, it'll even cache your packages in ~/.stack

[deepfire]

@bennofs I guess the problems I'm running into are still stack-intero-related, so I hope you don't mind if I ask here.

I've extended your snippet to use haskell.callPackage to establish the haskell package dependencies.

I'm still running into problems -- so the increasingly apparent question is -- should I give up on Nix entirely, and let stack manage state globally for me?

The problems appear during intero boot:

 deepfire@andromedae:~/src/tmp$ cat shell.nix
{ nixpkgs  ? import <nixpkgs> {}
, myemacs  ? (nixpkgs.pkgs.emacsPackagesNgGen nixpkgs.pkgs.emacs).emacsWithPackages (epkgs: [ epkgs.melpaPackages.intero ])
, compiler ? "default"
}:

with nixpkgs;

let

  f = { mkDerivation, base, base-unicode-symbols, containers, lens
      , nix, stack
      }:
      mkDerivation {
        pname = "shell-env";
        src = null;
        libraryHaskellDepends = [
          base base-unicode-symbols containers lens
      nix stack pkgs.git
      myemacs
        ];
    version = "0.0.0";
    license = "none";
    shellHook = ''
      export NIX_REMOTE=daemon
        '';
      };

  haskellPackages = if compiler == "default"
                       then pkgs.haskellPackages
                       else pkgs.haskell.packages.${compiler};

  drv = haskellPackages.callPackage f {};

in

  if pkgs.lib.inNixShell then drv.env else drv

deepfire@andromedae:~/src/tmp$ nix-shell --pure

nix-shell:~/src/tmp]$ cat tmp.cabal
name:                tmp
version:             0.0.1
synopsis:            Tmp.
license:             AGPL-3
license-file:        LICENSE
author:              Kosyrev Serge
maintainer:          _deepfire@feelingofgreen.ru
category:            Database
build-type:          Simple
cabal-version:       >=1.10

executable tmp
  main-is:             Utils.hs
  default-language:    Haskell2010
  ghc-options:         -threaded

  build-depends:       base >=4.8
                     , base-unicode-symbols
                     , containers
                     , lens

[nix-shell:~/src/tmp]$ emacs Utils.hs

..and then intero boot fails like this:

Error: While constructing the build plan, the following exceptions were encountered:

In the dependencies for tmp-0.0.1:
    base-unicode-symbols must match -any, but the stack configuration has no specified version
                         (latest applicable is 0.2.2.4)
    lens must match -any, but the stack configuration has no specified version (latest applicable is 4.15.1)

Recommended action: try adding the following to your extra-deps in /home/deepfire/src/tmp/stack.yaml:
- base-unicode-symbols-0.2.2.4
- lens-4.15.1

You may also want to try the 'stack solver' command

Error: Plan construction failed.

Warning: Build failed, but optimistically launching GHCi anyway
The following GHC options are incompatible with GHCi and have not been passed to it: -threaded
Configuring GHCi with the following packages: tmp
Intero 0.1.20 (GHC 8.0.1)
Type :intro and press enter for an introduction of the standard commands.

<command line>: cannot satisfy -package base-unicode-symbols
    (use -v for more information)

---

This is the buffer where Emacs talks to intero. It's normally hidden,
but a problem occcured.

[deepfire]

You need to let stack manage your haskell packages if you want to use intero.

Ouch. This has only really just started to sink in..

This is scary. So the fine and dandy definitions I make in shell.nix are relevant no longer? Centralised stack-driven package management?

But nix-driven environment creation used to work with not-so-much-older stack/intero (nixpkgs commit 96e1220813888f8870de6c3f323db4435f38e849, December 12 2016)!

And intero was booting up instantly, happily picking up whatever was constructed by nix..

Are we giving up on this?

[bennofs]

Stack has never supported this, if it worked it was a miracle. Stack's Nix Integration is only meant to install non-haskell dependencies via Nix, the Haskell deps are still compiled by stack

[bennofs]

If it worked, it was probably due to GHC_PACKAGE_PATH interactions or something... But stack generally tries to isolate its ghc as much as possible (it tells ghc to ignore system or user package db, so stack's ghc does not see your packages)

Or perhaps it was some interaction with nixpkgs ghc wrapper?

[anderspapitto]

you actually can get away with using nix to manage haskell deps, alongside intero. The trick is to set nix: enable: false and system-ghc: true, and to make sure that your haskellWithPackages includes intero. This way, stack will use the 'globally available' ghc/haskell packages, which are actually provided via nix.

there's more about it here, including an example project skeleton http://anderspapitto.com/posts/2016-11-12-skeleton-ghc-ghcjs-project.html

[bennofs]

@anderspapitto does this cause any conflicts if the versions of the haskell packages that the system-ghc uses do not match exactly the ones that stack would pick if you let it?

[anderspapitto]

yes, but you can avoid that and make everything work by setting resolver: to match the ghc version installed via nix - e.g. https://github.com/anderspapitto/nix-stack-ghcjs-demo/blob/master/stack.yaml#L1

[deepfire]

you actually can get away with using nix to manage haskell deps, alongside intero. The trick is to set nix: enable: false

This sounds like in case of intero "nix support" is actually counterproductive..

[deepfire]

@anderspapitto, citizens celebrated -- the addition of system-ghc: true to ~/.stack/config.yaml was the solution (that is on top of disabled "nix support")! The stack/intero combo from fresh nixpkgs (614ae8f66042dd9337d8fbc1375abc2cea700d39, 2016-12-29) picks up the nix-constructed environment and boots up instantly!

[deepfire]

Suggested action items:

1. document (where?) that the nix-controlled way to use intero is to have:
   • nix: enable: false
   • system-ghc: true
   • add intero into the list of haskell dependencies of the project being developed
2. add a test case, loosely based on an intero-providing shell.nix and the following shell snippet (the idea is that shell.nix ought to be exactly enough to provide a pre-installed, immediately available intero executable for emacs:

   
   #!/bin/sh

emacs --batch --no-init \ --eval '(load-library "intero")' \ --eval '(message "intero: %s" (intero-installed-p))'

[deepfire]

@peti, do you think it would be worth documenting this use case? Somewhere around section 8.5.2.4 in the manual (http://nixos.org/nixpkgs/manual/#how-to-build-a-haskell-project-using-stack) ?

[peti]

I think that having a section about use in Intero would be nice.

[PierreR]

@deepfire @anderspapitto I would be really interested in such a guide.

So far everything works if I use a global env (nix-env -f ./ghc-env.nix -i). It even feels snappier compared to "classic stack.yaml".

Unfortunately this is not really flexible because it requires me to install/erase ghc-env whenever I switch from one project to another (or to use a ghc-env that would work for every project which is not always possible).

Ideally I would like to make it work within a shell with something like nix-shell -p 'import ./ghc-env.nix. Unfortunately as soon as I do that, if stack build or even stack ghci work flawlessly, launching emacs in the shell and trying intero fails (intero won't boot). What should I do (I am using spacemacs) ?

PS: There has been a recent thread on reddit on that very subject and according to @ElvishJerricco the recipe described here fails short as soon as you want a nix sandbox per project.

[deepfire]

@PierreR What I do have working right now:

1. full control over the package set by Nix -- intero/stack install zero packages
2. per-project nix sandbox
3. near-instant intero boots
4. intero finds all packages from the nix sandbox

[deepfire]

@PierreR, one crucial detail is to have pkgs.stack and ghc.intero to be added to build inputs of your derivation.

In contrast, the way you define and use ghc-env.nix:

1. lacks the project-specific GHC context
2. is non-local, and hence does indeed require global installation (and the inevitable re-installation!)

[PierreR]

@deepfire sorry to bug you with this but while trying your suggestion I have had a collision problem when running nix-shell. I don't know how to git rid of it. Here are all the files involved:

http://pastebin.com/jrSD1gKU

Of course I did try to nix-collect-garbage -d but it does not make any difference.

I discover that the couple (trifecta, intero) is the culprit. For some reason with the same pinned nixpkgs, both of them bring 2 different hash for the scientific derivation (same version).

I have tried this with stable and 2 pinned versions of nixpkgs, no luck. The most recent version I have tried is

{
  "url": "https://github.com/NixOS/nixpkgs.git",
  "rev": "30a94deac57c3498e4a9ed6e7bdc8056a60ece04",
  "date": "2017-01-24T15:56:36+01:00",
  "sha256": "1hja7j9qh1g64106asbkxwgrgd3rihl1b2v2lgac3fkpjy27nly3"
}

So I am not sure if it is at all possible to make the recipe works if one of the deps is trifecta.

@peti Any idea ?

[deepfire]

@PierreR, I have since faced the same issue, actually -- and it appears there is a solution: instead of adding pkgs.stack and ghc.intero to the buildInputs of your derivation (in your default.nix), add them via haskell.lib.addBuildTools in your shell.nix:

let
  ghc = ...
in
  (haskell.lib.addBuildTools
  (ghc.callPackage (import ./.) { })
  [ pkgs.cabal-install
    pkgs.stack
    ghc.halive
    ghc.intero
  ]).env

[deepfire]

@PierreR, alternatively, open pkgs/development/haskell-modules/configuration-common.nix, and find the overrides for stack -- in my case they look like this:

  stack = super.stack.overrideScope (self: super: {
    http-client = self.http-client_0_5_5;
    http-client-tls = self.http-client-tls_0_3_3_1;
    http-conduit = self.http-conduit_2_2_3;
    optparse-applicative = dontCheck self.optparse-applicative_0_13_0_0;
    criterion = super.criterion.override { inherit (super) optparse-applicative; };
    aeson = self.aeson_1_0_2_1;
    hpack = self.hpack_0_15_0;
  });

..then add these overrides to your own derivation's haskellPackages overrides.

The issue here is that the overrides due to stack split the package set into two trunks -- the stack-stemming libraries, and the unoverridden libraries. Inevitably, when you merge the package dag into the resultant flat space the conflicts arise..

By globally choosing to follow stack's divergent versions of dependencies we can avoid the problem that accidentally appears when re-merging the trunks.

[deepfire]

For the record, the current setup I have is as follows: default.nix is in canonical form -- { ..deps..}: mkDerivation { ... }, nothing to see there. shell.nix:

{ nixpkgs     ? import <nixpkgs> {}
, pkgs        ? nixpkgs.pkgs, haskell ? pkgs.haskell
, compiler    ? "ghc801"
, ghcOrig     ? pkgs.haskell.packages."${compiler}"
}:
let
  overcabal = pkgs.haskell.lib.overrideCabal;
  hubsrc    =      repo: rev: sha256:       pkgs.fetchgit { url = "https://github.com/" + repo; rev = rev; sha256 = sha256; };
  overc     = old:                    args: overcabal old (oldAttrs: (oldAttrs // args));
  overhub   = old: repo: rev: sha256: args: overc old ({ src = hubsrc repo rev sha256; }       // args);
  overhage  = old: version:   sha256: args: overc old ({ version = version; sha256 = sha256; } // args);

  ghc       = ghcOrig.override (oldArgs: {
    overrides = with haskell.lib; new: old:
    let parent = (oldArgs.overrides or (_: _: {})) new old;
    in with new; parent // {
      halive = overhub   old.halive "lukexi/halive" "2f1c4c4b00a2a046a2df21432456d7dd9c87ea7f" "0if5pdvkkxcyl2ybnvsmavg453l8c7is72lyy0i6c7d3hh3rcgnb" { doCheck = false; };
      reflex = new.callPackage
         ({ stdenv, mkDerivation, base, containers, data-default, dependent-map, dependent-sum
          , exception-transformers, haskell-src-exts, haskell-src-meta, hlint
          , MemoTrie, lens, monad-control, mtl, primitive, prim-uniq, ref-tf, reflection, semigroups, split, syb
          , template-haskell, these, transformers, transformers-compat
          }:
          mkDerivation {
            pname = "reflex";
            version = "0.5.0";
        src = pkgs.fetchFromGitHub {
          owner = "reflex-frp";
          repo = "reflex";
              rev = "d78ba4318c425ca9b942dc387d7c5c7ab2d2e095";
              sha256 = "10sryvwdf88ajkp35yma8llkb38cp63vjr5mq2hba4s2d8yg649q";
            };
            libraryHaskellDepends = [
              base containers data-default dependent-map dependent-sum exception-transformers
              haskell-src-exts haskell-src-meta hlint lens MemoTrie monad-control mtl primitive prim-uniq ref-tf reflection
          semigroups split syb template-haskell these transformers transformers-compat
            ];
            testHaskellDepends = [
              base containers dependent-map MemoTrie mtl ref-tf
            ];
            homepage = "https://github.com/reflex-frp/reflex";
            description = "Higher-order Functional Reactive Programming";
            license = stdenv.lib.licenses.bsd3;
            hydraPlatforms = stdenv.lib.platforms.none;
          }) {};
    };
  });
in

(haskell.lib.addBuildTools
  (ghc.callPackage (import ./.) { })
  [ pkgs.cabal-install
    pkgs.stack
    ghc.halive
    ghc.intero
  ]).env

stack.yaml:

nix:
  enable: false
system-ghc: true
resolver: ghc-8.0.1
packages:
- '.'

[3noch]

Related: https://github.com/commercialhaskell/intero/issues/371

[oscarvarto]

Hitting this on September 2018 :(

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[cdepillabout]

As far as I know, most people have switched to HLS. There hasn't been comments on here in a few years, so I'll go ahead and close this.