lorri shell failure: package.yaml: Yaml file not found: /build/package.yaml

codygman commented 4 years ago

Describe the bug

It's worth noting a few things first:

Our default.nix uses developPackage

Our .envrc calls hpack first since lorri doesn't respect shellHook as noted in #390:

if [[ ! -f "smurf.cabal" ]]
then
echo "generating cabal file from package.yaml..."
nix-shell --pure --run "hpack" # so meta
fi
eval "$(lorri direnv)"

We have a shellHook that calls hpack conditionally as well in case someone uses nix-shell --pure directly:

# full default.nix and shell.nix at bottom
...
    shellHook = ''
if [[ ! -f "smurf.cabal" ]]
then
echo "generating cabal file from package.yaml..."
hpack
fi
'';
...

A teammate is getting:

$ nix-shell --pure
$ lorri shell
lorri shell                                                                                                                                                                                                                    [16:29:27]
lorri: building environment......... done
Jun 04 16:29:39.574 ERRO Build failed. No cached environment available.
Build error: Nix process returned exit code 100.
$ "nix-build" "--out-link" "/tmp/.tmptvAWEC/result" "--" "/tmp/.tmp7kSNVh/result"
these derivations will be built:
  /nix/store/8xp0ram46bm570863q6g17rm3bby89a3-lorri-keep-env-hack-ghc-shell-for-smurf-23.drv
building '/nix/store/8xp0ram46bm570863q6g17rm3bby89a3-lorri-keep-env-hack-ghc-shell-for-smurf-23.drv'...
generating cabal file from package.yaml...
package.yaml: Yaml file not found: /build/package.yaml
builder for '/nix/store/8xp0ram46bm570863q6g17rm3bby89a3-lorri-keep-env-hack-ghc-shell-for-smurf-23.drv' failed with exit code 1
error: build of '/nix/store/8xp0ram46bm570863q6g17rm3bby89a3-lorri-keep-env-hack-ghc-shell-for-smurf-23.drv' failed

While lorri-shell works fine for me:

cody@cody-VirtualBox:~/smurf$ lorri shell
lorri: building environment......... done

We should have the exact same os, exact same nix version, and exact same lorri version. Their nix-info is only different by mine in order of output (so probably nix-info is a different version):

Their (lorri shell fails) nix-info

system: "x86_64-linux"
host os: Linux 5.4.0-33-generic, Ubuntu, 20.04 LTS (Focal Fossa)
multi-user?: no
sandbox: yes
version: nix-env (Nix) 2.3.6
channels(cameron): "home-manager, nixpkgs-20.09pre228333.1ae28ebfdb7"
nixpkgs: /home/cameron/.nix-defexpr/channels/nixpkgs

My ((lorri shell works) nix-info

cody@cody-VirtualBox:~/smurf$ nix-shell -p nix-info --run "nix-info -m"

system: "x86_64-linux"
host os: Linux 5.4.0-33-generic, Ubuntu, 20.04 LTS (Focal Fossa)
multi-user?: no
sandbox: yes
version: nix-env (Nix) 2.3.6
channels(cody): "nixpkgs-20.09pre228333.1ae28ebfdb7, home-manager"
nixpkgs: /home/cody/.nix-defexpr/channels/nixpkgs

There is an interesting piece of our default.nix which generates the cabal file using hpack that is likely causing an issue here:

To Reproduce Steps to reproduce the behavior:

Be on ubuntu with nix 2.3.6 installed
...

Expected behavior

Metadata

$ lorri info
<please paste output here>

$ uname -a
<please paste output here>

Additional context

default.nix (using developPackage)

{ compiler ? "ghc883" ,
  pkgs ? import ./nix/nixpkgs.nix {},
}:
let
  inherit (import (builtins.fetchTarball "https://github.com/hercules-ci/gitignore/archive/7415c4f.tar.gz") { }) gitignoreSource;
  hpkgs = pkgs.haskell.packages."${compiler}";
  smurf = hpkgs.developPackage {
    root = pkgs.lib.cleanSourceWith { filter = (path: type:
      ! (builtins.any
        (r: (builtins.match r (builtins.baseNameOf path)) != null)
        [
          "README.md"
          "notes"
          "ops"
          "tf"
          ".ghcid"
          ".dir-locals.el"
          ".semaphore"
        ])
    ); src = gitignoreSource ./.; };
    name = "smurf";
    modifier = drv:
      with pkgs.haskellPackages;
      pkgs.haskell.lib.disableOptimization (pkgs.haskell.lib.overrideCabal drv (attrs: {
        doCoverage = false;
        doCheck = true; # whether to run tests
        enableLibraryProfiling = false;
        enableExecutableProfiling = false;
        doHaddock = false;
        shellHook = ''
if [[ ! -f "smurf.cabal" ]]
then
  echo "generating cabal file from package.yaml..."
  hpack
fi
'';
        buildTools = [ brittany
                       cabal-install
                       ghcid
                       ghcide
                       hlint
                       hpack
                     ];
      }));
  };
in if pkgs.lib.inNixShell
   then smurf
   else
     pkgs.haskell.lib.dontCoverage (
       pkgs.haskell.lib.dontCheck
         (pkgs.haskell.lib.disableLibraryProfiling
           (pkgs.haskell.lib.disableExecutableProfiling
             (pkgs.haskell.lib.disableOptimization smurf))))

shell.nix (just uses developPackage from default.nix):

let
  smurf = import ./default.nix {};
in
  smurf

Profpatsch commented 4 years ago

lorri direnv does run the shellHook, however inside of the nix build, which is probably why you are seeing the build error.

Profpatsch commented 4 years ago

This is another datapoint that we should probably not be running any shellHooks in lorri (cc @grahamc)

codygman commented 4 years ago

This is another datapoint that we should probably not be running any shellHooks in lorri (cc @grahamc)

From a user perspective this doesn't justify not running shellHooks in lorri. I have a shellHook there so that the cabal file is reliably in sync with the package.yaml and this does actually work with nix-shell --pure.

I don't see how I can use lorri with an hpack powered Haskell project if shellHooks aren't run. Perhaps you know of another simpler way to make that work without shellHooks?

If lorri is going the direction of not supporting as much as nix-shell I'd recommend at least a simple list of things not supported by approximate(intuitive guessing mostly) popularity of that feature.

codygman commented 4 years ago

Also, I'm still confused how things just work on NixOS and not on ubuntu+nix. Maybe I had the cabal file, it got copied over, and this path was never exercised on NixOS. I'll try to see if I can reproduce that difference.

Profpatsch commented 4 years ago

From a user perspective this doesn't justify not running shellHooks in lorri. I have a shellHook there so that the cabal file is reliably in sync with the package.yaml and this does actually work with nix-shell --pure.

I’ve been thinking we’d want to have lorriHook or similar, which is separate from the shellHook; we can’t ensure the same environment than shellHooks are run in, which makes them too dangerous to run outside of the nix build for lorri by default. But the user could just say lorriHook = shellHook if they are sure they want to, and we would make sure they get run with lorri direnv and lorri shell.

I don't see how I can use lorri with an hpack powered Haskell project if shellHooks aren't run. Perhaps you know of another simpler way to make that work without shellHooks?

Is your “run hpack when entering the shell” a good solution anyway? This means you need leave/reenter the project whenever you change your hpack file, which seems like something that should be orthogonal. How about adding a Makefile which always runs hpack before building your project (if the package,yaml changed).

On Sat, Jun 6, 2020 at 3:04 PM Cody Goodman notifications@github.com wrote:

This is another datapoint that we should probably not be running any shellHooks in lorri (cc @grahamc https://github.com/grahamc)

From a user perspective this doesn't justify not running shellHooks in lorri. I have a shellHook there so that the cabal file is reliably in sync with the package.yaml and this does actually work with nix-shell --pure.

I don't see how I can use lorri with an hpack powered Haskell project if shellHooks aren't run. Perhaps you know of another simpler way to make that work without shellHooks?

If lorri is going the direction of not supporting as much as nix-shell I'd recommend at least a simple list of things not supported by approximate(guess mostly) popularity of that feature.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/target/lorri/issues/415#issuecomment-640058152, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAYB5ZRG5C3BH37QGFXBWFLRVI5HFANCNFSM4NS7G4VQ .

codygman commented 4 years ago

Is your “run hpack when entering the shell” a good solution anyway? This means you need leave/reenter the project whenever you change your hpack file, which seems like something that should be orthogonal. How about adding a Makefile which always runs hpack before building your project (if the package,yaml changed).

First, thanks for the quick answer and help. I'm trying to figure out how to implement this with my current nix knowledge.

To answer though, I'm not sure. That does sound better in a way, but then from the nix perspective it changes my project from a Haskell project to a make based or shell based project I think.

Right now I use pkgs.haskell.packages.ghc883.developPackage and then ghci within that. Development in the presence of package.yaml across my team as well as I believe others using package.yaml basically accepts "changing package.yaml" means restarting stack repl or hpack && cabal repl.

I'm open personally to using a makefile based project or shell based project which is hpack && cabal build essentially, but that makes the burden of using nix for Haskell projects at least a little higher.

I get the desire for it to be orthogonal, but I don't like that it complicates things from the user perspective when nix is already hard enough to get adoption for.

Profpatsch commented 4 years ago

Usually it’s useful to distinguish between external and internal dependencies.

nix is pretty granular and perfect for setting up the environment. But for internal dependencies a more fine-grained build manager is needed, like make, redo or stack. In the project I’m working on we have a mix of nix (for deployment and external dependencies) and Make/cabal, where cabal handles the Haskell-related stuff and higher-level actions like “build the server” are phony targets in the Makefile. Something like redo could also work. If the project becomes bigger than just a few Haskell modules, something like bazel could replace both make and cabal/stack.

But essentially, make backend would make sure your cabal file is up to date (running hpack if not) and then run stack build or similar.

On Tue, Jun 9, 2020 at 4:44 PM Cody Goodman notifications@github.com wrote:

Is your “run hpack when entering the shell” a good solution anyway? This means you need leave/reenter the project whenever you change your hpack file, which seems like something that should be orthogonal. How about adding a Makefile which always runs hpack before building your project (if the package,yaml changed).

First, thanks for the quick answer and help. I'm trying to figure out how to implement this with my current nix knowledge.

To answer though, I'm not sure. That does sound better in a way, but then from the nix perspective it changes my project from a Haskell project to a make based or shell based project I think.

Right now I use pkgs.haskell.packages.ghc883.developPackage and then ghci within that. Development in the presence of package.yaml across my team as well as I believe others using package.yaml basically accepts "changing package.yaml" means restarting stack repl or hpack && cabal repl .

I'm open personally to using a makefile based project or shell based project which is hpack && cabal build essentially, but that makes the burden of using nix for Haskell projects at least a little higher.

I get the desire for it to be orthogonal, but I don't like that it complicates things from the user perspective when nix is already hard enough to get adoption for.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/target/lorri/issues/415#issuecomment-641344042, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAYB5ZS6AHBGFQIUKTY7EHDRVZDE3ANCNFSM4NS7G4VQ .

codygman commented 4 years ago

I'm confused at how to make this work. Right now I use direnv and direnv-emacs to give me an environment where ghcide is built with the same haskell dependencies as my project.

How do I ensure that when I enter nix-shell, call nix-build, or call ghcide from my nix-shell environment it first calls this makefile?

But essentially, make backend would make sure your cabal file is up to date (running hpack if not) and then run stack build or similar.

I'm not interested in running any of these commands and in the end I'd like to open a file buffer in my haskell project and ghcide (running in my nix-shell) work. My understanding of direnv was it makes "just run make foo after cd'ing into directory" unnecessary and that's the end I've been using it toward.

I don't see how to get to the end goal of "open buffer, correct ghcide in haskell environment used to build your project gets run, you develop" with your suggestions.

Is there an example I could look at to try and make sense of this? Or if I'm not adequately describing this workflow I have with direnv/emacs-direnv that I'm trying to use lorri for I can make a screencast and sample project to clarify.

codygman commented 4 years ago

~Disregard the above... I think you mean just to call that make command in your .envrc~

Edit: No nevermind, I'm just plain confused.

Profpatsch commented 4 years ago

Sounds like that could work, yes. In the end, you always need some triggers that tell your build system to run, so whether it’s a watcher or something else is a matter of setup/taste.

lorri is only concerned with watching nix dependencies, though maybe there is a way to expose our list of files to watch to some more general watcher implementation which triggers this stuff? I haven’t seen such a high-level watcher project yet, but I haven’t looked very hard.

On Tue, Jun 9, 2020 at 9:33 PM Cody Goodman notifications@github.com wrote:

Disregard the above... I think you mean just to call that make command in your .envrc 😆

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/target/lorri/issues/415#issuecomment-641525973, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAYB5ZXKWRL3WYSGC567SX3RV2FBTANCNFSM4NS7G4VQ .

blaggacao commented 4 years ago

@Profpatsch I'd very much appreciate this https://github.com/target/lorri/issues/415#issuecomment-640309487 It would allow me to reopen https://github.com/cachix/pre-commit-hooks.nix/pull/48

Looks like I'm deadlocked by #264 purposefully not being address in #265 during #444 .

target / lorri