gpgme-1.11.1 - does not provide gpgme.h header file

[wilfreddenton]

Issue description

I am working on a Haskell project that uses bindings-gpgme. Compiling the project fails with:

    Process exited with code: ExitFailure 1
    Logs have been written to: /Users/wilfred/Symbiont/symbiont-node/src/capybara/.stack-work/logs/b
indings-gpgme-0.1.7.log                                                                            

    Configuring bindings-gpgme-0.1.7...
    clang: warning: argument unused during compilation: '-nopie' [-Wunused-command-line-argument]
    Cabal-simple_mPHDZzAJ_2.0.1.0_ghc-8.2.2: Missing dependency on a foreign
    library:
    * Missing (or bad) header file: gpgme.h
    * Missing C library: gpgme
    This problem can usually be solved by installing the system package that
    provides this library (you may need the "-dev" version). If the library is
    already installed but in a non-standard location then you can use the flags
    --extra-include-dirs= and --extra-lib-dirs= to specify where it is.
    If the header file does exist, it may contain errors that are caught by the C
    compiler at the preprocessing stage. In this case you can re-run configure
    with the verbosity flag -v3 to see the error messages.

From my limited understanding of Nix, header files are stored in ~/.nix-profile/include and gpgme.h is not there.

Steps to reproduce

1. nix-env -i gpgme
2. ls ~/.nix-profile/include

Technical details

 - system: `"x86_64-darwin"`
 - host os: `Darwin 17.5.0, macOS 10.13.4`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.0.2`
 - channels(wilfred): `"nixos-18.03-18.03.132229.7cbf6ca1c84"`
 - channels(root): `"nixpkgs-18.09pre139198.9d0b6b9dfc9"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

[matthewbauer]

The default gpgme does not include the development headers. Try running from a nix-shell:

nix-shell -p gpgme

[wilfreddenton]

Thanks for the response. I tried nix-shell -p gpgme and then stack build. This resulted in the same error as before. Could you please explain why we'd expect using nix-shell to work in this case?

Is there a simple modification that can be made to the gpgme derivation so that it includes the development headers or would creating an entirely new derivation, perhaps named libgpgme, be a better option?

[matthewbauer]

Ok sorry forgot, that you probably need to do this:

$ nix-shell -p gpgme stack --run 'stack install bindings-gpgme'

Which is working for me at least.

Alternatively, you should be able to call stack with the --nix arg. (Sadly it does not work currently with GHCJS):

$ stack build --nix

[wilfreddenton]

I have actually gone down the stack + nix route. I created a shell.nix file:

{ghc}:

with let
  hostPkgs = import <nixpkgs> {};
  pinnedPkgs = hostPkgs.fetchFromGitHub {
    owner = "NixOS";
    repo = "nixpkgs";
    rev = "8ce802e8c8e99a0a7aed26e4f22725c56dd4cf80";
    sha256 = "15392s8gz6pn4ngh5lnl3xwifrja7vnvw732pm1bk3fil9aaf6g5";
  };
in import pinnedPkgs {};

haskell.lib.buildStackProject {
  inherit ghc;
  name = "env";
  buildInputs = [
    gpgme
    kubectl
    kubernetes-helm
    openssl
    postgresql
  ];
}

and added

nix:
  enable: true
  shell-file: shell.nix

to my stack.yaml file.

This resulted in the same error as above. This is why I've gone back now to just using nix-env to see if I could get it to work in this more simple setting and then try nix-shell later on.

Just now I tried your suggestion nix-shell -p gpgme stack --run 'stack install bindings-gpgme' and this also resulted in the same error.

[matthewbauer]

It must be something weird going on with Stack. You definitely shouldn't have to manually install gpgme. Anyway, bindings-gpgme is working for me:

#! /usr/bin/env nix-shell
#! nix-shell -i runghc -p "haskellPackages.ghcWithPackages (pkgs: with pkgs; [bindings-gpgme])"
import Bindings.Gpgme
import Foreign
import Foreign.C.String
import Foreign.C.Types
main = do
  version <- c'gpgme_check_version nullPtr >>= peekCString
  print version

[wilfreddenton]

The above worked for me. It finished by printing out "1.11.1". It did also say ignoring (possibly broken) abi-depends field for packages; now sure what that means. Any ideas on why this works but doesn't when I just install gpgme by itself? I'm guessing somewhere in the Nix machinery it knows bindings-gpgme depends on gpgme and so it installs it in a specialized way?

At the moment I'd like to avoid having Nix manage all the haskell packages. My goals is to slowly move the organization I work at towards Nix. I think a good first step would be to provide developers with a reproducible development environment that contains all the non-haskell deps.

[wilfreddenton]

I was able to get thestack + nix integration to work with the following:

{ghc}:

with let
  hostPkgs = import <nixpkgs> {};
  pinnedPkgs = hostPkgs.fetchFromGitHub {
    owner = "NixOS";
    repo = "nixpkgs";
    rev = "8ce802e8c8e99a0a7aed26e4f22725c56dd4cf80";
    sha256 = "15392s8gz6pn4ngh5lnl3xwifrja7vnvw732pm1bk3fil9aaf6g5";
  };
in import pinnedPkgs {};

haskell.lib.buildStackProject {
  inherit ghc;
  name = "env";
  buildInputs = [
    autoconf
    automake
    gpgme
    kubectl
    kubernetes-helm
    libtool
    openssl
    postgresql
  ];
  STACK_IN_NIX_EXTRA_ARGS = " --extra-lib-dirs=${gpgme}/lib";
}

Turns out what I needed was STACK_IN_NIX_EXTRA_ARGS = " --extra-lib-dirs=${gpgme}/lib"; which is explained in the stack documentation here.

I still wish there was a way to get this to work without having to rely on a stack feature.

[matthewbauer]

That's very weird! We should be doing this for you in buildStackProject:

https://github.com/NixOS/nixpkgs/blob/c91a8b78b493fc51f9437d93081899491aa97c3e/pkgs/development/haskell-modules/generic-stack-builder.nix#L23-L26

I get this with nix-shell --run 'echo $STACK_IN_NIX_EXTRA_ARGS' shell.nix:

--extra-lib-dirs=/nix/store/x7v4bw0y8zsja14f64v3vjdny8inlb7y-autoconf-2.69/lib --extra-include-dirs=/nix/store/x7v4bw0y8zsja14f64v3vjdny8inlb7y-autoconf-2.69/include --extra-lib-dirs=/nix/store/i2z77da50m5m0k2alba0zlffqfrzwv1p-automake-1.16.1/lib --extra-include-dirs=/nix/store/i2z77da50m5m0k2alba0zlffqfrzwv1p-automake-1.16.1/include --extra-lib-dirs=/nix/store/6ilwnfqar3k2vx9baazzagbfh25xgcfg-gpgme-1.11.1/lib --extra-include-dirs=/nix/store/15jap16aibnk81idwlf37g91mx8akaqv-gpgme-1.11.1-dev/include --extra-lib-dirs=/nix/store/365fnzq2avnanind2kxchv20nvnrsi92-kubectl-1.9.1/lib --extra-include-dirs=/nix/store/365fnzq2avnanind2kxchv20nvnrsi92-kubectl-1.9.1/include --extra-lib-dirs=/nix/store/17l35g9w9vvkfa5h3ypmc882hpvlrz4d-helm-2.8.2/lib --extra-include-dirs=/nix/store/17l35g9w9vvkfa5h3ypmc882hpvlrz4d-helm-2.8.2/include --extra-lib-dirs=/nix/store/7v86ywdid56s5nyjnrpaxix2lfbs6ain-libtool-2.4.6-lib/lib --extra-include-dirs=/nix/store/a4iyj890mzb1n1zd5cyxgyd3b9z3gqrk-libtool-2.4.6/include --extra-lib-dirs=/nix/store/29dslwbdcj62xbip0skfl6hi196jwwii-openssl-1.0.2o/lib --extra-include-dirs=/nix/store/pf16lx06p07zklkdra52p7b5akpd35qj-openssl-1.0.2o-dev/include --extra-lib-dirs=/nix/store/lrm8kakbs2qf9w2681fh78l47crzssd5-postgresql-9.6.8-lib/lib --extra-include-dirs=/nix/store/5yb3inwra58x3x846v7yd1ggzqffv430-postgresql-9.6.8/include

[matthewbauer]

Ok- I might have found the problem. Try this shell.nix,

{ghc}:

with let
  hostPkgs = import <nixpkgs> {};
  pinnedPkgs = hostPkgs.fetchFromGitHub {
    owner = "NixOS";
    repo = "nixpkgs";
    rev = "8ce802e8c8e99a0a7aed26e4f22725c56dd4cf80";
    sha256 = "15392s8gz6pn4ngh5lnl3xwifrja7vnvw732pm1bk3fil9aaf6g5";
  };
in import pinnedPkgs {};

haskell.lib.buildStackProject {
  inherit ghc;
  name = "env";
  buildInputs = [
    autoconf
    automake
    gpgme libgpgerror
    kubectl
    kubernetes-helm
    libtool
    openssl
    postgresql
  ];
}

You need libgpgerror for the gpgme header to work properly.

Update:

It looks like we are not properly propagating inputs to generic-stack-builder.nix. libgpgerror is not being pulled into $STACK_IN_NIX_EXTRA_ARGS despite being necessary for gpgme.h to work (and a propagated input).

/cc @peti @ElvishJerricco @Ericson2314

[Ericson2314]

I've never looked at buildStackProject. it might be screwed up in arbitrary ways.

[matthewbauer]

Is there any kind of tool to pull in the deps that Nixpkgs would with buildInputs? I think findInputs does this but it would be nice to avoid having a script. Something like this:

  STACK_IN_NIX_EXTRA_ARGS =
    concatMap (pkg: ["--extra-lib-dirs=${getLib pkg}/lib"
                     "--extra-include-dirs=${getDev pkg}/include"]) (findAllInputs buildInputs) ++
    extraArgs;

where findAllInputs includes propagated dependencies.

[Ericson2314]

Yeah findInputs mutates all these globals cause bash so it's very inconvenient to call more than once. Maybe try dredging up https://github.com/NixOS/nixpkgs/blob/master/lib/deprecated.nix#L211-L228 and updating it work as findInputs does today :D.

This won't be quite the same as there's a few cases of packages hand-modiying the propagated input files 🤮.

[Ericson2314]

@matthewbauer but a far easier solution is just to loop over pkgsHost* in bash (or add an env hook to let the looping be done for you).

[matthewbauer]

Hmm... I guess we will need to put it in shellHook too for it to work with stack.

The problem with the env hook is that gpgme shouldn't have to know Stack exists at all. It might make more sense to just replace NIX_LDFLAGS & NIX_CFLAGS_COMPILE directly to emulate cc-wrapper behavior (which stack should be calling anyway right?)

Has anyone looked into rewriting all headers to give absolute path names for include calls? That would solve most of these propagated build cases I think & Nix would treat them like it does dynamic libraries. Probably a huge change but could be worth doing & avoids some of the propagated cases where it really just needs the header file (not all though).

[Ericson2314]

Hmm... I guess we will need to put it in shellHook too for it to work with stack.

The problem with the env hook is that gpgme shouldn't have to know Stack exists at all.

Ahh but generic-stack-build can add the env hook, and everything will work out.

[Ericson2314]

Has anyone looked into rewriting all headers to give absolute path names for include calls?

That would be really cool! But CPP sadly makes these things quite hard. Because (CPP) identifiers are ambiguous, one would have to bail out on the first unresolved #include as further name resolution is ambiguous.

[wilfreddenton]

@matthewbauer I added libgpgerror to the buildInputs and removed the STACK_IN_NIX_EXTRA_ARGS line. This worked. Thank you. However, if I use stdenv.mkDerivation in place of haskell.lib.buildStackProject or if I work outside of a nix-shell, installing gpgme and libgpg-error with nix-env -i, my haskell project does not compile and results in the same error that I initially posted. Is this to be expected? If so, can you please explain why?

[matthewbauer]

Awesome!

However, if I use stdenv.mkDerivation in place of haskell.lib.buildStackProject or if I work outside of a nix-shell, installing gpgme and libgpg-error with nix-env -i, my haskell project does not compile and results in the same error that I initially posted. Is this to be expected? If so, can you please explain why?

So when you run nix-env -i it will symlink things into $HOME/.nix-profile. By default, Nix will set some variables like PATH & MANPATH to pick up things in $HOME/.nix-profile, but we don't intend to set up too much besides these "standard" variables. Instead, we prefer to do things inside nix-shell (which I think is also what stack --nix uses) that are closer to how Nixpkgs are built.

Here is maybe what you wanted to do originally (which still works but gets ugly with lots of dependencies):

stack build --extra-lib-dirs=$(nix-build '<nixpkgs>' -A gpgme.out)/lib --extra-include-dirs=$(nix-build '<nixpkgs>' -A gpgme.dev)

Hopefully we can fix the issue with propagation though.

[wilfreddenton]

Thanks for the explanation. So it seems my idea of using nix-shell to build a general purpose development environment doesn't really align with what the tool was intended to be used for?

And just to confirm, the "issue with propagation" relates only to haskell.lib.buildStackProject?

[matthewbauer]

1 file changed, 11 insertions(+), 4 deletions(-)
.../development/haskell-modules/generic-stack-builder.nix | 15 +++++++++++----

modified   pkgs/development/haskell-modules/generic-stack-builder.nix
@@ -20,10 +20,7 @@ in stdenv.mkDerivation (args // {

   STACK_PLATFORM_VARIANT="nix";
   STACK_IN_NIX_SHELL=1;
-  STACK_IN_NIX_EXTRA_ARGS =
-    concatMap (pkg: ["--extra-lib-dirs=${getLib pkg}/lib"
-                     "--extra-include-dirs=${getDev pkg}/include"]) buildInputs ++
-    extraArgs;
+  STACK_IN_NIX_EXTRA_ARGS = extraArgs;

   # XXX: workaround for https://ghc.haskell.org/trac/ghc/ticket/11042.
   LD_LIBRARY_PATH = makeLibraryPath (LD_LIBRARY_PATH ++ buildInputs);
@@ -37,6 +34,16 @@ in stdenv.mkDerivation (args // {

   preferLocalBuild = true;

+  shellHook = ''
+    for pkg in ''${pkgsHostHost[@]} ''${pkgsHostBuild[@]} ''${pkgsHostTarget[@]}
+    do
+      [ -d "$pkg/lib" ] && \
+        export STACK_IN_NIX_EXTRA_ARGS+=" --extra-lib-dirs=$pkg/lib"
+      [ -d "$pkg/include" ] && \
+        export STACK_IN_NIX_EXTRA_ARGS+=" --extra-include-dirs=$pkg/include"
+    done
+  '';
+
   configurePhase = args.configurePhase or ''
     export STACK_ROOT=$NIX_BUILD_TOP/.stack
   '';

@Ericson2314 Is this what you were thinking? I am not able to get the setupHook route to be run correctly in nix-shell so the addEnvHooks doesn't work.