search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
18.13k stars 14.16k forks source link

compilation with stack fails because argument list to hsc2hs is too long #49206

Open cdepillabout opened 6 years ago

cdepillabout commented 6 years ago

Issue description

I am seeing an error when using stack to compile Haskell code on NixOS. It appears that the argument list passed to hsc2hs is extremely long, and hsc2hs calls gcc with that long argument list. gcc fails to deal with an extremely long argument list.

This seems to be another symptom of #41340.

Steps to reproduce

I've created a reproduction with the haskell-gi package. You can try it out with the following steps:

$ git clone git@github.com:cdepillabout/haskell-gi.git
$ git checkout hsc2hs-error
$ stack build --verbose --cabal-verbose --ghc-options '-v3'
...
2018-10-27 12:14:01.626632: [info] Preprocessing library for haskell-gi-0.21.5..
2018-10-27 12:14:01.626660: [info] creating .stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/build/Data/GI/CodeGen
2018-10-27 12:14:01.766178: [info] /nix/store/f31slvv0xydlgf6jhvkyz5m6xchsa54q-ghc-8.4.3/bin/hsc2hs '--cc=/nix/store/10yq7kwlvbc6h658izmrlsspry1g9f3c-gcc-wrapper-7.3.0/bin/cc' '--ld=/nix/store/10yq7kwlvbc6h658izmrlsspry1g9f3c-gcc-wrapper-7.3.0/bin/cc' '--cflag=-fno-stack-protector' '--lflag=-fno-stack-protector' '--cflag=-D__GLASGOW_HASKELL__=804' '--cflag=-Dlinux_BUILD_OS=1' '--cflag=-Dx86_64_BUILD_ARCH=1' '--cflag=-Dlinux_HOST_OS=1' '--cflag=-Dx86_64_HOST_ARCH=1' '--cflag=-I/nix/store/mdfyly2rf93k0xfx9h6dagl4ycclkswg-gobject-introspection-1.56.0-dev/include/gobject-introspection-1.0' '--cflag=-I/nix/store/gbmk2qgalg06yz78fvhlyy87rzpqgjcy-glib-2.56.0-dev/include/glib-2.0' '--cflag=-I/nix/store/hknrb3kdacm9iqwky9bsj0ap45ngbc9z-glib-2.56.0/lib/glib-2.0/include' '--cflag=-I/nix/store/mdfyly2rf93k0xfx9h6dagl4ycclkswg-gobject-introspection-1.56.0-dev/include/gobject-introspection-1.0' '--cflag=-I/nix/store/gbmk2qgalg06yz78fvhlyy87rzpqgjcy-glib-2.56.0-dev/include/glib-2.0' '--cflag=-I/nix/store/hknrb3kdacm9iqwky9bsj0ap45ngbc9z-glib-2.56.0/lib/glib-2.0/include' '--cflag=-I.stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/build/autogen' '--cflag=-I.stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/build/global-autogen' '--cflag=-include' '--cflag=.stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/build/autogen/cabal_macros.h' '--lflag=-L/nix/store/f31slvv0xydlgf6jhvkyz5m6xchsa54q-ghc-8.4.3/lib' '--lflag=-L/nix/store/jg88la9hmpxj34gss9fsvkz9d983mb7x-glibc-locales-2.27/lib' '--lflag=-L/nix/store/sbmbixrjalyq1w679ypjfg90pq81b6ag-gobject-introspection-1.56.0/lib' '--lflag=-L/nix/store/hknrb3kdacm9iqwky9bsj0ap45ngbc9z-glib-2.56.0/lib' '--lflag=-Wl,-R,/nix/store/f31slvv0xydlgf6jhvkyz5m6xchsa54q-ghc-8.4.3/lib' '--lflag=-Wl,-R,/nix/store/jg88la9hmpxj34gss9fsvkz9d983mb7x-glibc-locales-2.27/lib' '--lflag=-Wl,-R,/nix/store/sbmbixrjalyq1w679ypjfg90pq81b6ag-gobject-introspection-1.56.0/lib' '--lflag=-Wl,-R,/nix/store/hknrb3kdacm9iqwky9bsj0ap45ngbc9z-glib-2.56.0/lib' '--lflag=-lgirepository-1.0' '--lflag=-lgobject-2.0' '--lflag=-lglib-2.0' '--cflag=-I/nix/store/0081p4ygwd0i54rzwqdzhbz13fv9vzwq-cups-2.2.6-dev/include' '--cflag=-I/nix/store/0713bi6xyfbh9fkal7h9y2nd8ricfplc-fribidi-1.0.5/include' '--cflag=-I/nix/store/1nha78cyiip6sb288sb7a9jfr7pxwjjf-gtk+3-3.22.30-dev/include' '--cflag=-I/nix/store/1vqby5ck2qzf7374iq30irhwk53z0v2l-libpng-apng-1.6.34-dev/include' '--cflag=-I/nix/store/27x4ncqbj01qas11mq875ym9y10l2w8d-freetype-2.9-dev/include' '--cflag=-I/nix/store/2fiabpwraw8d7qcb6i4aij7f0bpfl37n-libXrandr-1.5.1-dev/include' '--cflag=-I/nix/store/3ribnzxnl0vr6cxzqj8882dc58ip3cyi-renderproto-0.11.1/include' '--cflag=-I/nix/store/3zdsx2a7dama2bhj2dnhrn8c888dg50w-wayland-1.16.0/include' '--cflag=-I/nix/store/473mnkkmm8k6pjy5drzsfkxqyxsigwv2-vte-0.52.2/include' '--cflag=-I/nix/store/4mjzmf7kdhxazy3lvg8b45vj5q6gw1zc-expat-2.2.6-dev/include' ... (very long line with many, many duplicated arguments) ...
2018-10-27 12:14:02.743304: [warn] gcc: error trying to exec '/nix/store/6qmfmricr58bjivgr16y9sjisakdykag-gcc-7.3.0/libexec/gcc/x86_64-unknown-linux-gnu/7.3.0/cc1': execv: Argument list too long
2018-10-27 12:14:02.743809: [warn] compiling .stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/build/Data/GI/CodeGen/GType_hsc_make.c failed (exit code 1)
2018-10-27 12:14:02.986312: [warn] command was: /nix/store/10yq7kwlvbc6h658izmrlsspry1g9f3c-gcc-wrapper-7.3.0/bin/cc -c .stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/build/Data/GI/CodeGen/GType_hsc_make.c -o .stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/build/Data/GI/CodeGen/GType_hsc_make.o -fno-stack-protector -fno-stack-protector -D__GLASGOW_HASKELL__=804 -Dlinux_BUILD_OS=1 -Dx86_64_BUILD_ARCH=1 -Dlinux_HOST_OS=1 -Dx86_64_HOST_ARCH=1 -I/nix/store/mdfyly2rf93k0xfx9h6dagl4ycclkswg-gobject-introspection-1.56.0-dev/include/gobject-introspection-1.0 -I/nix/store/gbmk2qgalg06yz78fvhlyy87rzpqgjcy-glib-2.56.0-dev/include/glib-2.0 -I/nix/store/hknrb3kdacm9iqwky9bsj0ap45ngbc9z-glib-2.56.0/lib/glib-2.0/include -I/nix/store/mdfyly2rf93k0xfx9h6dagl4ycclkswg-gobject-introspection-1.56.0-dev/include/gobject-introspection-1.0 -I/nix/store/gbmk2qgalg06yz78fvhlyy87rzpqgjcy-glib-2.56.0-dev/include/glib-2.0 -I/nix/store/hknrb3kdacm9iqwky9bsj0ap45ngbc9z-glib-2.56.0/lib/glib-2.0/include -I.stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/build/autogen -I.stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/build/global-autogen -include .stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/build/autogen/cabal_macros.h -I/nix/store/0081p4ygwd0i54rzwqdzhbz13fv9vzwq-cups-2.2.6-dev/include -I/nix/store/0713bi6xyfbh9fkal7h9y2nd8ricfplc-fribidi-1.0.5/include -I/nix/store/1nha78cyiip6sb288sb7a9jfr7pxwjjf-gtk+3-3.22.30-dev/include -I/nix/store/1vqby5ck2qzf7374iq30irhwk53z0v2l-libpng-apng-1.6.34-dev/include -I/nix/store/27x4ncqbj01qas11mq875ym9y10l2w8d-freetype-2.9-dev/include -I/nix/store/2fiabpwraw8d7qcb6i4aij7f0bpfl37n-libXrandr-1.5.1-dev/include -I/nix/store/3ribnzxnl0vr6cxzqj8882dc58ip3cyi-renderproto-0.11.1/include -I/nix/store/3zdsx2a7dama2bhj2dnhrn8c888dg50w-wayland-1.16.0/include -I/nix/store/473mnkkmm8k6pjy5drzsfkxqyxsigwv2-vte-0.52.2/include -I/nix/store/4mjzmf7kdhxazy3lvg8b45vj5q6gw1zc-expat-2.2.6-dev/include -I/nix/store/5wa2iz55m5wxihqgyavn3aqlx9xgbkfq-libXft-2.3.2-dev/include -I/nix/store/667iiwrza3vijjan3kdjkim8lbqn0kgw-libXcomposite-0.4.4-dev/include -I/nix/store/7hhl013303zyvw3srgq3bm5674c9i5ka-pango-1.42.4-dev/include -I/nix/store/8hvap2i7mvmk0fbc187rz3gqsxr5pily-nettle-3.4-dev/include -I/nix/store/8mxjfpzh86w64rvx9yp9dz5y9xwx34z2-inputproto-2.3.2/include -I/nix/store/90ba2awivci6a2wzycgd65ib31irmky2-bzip2-1.0.6.0.1-dev/include -I/nix/store/a05xi83cwl4ar1107wnpfsadhzv77561-gsettings-desktop-schemas-3.28.0/include -I/nix/store/admjzx2dh8f3qfx5dlaqgpli0dm8nyb5-libGL-1.0.0-dev/include -I/nix/store/aihspv6bj7c9nypynrvk1jhbkz449mdg-pcre2-10.31-dev/include -I/nix/store/akg4r91sskr24vqyj5vjd1ypjvnf0d6p-kbproto-1.0.7/include -I/nix/store/asj2jl9wmzb8855lwkz35bhaw6nj849i-harfbuzz-1.8.8-dev/include -I/nix/store/avjpsh02a7m53dz98njdb8clwi451izn-libICE-1.0.9-dev/include -I/nix/store/b0xrl3mx0gfvb2pkqlzb9dcnhyjp0saa-xproto-7.0.31/include -I/nix/store/c12vcx8wjlymy0hlnndh5xrrm0andlaz-libX11-1.6.6-dev/include -I/nix/store/ciyl95gffhazl6b578ig2yls5ppck4z3-xcb-util-0.4.0-dev/include -I/nix/store/craky6lkv02qd5gh4ckdlrmbv4jdsvfi-graphite2-1.3.6/include ... (very long line with many, many duplicated arguments) ...
2018-10-27 12:14:02.986672: [debug] Start: getPackageFiles /home/illabout/git/haskell-gi/haskell-gi.cabal
2018-10-27 12:14:03.010702: [debug] Finished in 24ms: getPackageFiles /home/illabout/git/haskell-gi/haskell-gi.cabal
2018-10-27 12:14:03.010912: [debug] Encoding /home/illabout/git/haskell-gi/.stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/stack-build-caches/lib
2018-10-27 12:14:03.011104: [debug] Finished writing /home/illabout/git/haskell-gi/.stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/stack-build-caches/lib

--  While building package haskell-gi-0.21.5 using:
      /home/illabout/.stack/setup-exe-cache/x86_64-linux-nix/Cabal-simple_mPHDZzAJ_2.2.0.1_ghc-8.4.3 --verbose --builddir=.stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1 build lib:haskell-gi --ghc-options " -ddump-hi -ddump-to-file"
    Process exited with code: ExitFailure 1

I thought this might be fixed with https://github.com/haskell/cabal/pull/5356, so I tried backporting it to Cabal-2.2 in ghc-8.4.3, but this did not appear to fix the problem. You can try out my fix like the following, although it recompiles GHC, so it takes a long time:

$ git clone git@github.com:cdepillabout/haskell-gi.git
$ git checkout hsc2hs-error-try-fix-cabal
$ stack build --verbose --cabal-verbose --ghc-options '-v3'
...
(fails with the exact same error as above)

Just like #41340, I guess this is ultimately a Nix problem, but it would be nice if Cabal could be fixed to not send duplicate arguments to hsc2hs. It would also be nice if hsc2hs would be fixed similarly.

Should I open up an issue on the Cabal or hsc2hs repos?

Pinging @nh2, @ElvishJerricco, @Ericson2314, @domenkozar, and @dmjio since they seemed to be active on issue #41340. Also pinging @23Skidoo, @IvanMalison, and @angerman since they were active on https://github.com/haskell/cabal/pull/5356. Finally, pinging @garetxe since he is the author of the haskell-gi library, which I am using to cause the error above (although I don't think haskell-gi is particularly doing anything bad here).

Technical details

cdepillabout commented 6 years ago

Note that this seems to be a problem with either a newer version of ghc, cabal, or hsc2hs.

If I use ghc-8.0.2 to compile haskell-gi, there is no problem. Here is a reproducible example of this working:

$ git clone git@github.com:cdepillabout/haskell-gi.git
$ git checkout hsc2hs-error-ghc-8.0
$ stack build --verbose --cabal-verbose --ghc-options '-v3'
...
(everything succeeds)
...
nh2 commented 6 years ago

Note that this seems to be a problem with either a newer version of ghc, cabal, or hsc2hs.

Can you check if this is just coincidence (e.g. the command lines involved being slightly shorter in that version)?

cdepillabout commented 6 years ago

@nh2 Thanks for the response.

Can you check if this is just coincidence (e.g. the command lines involved being slightly shorter in that version)?

I went back to try to figure this out and... I couldn't reproduce the error from above! Yesterday right after I posted this issue, I just happened to upgrade from 18.03 to 18.09. And now I can no longer reproduce the error.

I guess I'm fine with this, and I'm not interested in investigating further if the solution seems to be just to upgrade to a later release of NixOS. I'll go ahead and close this issue.

Hopefully problem will now be google-able.

cdepillabout commented 6 years ago

Unfortunately I've run into this problem again.

As far as I can tell, it only occurs when building with stack when many other packages have already been built with stack (using different lts and compiler combinations).

I'm going to reopen this since it doesn't seem like it is fixed just by upgrading to a newer version of nixpkgs.

When I have more time I'm going to try to dig more into this to figure out what is going on.

domenkozar commented 6 years ago

What is the package count?

matthewbauer commented 6 years ago

Can you verify that you have the latest version of Nixpkgs in your shell? For instance try this diff on your project:

diff --git a/stack-nix-shell.nix b/stack-nix-shell.nix
index 334b3a1..04cde41 100644
--- a/stack-nix-shell.nix
+++ b/stack-nix-shell.nix
@@ -10,10 +10,8 @@ let
     sha256 = "03n4bacfk1bbx3v0cx8xcgcmz44l0knswzh7hwih9nx0hj3x41yc";
   };

-  nixpkgs =
-    if pkgs == null
-    then import nixpkgsTarball {}
-    else import pkgs {};
+  nixpkgs = import nixpkgsTarball {};
+
 in

 with nixpkgs;

I was pretty sure we fixed this problem with Haskell by 18.09 (so it should only appear in nixpkgs-unstable from like April to July). I think stack uses your builtin channels unless you do something like the above.

cdepillabout commented 5 years ago

I got some free time this weekend so I've tried to come up with a way to reliably reproduce this on NixOS. Here's the method I've come up with. You need stack installed. I am installing stack-1.9.1 from the latest nixpkgs master branch:

  1. Run my update-hledger-monthly script from here:

    https://github.com/cdepillabout/docs/blob/6fb5b2a2ef09a2a64b10fbc0ee72ae969263a73c/code/update-hledger-monthly.hs

    You should be able to run this just with ./update-hledger-monthly.hs.

    This will compile about 60 haskell packages in lts-12.14, but the script itself won't do anything as long as you don't give it any arguments.

  2. Run my XMonad build script:

    https://github.com/cdepillabout/docs/tree/1fe6e0eb5be25789e79bc95ab65c0015c258658e/dot_files/dot_xmonad

    Copy the three above files to the ~/.xmonad/ directory and then run ./build. This should build an XMonad binary. This installs about 60 packages from lts-12.14.

  3. Try to compile Termonad:

    Clone Termonad: git clone git@github.com:cdepillabout/termonad.git. Try to build this with stack build (or stack build --cabal-verbose).

This should fail with the following error when trying to build the haskell-gi package:

haskell-gi-0.21.5: configure
haskell-gi-0.21.5: build

--  While building package haskell-gi-0.21.5 using:
      /home/illabout/.stack/setup-exe-cache/x86_64-linux-nix/Cabal-simple_mPHDZzAJ_2.2.0.1_ghc-8.4.3 --verbose --builddir=.stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1 build --ghc-options " -ddump-hi -ddump-to-file"
    Process exited with code: ExitFailure 1
    Logs have been written to: /home/illabout/git/termonad/.stack-work/logs/haskell-gi-0.21.5.log

    Using Parsec parser
    Configuring haskell-gi-0.21.5...
    Dependency Cabal ==2.2.0.1: using Cabal-2.2.0.1
    Dependency attoparsec ==0.13.2.2: using attoparsec-0.13.2.2
    Dependency base ==4.11.1.0: using base-4.11.1.0
    Dependency bytestring ==0.10.8.2: using bytestring-0.10.8.2
    Dependency containers ==0.5.11.0: using containers-0.5.11.0
    Dependency directory ==1.3.1.5: using directory-1.3.1.5
    Dependency filepath ==1.4.2: using filepath-1.4.2
    Dependency haskell-gi-base ==0.21.4: using haskell-gi-base-0.21.4
    Dependency mtl ==2.2.2: using mtl-2.2.2
    Dependency pretty-show ==1.7: using pretty-show-1.7
    Dependency process ==1.6.3.0: using process-1.6.3.0
    Dependency regex-tdfa ==1.2.3.1: using regex-tdfa-1.2.3.1
    Dependency safe ==0.3.17: using safe-0.3.17
    Dependency text ==1.2.3.1: using text-1.2.3.1
    Dependency transformers ==0.5.5.0: using transformers-0.5.5.0
    Dependency xdg-basedir ==0.2.2: using xdg-basedir-0.2.2
    Dependency xml-conduit ==1.8.0.1: using xml-conduit-1.8.0.1
    Dependency gobject-introspection-1.0 >=1.32: using version 1.56.0
    Dependency gobject-2.0 >=2.32: using version 2.56.0
    Source component graph: component lib
    Configured component graph:
        component haskell-gi-0.21.5-3KJqKCmAlie768dvIWSfgQ
            include base-4.11.1.0
            include haskell-gi-base-0.21.4-8LMX5ZxYtUDBydiBT2uxKL
            include Cabal-2.2.0.1-FxCc0EZHCjFFtbI0fQDg8Q
            include attoparsec-0.13.2.2-4Pinsgh4vTk6NhkexjmpGO
            include containers-0.5.11.0
            include directory-1.3.1.5
            include filepath-1.4.2
            include mtl-2.2.2
            include transformers-0.5.5.0
            include pretty-show-1.7-LW9AeVv4rnVJlF0xEqDucF
            include process-1.6.3.0
            include safe-0.3.17-HEmZRpXGD3s2ywM3rHJ1ur
            include bytestring-0.10.8.2
            include xdg-basedir-0.2.2-I9cOiu1LJdUGTkJ3k3nkuZ
            include xml-conduit-1.8.0.1-3H9S2eI7VYM7qug8IQKJnw
            include regex-tdfa-1.2.3.1-Els0MxxgIzJ4ypVBjgoTSQ
            include text-1.2.3.1-8ApPkOe77J58YHRjtBm7I8
    Linked component graph:
        unit haskell-gi-0.21.5-3KJqKCmAlie768dvIWSfgQ
            include base-4.11.1.0
            include haskell-gi-base-0.21.4-8LMX5ZxYtUDBydiBT2uxKL
            include Cabal-2.2.0.1-FxCc0EZHCjFFtbI0fQDg8Q
            include attoparsec-0.13.2.2-4Pinsgh4vTk6NhkexjmpGO
            include containers-0.5.11.0
            include directory-1.3.1.5
            include filepath-1.4.2
            include mtl-2.2.2
            include transformers-0.5.5.0
            include pretty-show-1.7-LW9AeVv4rnVJlF0xEqDucF
            include process-1.6.3.0
            include safe-0.3.17-HEmZRpXGD3s2ywM3rHJ1ur
            include bytestring-0.10.8.2
            include xdg-basedir-0.2.2-I9cOiu1LJdUGTkJ3k3nkuZ
            include xml-conduit-1.8.0.1-3H9S2eI7VYM7qug8IQKJnw
            include regex-tdfa-1.2.3.1-Els0MxxgIzJ4ypVBjgoTSQ
            include text-1.2.3.1-8ApPkOe77J58YHRjtBm7I8
            Data.GI.CodeGen.API=haskell-gi-0.21.5-3KJqKCmAlie768dvIWSfgQ:Data.GI.CodeGen.API...Data.GI.GIR.XMLUtils=haskell-gi-0.21.5-3KJqKCmAlie768dvIWSfgQ:Data.GI.GIR.XMLUtils
    Ready component graph:
        definite haskell-gi-0.21.5-3KJqKCmAlie768dvIWSfgQ
            depends base-4.11.1.0
            depends haskell-gi-base-0.21.4-8LMX5ZxYtUDBydiBT2uxKL
            depends Cabal-2.2.0.1-FxCc0EZHCjFFtbI0fQDg8Q
            depends attoparsec-0.13.2.2-4Pinsgh4vTk6NhkexjmpGO
            depends containers-0.5.11.0
            depends directory-1.3.1.5
            depends filepath-1.4.2
            depends mtl-2.2.2
            depends transformers-0.5.5.0
            depends pretty-show-1.7-LW9AeVv4rnVJlF0xEqDucF
            depends process-1.6.3.0
            depends safe-0.3.17-HEmZRpXGD3s2ywM3rHJ1ur
            depends bytestring-0.10.8.2
            depends xdg-basedir-0.2.2-I9cOiu1LJdUGTkJ3k3nkuZ
            depends xml-conduit-1.8.0.1-3H9S2eI7VYM7qug8IQKJnw
            depends regex-tdfa-1.2.3.1-Els0MxxgIzJ4ypVBjgoTSQ
            depends text-1.2.3.1-8ApPkOe77J58YHRjtBm7I8
    Using Cabal-2.2.0.1 compiled by ghc-8.4
    Using compiler: ghc-8.4.3
    Using install prefix: /home/illabout/.cabal
    Executables installed in:
    /home/illabout/.stack/snapshots/x86_64-linux-nix/lts-12.13/8.4.3/bin
    Libraries installed in:
    /home/illabout/.stack/snapshots/x86_64-linux-nix/lts-12.13/8.4.3/lib/x86_64-linux-ghc-8.4.3/haskell-gi-0.21.5-3KJqKCmAlie768dvIWSfgQ
    Dynamic Libraries installed in:
    /home/illabout/.stack/snapshots/x86_64-linux-nix/lts-12.13/8.4.3/lib/x86_64-linux-ghc-8.4.3
    Private executables installed in:
    /home/illabout/.stack/snapshots/x86_64-linux-nix/lts-12.13/8.4.3/libexec/x86_64-linux-ghc-8.4.3/haskell-gi-0.21.5
    Data files installed in:
    /home/illabout/.stack/snapshots/x86_64-linux-nix/lts-12.13/8.4.3/share/x86_64-linux-ghc-8.4.3/haskell-gi-0.21.5
    Documentation installed in:
    /home/illabout/.stack/snapshots/x86_64-linux-nix/lts-12.13/8.4.3/doc/haskell-gi-0.21.5
    Configuration files installed in:
    /home/illabout/.stack/snapshots/x86_64-linux-nix/lts-12.13/8.4.3/etc
    No alex found
    Using ar found on system at:
    /nix/store/vcc4svb8gy29g4pam2zja6llkbcwsyiq-binutils-2.30/bin/ar
    No c2hs found
    No cpphs found
    No doctest found
    Using gcc version 7.3.0 found on system at:
    /nix/store/10yq7kwlvbc6h658izmrlsspry1g9f3c-gcc-wrapper-7.3.0/bin/cc
    Using ghc version 8.4.3 given by user at:
    /nix/store/f31slvv0xydlgf6jhvkyz5m6xchsa54q-ghc-8.4.3/bin/ghc
    Using ghc-pkg version 8.4.3 given by user at:
    /nix/store/f31slvv0xydlgf6jhvkyz5m6xchsa54q-ghc-8.4.3/bin/ghc-pkg
    No ghcjs found
    No ghcjs-pkg found
    No greencard found
    Using haddock version 2.20.0 found on system at:
    /nix/store/f31slvv0xydlgf6jhvkyz5m6xchsa54q-ghc-8.4.3/bin/haddock
    Using happy version 1.19.9 found on system at:
    /home/illabout/.stack/snapshots/x86_64-linux-nix/lts-12.13/8.4.3/bin/happy
    Using haskell-suite found on system at: haskell-suite-dummy-location
    Using haskell-suite-pkg found on system at: haskell-suite-pkg-dummy-location
    No hmake found
    Using hpc version 0.67 found on system at:
    /nix/store/f31slvv0xydlgf6jhvkyz5m6xchsa54q-ghc-8.4.3/bin/hpc
    Using hsc2hs version 0.68.3 found on system at:
    /nix/store/f31slvv0xydlgf6jhvkyz5m6xchsa54q-ghc-8.4.3/bin/hsc2hs
    No hscolour found
    No jhc found
    Using ld found on system at:
    /nix/store/an3icm1hajqx1lld6n8q4qr40nwbs544-binutils-wrapper-2.30/bin/ld
    No lhc found
    No lhc-pkg found
    Using pkg-config version 0.29.2 found on system at:
    /nix/store/zkakybl5c3r4rwgc129238ivlbxi6p5a-pkg-config-0.29.2/bin/pkg-config
    Using runghc version 8.4.3 found on system at:
    /nix/store/f31slvv0xydlgf6jhvkyz5m6xchsa54q-ghc-8.4.3/bin/runghc
    Using strip version 2.30 found on system at:
    /nix/store/vcc4svb8gy29g4pam2zja6llkbcwsyiq-binutils-2.30/bin/strip
    Using tar found on system at:
    /nix/store/y9nwn094yhwkg11pdl1w3427lyk9nvg3-gnutar-1.30/bin/tar
    No uhc found
    Component build order: library
    /nix/store/f31slvv0xydlgf6jhvkyz5m6xchsa54q-ghc-8.4.3/bin/ghc-pkg init .stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/package.conf.inplace
    creating .stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/build
    creating .stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/build/autogen
    creating .stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/build/autogen
    Preprocessing library for haskell-gi-0.21.5..
    creating .stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/build/Data/GI/CodeGen
    creating .stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/build/Data/GI
    creating .stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/build/Data
    creating .stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/build/Data/GI
    creating .stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/build/Data/GI/CodeGen
    /nix/store/f31slvv0xydlgf6jhvkyz5m6xchsa54q-ghc-8.4.3/bin/hsc2hs '--cc=/nix/store/10yq7kwlvbc6h658izmrlsspry1g9f3c-gcc-wrapper-7.3.0/bin/cc' ... '--lflag=-ldl' -o .stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/build/Data/GI/CodeGen/GType.hs lib/Data/GI/CodeGen/GType.hsc
    gcc: error trying to exec '/nix/store/6qmfmricr58bjivgr16y9sjisakdykag-gcc-7.3.0/libexec/gcc/x86_64-unknown-linux-gnu/7.3.0/cc1': execv: Argument list too long
    compiling .stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/build/Data/GI/CodeGen/GType_hsc_make.c failed (exit code 1)
    command was: /nix/store/10yq7kwlvbc6h658izmrlsspry1g9f3c-gcc-wrapper-7.3.0/bin/cc -c .stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/build/Data/GI/CodeGen/GType_hsc_make.c -o .stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/build/Data/GI/CodeGen/GType_hsc_make.o -fno-stack-protector ... -I/nix/store/f31slvv0xydlgf6jhvkyz5m6xchsa54q-ghc-8.4.3/lib/ghc-8.4.3/include -I/nix/store/f31slvv0xydlgf6jhvkyz5m6xchsa54q-ghc-8.4.3/lib/ghc-8.4.3/include/

The hsc2hs line (4th to the end) was actually 926,359 characters long, but I've elided the middle. There were 11,509 individual flags. However, there were only 435 unique flags. The rest were repeats.

Here is the full list of unique flags. Nothing looks too strange. The only strange thing is that there were originally tons of repeats:

https://gist.github.com/cdepillabout/0a7d79e5bf856bd8d14ca9ab271bc488

The last line (command was: ...) was actually 192,332 characters long.

cdepillabout commented 5 years ago

I thought that the above (step 3) would only happen after installing many Haskell packages, but I was incorrect. The above still happens when trying to install termonad even without having done steps 1 and 2 first.

It appears the the error from step 3 will happen if all you do is just try to build termonad with stack.

matthewbauer commented 5 years ago

@cdepillabout Could you try this commit:

https://github.com/matthewbauer/nixpkgs/commit/83fef595b1c8ee1b1907ec214aae555a5d4fa60a

This is definitely a generic-stack-builder thing because we aren't getting it in other places.

cdepillabout commented 5 years ago

@matthewbauer Thanks for taking a look at this.

Unfortunately https://github.com/matthewbauer/nixpkgs/commit/83fef595b1c8ee1b1907ec214aae555a5d4fa60a doesn't seem to help. I am still getting the same error as above.

Here is a commit for termonad where I change the nixpkgs pin:

https://github.com/cdepillabout/termonad/commit/c4f14cd0006ca91b547e28530735847c07699613

It is on this branch:

https://github.com/cdepillabout/termonad/tree/try-to-fix-hsc2hs-problem

If you are interested in testing this out yourself, you could clone Termonad, checkout the above branch, and run stack build. It should fail when building the haskell-gi package. (Although it builds about 100 other packages before it gets to haskell-gi.)

I thought this problem might be happening because of the environment variables that are set by nix (and stack?), so here is the output stack exec -- env with the new pinned nixpkgs from @matthewbauer in the previous commit:

https://gist.github.com/cdepillabout/74e7560440997973aa56ad3b7e26d8b1

There are a couple environment variables with relatively long values:

cdepillabout commented 5 years ago

The STACK_IN_NIX_EXTRA_ARGS environment variable looks like it has 178 separate arguments, but only 4 of those are duplicates.

I'm thinking that this is not a problem in the generic-stack-builder, but instead in Cabal. I think Cabal is passing too many arguments to hsc2hs, which then passes too many arguments to cc.

cdepillabout commented 5 years ago

I've tried to update Termonad to a more recent version of nixpkgs (master from around 2018-12-23), but it doesn't look like it has fixed this problem.

Here's a commit that can be used to reproduce this problem:

$ git clone git@github.com:cdepillabout/termonad.git
$ git checkout 5a4e05d1c4
$ stack build
...

This should fail when building haskell-gi with the same errors as above.

I had an idea of trying to work around this with something like buildFHSUserEnv. I'll have to see if I can get something like that working.

cdepillabout commented 5 years ago

I was able to use buildFHSUserEnv to build a chroot environment to run stack. This has enabled me to workaround the error from above and successfully build termonad on NixOS using stack.

Here's the PR where I add the derivation using buildFHSUserEnv:

https://github.com/cdepillabout/termonad/pull/101

matthewbauer commented 5 years ago

@cdepillabout can you try #53618 with your project. It looks to have fixed it by I am not sure if I am reproducing exactly what you describe.

cdepillabout commented 5 years ago

@matthewbauer Unfortunately this doesn't appear to fix it for me.

In https://github.com/NixOS/nixpkgs/pull/53618#issuecomment-452180929 you said:

[#53618] doesn't look like it fixes [#49206]. I'm starting to wonder if there's something going on in either stack/cabal here?

That's what I was thinking as well.

@nh2 sent a PR to cabal that fixed #41340 for most things:

https://github.com/haskell/cabal/pull/5356

However I'm wondering if somewhere else in Cabal also needs to be fixed. I'm wondering if the place where hsc2hs is called from cabal also needs a similar fix to https://github.com/haskell/cabal/pull/5356. I actually tried to look into this myself, but I couldn't really figure out a good workflow for patching cabal, rebuilding GHC, testing with stack, etc.

matthewbauer commented 5 years ago

This is fixed for me on 19.03 with this commit https://github.com/matthewbauer/haskell-gi/commit/ac1134b8ef9b6ba8b6f06436f1975980a09a36b7

Feel free to reopen if this is still an issue.

cdepillabout commented 5 years ago

@matthewbauer

Unfortunately this doesn't seem to be fixed. I'm still seeing this error when trying to build Termonad with stack, even when using 19.03.

This should (hopefully) be reproducible:

$ git clone git@github.com:cdepillabout/termonad.git
$ cd termonad/
$ git checkout f4776be737f60
$ stack build
...
--  While building package haskell-gi-0.21.5 using:
      /home/illabout/.stack/setup-exe-cache/x86_64-linux-nix/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.3 --builddir=.stack-work/dist/x86_64-linux-nix/Cabal-2.4.0.1 build --ghc-options " -ddump-hi -ddump-to-file -fdiagnostics-color=always"
    Process exited with code: ExitFailure 1
    Logs have been written to: /home/illabout/git/termonad/.stack-work/logs/haskell-gi-0.21.5.log

    Configuring haskell-gi-0.21.5...
    Preprocessing library for haskell-gi-0.21.5..
    gcc: error trying to exec '/nix/store/g8wz2x9ai0i3rkl0h09ydz20jfwbgjfk-gcc-7.4.0/libexec/gcc/x86_64-unknown-linux-gnu/7.4.0/cc1': execv: Argument list too long
    compiling .stack-work/dist/x86_64-linux-nix/Cabal-2.4.0.1/build/Data/GI/CodeGen/GType_hsc_make.c failed (exit code 1)
    command was: /nix/store/8zfm4i1aw4c3l5n6ay311ds6l8vd9983-gcc-wrapper-7.4.0/bin/cc -c .stack-work/dist/x86_64-linux-nix/Cabal-2.4.0.1/build/Data/GI/CodeGen/GType_hsc_make.c -o .stack-work/dist/x86_64-linux-nix/Cabal-2.4.0.1/build/Data/GI/CodeGen/GType_hsc_make.o -fno-stack-protector -fno-stack-protector -D__GLASGOW_HASKELL__=806 -Dlinux_BUILD_OS=1 -Dx86_64_BUILD_ARCH=1 -Dlinux_HOST_OS=1 -Dx86_64_HOST_ARCH=1 -I/nix/store/04jjbh9j9bpac3iwzcb55cka79n588hr-libSM-1.2.3-dev/include -I/nix/store/0pxrs17p72cazsjfcpan9p1mca843rn8-libjpeg-turbo-2.0.1-dev/include -I/nix/store/2661i5rjmwzkf6qw3kwjajq5zd7dss16-mesa-noglu-18.3.3-dev/include -I/nix/store/26d9f7qg7cgkyshcas1z916j55h17xk0-fribidi-1.0.5/include ...

The above is trying to build Termonad at commit f4776be73, which changes to use the latest commit for the nixos-19.03 channel as of right now.

It looks like I don't have permissions to reopen this issue though.

matthewbauer commented 5 years ago

Is this on Linux or macOS?

Have you been able to file this with stack? I think there is something weird going on with how they handle the --extra-lib-dirs and --extra-include-dirs flag. It looks like they are somehow duplicating them for each dependency so you get tons and tons of duplicates. Right now Nixpkgs should only be setting the flag once for each place. At the very least, the fact we can build haskell-gi on Cabal+Nix makes me think Stack is accumulating these variables weirdly.

cdepillabout commented 5 years ago

@matthewbauer The error from https://github.com/NixOS/nixpkgs/issues/49206#issuecomment-468883938 was on Linux.

I just tried compiling on MacOS with stack and I got a slightly different error (although it may be caused from the same underlying reason):

$ stack build --nix
...
--  While building package haskell-gi-0.21.5 using:
      /Users/illabout/.stack/setup-exe-cache/x86_64-osx-nix/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.3 --builddir=.stack-work/dist/x86_64-osx-nix/Cabal-2.4.0.1 build --ghc-options " -ddump-hi -ddump-to-file -fdiagnostics-color=always"
    Process exited with code: ExitFailure 1
    Logs have been written to: /Users/illabout/git/termonad/.stack-work/logs/haskell-gi-0.21.5.log

    Configuring haskell-gi-0.21.5...
    Preprocessing library for haskell-gi-0.21.5..
    hsc2hs: /nix/store/wdjf5qf597b5pcxfw031zxm5rk9w4p1y-clang-wrapper-5.0.2/bin/cc: rawSystem: runInteractiveProcess: exec: resource exhausted (Argument list too long)

Have you been able to file this with stack? I think there is something weird going on with how they handle the --extra-lib-dirs and --extra-include-dirs flag. It looks like they are somehow duplicating them for each dependency so you get tons and tons of duplicates.

I actually hadn't thought about stack being to blame, but now that you mention it, it does seem like it could be possible. I'll look into this a little bit from the stack side and see what I can find out.

matthewbauer commented 5 years ago

Yeah, so each include directory is listed 38 times each. I believe what's happening is that stack passes the --extra-lib-dirs and --extra-include-dirs to Cabal and then Cabal saves them permanently to the package conf file for the built package. Every time it gets depended on by another package, it pulls in those extra dirs. I'm not sure how to fix this though. More and more this seems like a Cabal issue (which I thought we had fixed the duplication issue in).

nh2 commented 5 years ago

More and more this seems like a Cabal issue (which I thought we had fixed the duplication issue in).

I believed I had fixed at least one instance of this in Cabal with https://github.com/haskell/cabal/pull/5356/files, but maybe there is more.

cdepillabout commented 5 years ago

@nh2 I haven't looked into this issue deeply on the Cabal-side, but it appears to be occurring when cabal calls hsc2hs (and not when compiling Haskell modules normally).

My guess is that https://github.com/haskell/cabal/pull/5356 fixed the problem when compiling Haskell modules, but there is still a lingering problem when calling external programs (like hsc2hs).

cdepillabout commented 5 years ago

This issue might be related to https://github.com/haskell/hsc2hs/issues/22, as is posted by @gbaz.

gbaz commented 5 years ago

building with an hsc2hs from the branch of that PR indeed fixes this issue, but it then just opens the way for the next iteration of the problem, which is that the cc-wrapper itself runs into problems, as discussed in https://github.com/NixOS/nixpkgs/issues/41340

jdevoto commented 5 years ago

I am having the same problem. I am using linux Mint 19 and stack version 2.1.3. ghc version 8.6.5. Any help is welcomed. I get

-- While building package haskell-gi-0.21.5 using: /home/jorge/.stack/setup-exe-cache/x86_64-linux/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.5 --builddir=.stack-work/dist/x86_64-linux/Cabal-2.4.0.1 configure --user --package-db=clear --package-db=global --package-db=/home/jorge/.stack/snapshots/x86_64-linux/91ebd899bce2a0b8d1c3cb377be2d3dc1c792aca539e4e193481b0124dadf70f/8.6.5/pkgdb --libdir=/home/jorge/.stack/snapshots/x86_64-linux/91ebd899bce2a0b8d1c3cb377be2d3dc1c792aca539e4e193481b0124dadf70f/8.6.5/lib --bindir=/home/jorge/.stack/snapshots/x86_64-linux/91ebd899bce2a0b8d1c3cb377be2d3dc1c792aca539e4e193481b0124dadf70f/8.6.5/bin --datadir=/home/jorge/.stack/snapshots/x86_64-linux/91ebd899bce2a0b8d1c3cb377be2d3dc1c792aca539e4e193481b0124dadf70f/8.6.5/share --libexecdir=/home/jorge/.stack/snapshots/x86_64-linux/91ebd899bce2a0b8d1c3cb377be2d3dc1c792aca539e4e193481b0124dadf70f/8.6.5/libexec --sysconfdir=/home/jorge/.stack/snapshots/x86_64-linux/91ebd899bce2a0b8d1c3cb377be2d3dc1c792aca539e4e193481b0124dadf70f/8.6.5/etc --docdir=/home/jorge/.stack/snapshots/x86_64-linux/91ebd899bce2a0b8d1c3cb377be2d3dc1c792aca539e4e193481b0124dadf70f/8.6.5/doc/haskell-gi-0.21.5 --htmldir=/home/jorge/.stack/snapshots/x86_64-linux/91ebd899bce2a0b8d1c3cb377be2d3dc1c792aca539e4e193481b0124dadf70f/8.6.5/doc/haskell-gi-0.21.5 --haddockdir=/home/jorge/.stack/snapshots/x86_64-linux/91ebd899bce2a0b8d1c3cb377be2d3dc1c792aca539e4e193481b0124dadf70f/8.6.5/doc/haskell-gi-0.21.5 --dependency=Cabal=Cabal-2.4.1.0-4t2ut7bCQNuEj8DDES6BZk --dependency=attoparsec=attoparsec-0.13.2.2-6XZWdvg7dAKGuzy18iXIM3 --dependency=base=base-4.12.0.0 --dependency=bytestring=bytestring-0.10.8.2 --dependency=containers=containers-0.6.0.1 --dependency=directory=directory-1.3.3.0 --dependency=filepath=filepath-1.4.2.1 --dependency=haskell-gi-base=haskell-gi-base-0.21.5-B7L3HQFSPkOFeLcsLCK63O --dependency=mtl=mtl-2.2.2 --dependency=pretty-show=pretty-show-1.9.5-4s9IVBWSe01IF3L4DDRCg8 --dependency=process=process-1.6.5.0 --dependency=regex-tdfa=regex-tdfa-1.2.3.2-LSpIJtcdqVBF6YmuXDgkYa --dependency=safe=safe-0.3.17-43oyx4B630gDZMbTh3Ttji --dependency=text=text-1.2.3.1 --dependency=transformers=transformers-0.5.6.2 --dependency=xdg-basedir=xdg-basedir-0.2.2-5JeadMzUyJb8NlvsA3m40x --dependency=xml-conduit=xml-conduit-1.8.0.1-yjX93nFqeiGJDLaI0qdo6 --exact-configuration --ghc-option=-fhide-source-paths Process exited with code: ExitFailure 1

cdepillabout commented 5 years ago

@jdevoto I don't think the underlying problem has been solved here yet.

However, as a workaround, I'm running stack in an fhsuserenv:

https://github.com/cdepillabout/termonad/blob/144b103c178c59e715812e1535c6503c457fd768/.nix-helpers/stack-fhs-env.nix

You should be able to use this like the following:

$ nix-shell ./stack-fhs-env.nix
$ stack build  # this is within the nix-shell

Or, just directly using cabal new-build like normal (instead of stack in an fhsuserenv) should work as well.

Also, since you're not actually on NixOS, one possibility is just to install the GTK libraries with your system package manager, and not involve nix at all.

jdevoto commented 5 years ago

Thanks for the help

stale[bot] commented 4 years ago

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.