search

commercialhaskell / stack

The Haskell Tool Stack
http://haskellstack.org
BSD 3-Clause "New" or "Revised" License
4k stars 843 forks source link

GHC panic when building Stack on macOS Sierra #2577

Closed bitemyapp closed 7 years ago

bitemyapp commented 8 years ago

GHC 7.10.3

stack upgrade --git
$ uname -a
Darwin cumae.local 16.0.0 Darwin Kernel Version 16.0.0: Tue Aug 23 17:02:44 PDT 2016; root:xnu-3789.1.31~2/RELEASE_X86_64 x86_64

Xcode 8.0 GM, macOS Sierra

Preprocessing library stack-1.2.1...
[ 1 of 96] Compiling Text.PrettyPrint.Leijen.Extended ( src/Text/PrettyPrint/Leijen/Extended.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.5.0/build/Text/PrettyPrint/Leijen/Extended.o )
[ 2 of 96] Compiling Stack.Ghci.Script ( src/Stack/Ghci/Script.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.5.0/build/Stack/Ghci/Script.o )
[ 3 of 96] Compiling Stack.FileWatch  ( src/Stack/FileWatch.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.5.0/build/Stack/FileWatch.o )
[ 4 of 96] Compiling System.Process.PagerEditor ( src/System/Process/PagerEditor.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.5.0/build/System/Process/PagerEditor.o )
[ 5 of 96] Compiling System.Process.Log ( src/System/Process/Log.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.5.0/build/System/Process/Log.o )
[ 6 of 96] Compiling Paths_stack      ( .stack-work/dist/x86_64-osx/Cabal-1.22.5.0/build/autogen/Paths_stack.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.5.0/build/Paths_stack.o )
[ 7 of 96] Compiling Path.Find        ( src/Path/Find.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.5.0/build/Path/Find.o )
[ 8 of 96] Compiling Path.Extra       ( src/Path/Extra.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.5.0/build/Path/Extra.o )
[ 9 of 96] Compiling System.Process.Read ( src/System/Process/Read.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.5.0/build/System/Process/Read.o )
ghc: panic! (the 'impossible' happened)
  (GHC version 7.10.3 for x86_64-apple-darwin):
        Loading temp shared object failed: dlopen(/var/folders/79/6lft40kj3cqb9f08_8yfg3vr0000gn/T/ghc29687_0/libghc_55.dylib, 5): no suitable image found.  Did find:
        /var/folders/79/6lft40kj3cqb9f08_8yfg3vr0000gn/T/ghc29687_0/libghc_55.dylib: malformed mach-o: load commands size (46680) > 32768

Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug

Completed 178 action(s).

--  While building package stack-1.2.1 using:
      /private/var/folders/79/6lft40kj3cqb9f08_8yfg3vr0000gn/T/stack-upgrade18342/stack/.stack-work/dist/x86_64-osx/Cabal-1.22.5.0/setup/setup --builddir=.stack-work/dist/x86_64-osx/Cabal-1.22.5.0 build lib:stack exe:stack --ghc-options " -ddump-hi -ddump-to-file"
    Process exited with code: ExitFailure 1
ilovezfs commented 7 years ago

https://github.com/Homebrew/homebrew-core/blob/master/Formula/haskell-stack.rb#L45

You need to pass --stack-yaml=stack-8.0.yaml

jasonstolaruk commented 7 years ago

Thank you for the quick reply! Unfortunately that didn't work and I'm still seeing the same error message...

borsboom commented 7 years ago

@jasonstolaruk stack upgrade in older versions of Stack are going to try to use the preferred LTS version to build stack-1.4.0, which is lts-6, regardless of your global project's default resolver. Newer versions of stack upgrade will just download a binary instead. Your best bet is to do a manual download from https://github.com/commercialhaskell/stack/releases. Future upgrades should be easier.

jasonstolaruk commented 7 years ago

@borsboom Ah, that would explain it! Yeah, I think my current version of stack is fairly old. I'll try that, thanks!

bitemyapp commented 7 years ago

@borsboom @jasonstolaruk I just got this with Stack 1.4.1, LTS 8.4, and Sierra. Not sure where to report, so chiming in here again.

    [ 2 of 16] Compiling Settings.StaticFiles ( src/Settings/StaticFiles.hs, .stack-work/dist/x86_64-osx/Cabal-1.24.2.0/build/Settings/StaticFiles.o )
    ghc: panic! (the 'impossible' happened)
      (GHC version 8.0.2 for x86_64-apple-darwin):
        Loading temp shared object failed: dlopen(/var/folders/gq/rh0h8yns2p1bb5d31mdx69b40000gn/T/ghc37966_0/libghc_33.dylib, 5): no suitable image found.  Did find:
        /var/folders/gq/rh0h8yns2p1bb5d31mdx69b40000gn/T/ghc37966_0/libghc_33.dylib: malformed mach-o: load commands size (33560) > 32768

Jason, are you sorted now?

It's one project of several at work that is tripping over this. Everything else builds.

Should I file a new ticket? Should it be with Stack?

bitemyapp commented 7 years ago

Just to confirm, still getting this with a Stack binary provisioned by the website via curl or manual download.

$ stack --version
Version 1.4.0, Git revision e714f1dd3fade19496d91bd6a017e435a96a6bcd (4640 commits) x86_64 hpack-0.17.0
mgsloan commented 7 years ago

@bitemyapp Not sure where that should be filed. Feel free to open a ticket, but it'll take a minimal repro for us to investigate it. Seems more likely to be a ghc issue.

jasonstolaruk commented 7 years ago

@bitemyapp Yep, I was able to build just now. Looks like I'm good.

macOS Sierra 10.12.3 Stack Version 1.4.0, Git revision e714f1dd3fade19496d91bd6a017e435a96a6bcd (4640 commits) x86_64 hpack-0.17.0 resolver: nightly-2017-04-02

rwbarton commented 7 years ago

This isn't and never was a GHC issue, it's a new limit added by Apple, for which a workaround was added to Cabal, which required a new feature in GHC.

If your project has enough dependencies it's possible to hit the limit again even with the workaround in place. The output of otool -l on the temporary shared object (invoke GHC with -keep-tmp-files) will clarify what is happening.

borsboom commented 7 years ago

I don't think it's possible for Stack to do anything about this. Even if it is possible to add a workaround to Stack, it's properly fixed in Cabal and/or GHC. It's annoying when Apple does this sort of thing, but unfortunately we're stuck with it (it's not a macOS bug; they intentionally added that limit, and we will just have to adapt).

steshaw commented 7 years ago

@chrisdone I can confirm that this still comes up for me on ghc-8.0.2 and lts-8.5 while building a yesod-based application:

    ghc: panic! (the 'impossible' happened)
      (GHC version 8.0.2 for x86_64-apple-darwin):
        Loading temp shared object failed: dlopen(/var/folders/l2/zpkx9mks1n12dbj0cy16nfn40000gn/T/ghc34456_0/libghc_10.dylib, 5): no suitable image found.  Did find:
        /var/folders/l2/zpkx9mks1n12dbj0cy16nfn40000gn/T/ghc34456_0/libghc_10.dylib: malformed mach-o: load commands size (33280) > 32768

    Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug

As @rwbarton points out, you can still hit the limit even with the workaround added to Cabal.

In my case, I may be able to shorted the length of some paths because I currently soft link ~/.stack to elsewhere (in my dotfiles repo) so that it picks up my .stack/config.yaml.

I wonder if there might be any benefit in soft linking ~/.stack to say /s to shorten paths further?

steshaw commented 7 years ago

A quick follow-up. For this app, I wasn't able to get it to build when I moved .stack to the usual location (ie. /Users/steshaw/.stack). However, when I soft linked /s to ~/.stack, I managed to get a successful build.

The tail end of the output from otool -l shows that the difference is made when addressing packages that come with ghc. Also, there is only a couple of hundred bytes to spare for sizeofcmds.

$ otool -l <file>.dylib | grep /s/programs
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/process-1.4.3.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/directory-1.3.0.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/filepath-1.4.1.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/template-haskell-2.11.1.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/pretty-1.1.3.3 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/ghc-boot-th-8.0.2 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/unix-2.7.2.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/time-1.6.0.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/transformers-0.5.2.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/binary-0.8.3.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/containers-0.5.7.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/bytestring-0.10.8.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/deepseq-1.4.2.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/array-0.5.1.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/base-4.9.1.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/integer-gmp-1.0.0.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/ghc-prim-0.5.0.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/rts (offset 12)
bitemyapp commented 7 years ago 
      cmdsize 80
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/directory-1.3.0.0 (offset 12)
Load command 125
          cmd LC_RPATH
      cmdsize 80
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/filepath-1.4.1.1 (offset 12)
Load command 126
          cmd LC_RPATH
      cmdsize 80
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/ghc-boot-th-8.0.2 (offset 12)
Load command 127
          cmd LC_RPATH
      cmdsize 80
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/ghc-prim-0.5.0.0 (offset 12)
Load command 128
          cmd LC_RPATH
      cmdsize 80
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/integer-gmp-1.0.0.1 (offset 12)
Load command 129
          cmd LC_RPATH
      cmdsize 80
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/pretty-1.1.3.3 (offset 12)
Load command 130
          cmd LC_RPATH
      cmdsize 80
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/process-1.4.3.0 (offset 12)

...

$ otool -l .stack-work/install/x86_64-osx/lts-8.4/8.0.2/lib/x86_64-osx-ghc-8.0.2/libHSyesod-core-1.4.32-G0tj9L3iIRq7jY7gWqui14-ghc8.0.2.dylib | grep /s/programs
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/array-0.5.1.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/base-4.9.1.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/binary-0.8.3.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/bytestring-0.10.8.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/containers-0.5.7.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/deepseq-1.4.2.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/directory-1.3.0.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/filepath-1.4.1.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/ghc-boot-th-8.0.2 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/ghc-prim-0.5.0.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/integer-gmp-1.0.0.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/pretty-1.1.3.3 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/process-1.4.3.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/rts (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/template-haskell-2.11.1.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/time-1.6.0.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/transformers-0.5.2.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/unix-2.7.2.1 (offset 12)

Thank you for sharing this tactic @steshaw, still not quite enough in my case:

(GHC version 8.0.2 for x86_64-apple-darwin):
        Loading temp shared object failed: dlopen(/var/folders/gq/rh0h8yns2p1bb5d31mdx69b40000gn/T/ghc76548_0/libghc_33.dylib, 5): no suitable image found.  Did find:
        /var/folders/gq/rh0h8yns2p1bb5d31mdx69b40000gn/T/ghc76548_0/libghc_33.dylib: malformed mach-o: load commands size (33288) > 32768

Other than the irritation of getting dyld to compile (you have to start vendoring the darwin userland into a build env), is there a reason people haven't made a fork that disables the byte limit? /cc @borsboom

ilovezfs commented 7 years ago

It's the loader itself that has the limit: https://github.com/commercialhaskell/stack/issues/2577#issuecomment-262725233

bitemyapp commented 7 years ago

@ilovezfs any idea as to why Apple did this?

ilovezfs commented 7 years ago

@bitemyapp I believe it's purely an optimization measure to improve load times, but there may also be undisclosed security issues with the prior versions for all we know. (Also, no one should run into the limit in the first place when libraries are arranged as intended.)

rwbarton commented 7 years ago

The LC_LOAD_DYLIB load command for one library is typically 72 bytes long (depending on the library's file name length), so even with no other load commands at all (in particular, not counting any LC_RPATH entries) that puts a limit of around 500 on the number of dependencies. My understanding is that many Yesod-based projects already comfortably exceed half this number so it's a limit that does not seem very far off.

borsboom commented 7 years ago

@rwbarton: would it help to shorten the library paths within ~/.stack? On Windows replace some of the deeper paths with a short hash to work around path length limits, and we could do something like that on macOS too (although I'd much rather not, since it makes it harder for a human to navigate ~/.stack).

bitemyapp commented 7 years ago

@borsboom My load commands typically look something like this:

          cmd LC_LOAD_DYLIB
      cmdsize 72
         name @rpath/libHSghc-boot-th-8.0.2-ghc8.0.2.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969

It's using @rpath to handle the path prefix. I don't know how the bytes are calculated by the runtime loader, but I'm not sure what more you can do.

This is a touch frustrating as the project that won't build is a pretty young Yesod project with not many extra dependencies beyond Yesod + lens.

domenkozar commented 7 years ago

Stack could create a library in intermediate steps, while the tree of dynamic libraries would be bigger, the limit could be mitigated.

I'm looking into possible solutions, for Nix one would be just to patch Apple source.

steshaw commented 6 years ago

Can anyone confirm that this is fixed in macOS High Sierra? I was previously on Sierra but had to wipe my machine and downgrade to El Capitan to get work done. I'm wondering if it's safe to upgrade to High Sierra for a yesod project with lots of dependencies (342 ?!?).

As far as I can tell, the GHC bug is fixed ... but it does look like someone had a similar problem on Sierra with it 7 weeks ago. Probably not fixed?

domenkozar commented 6 years ago

@steshaw it's not fixed, as the GHC fix only makes it harder to hit the limit in linker.

In Nix, we have a hack to reexport symbols to never reach the limit in one executable, so that should work with Stack.

I suggest you try out your project on Sierra (High Sierra doesn't change anything) and try to use Nix to provide the linker hack.

Long term solution is for haskell libraries not to use such long paths, now that stack/nix use separate folders and naming doesn't have to account for namespacing.

EugeneN commented 6 years ago

It’s still broken for me on High Sierra. I use docker container with ubuntu and some rsync-ing to get the job done.

On Sun, Dec 10, 2017 at 06:58 Domen Kožar notifications@github.com wrote:

@steshaw https://github.com/steshaw it's not fixed, as the GHC fix only makes it harder to hit the limit in linker.

In Nix, we have a hack to reexport symbols to never reach the limit in one executable, so that should work with Stack.

I suggest you try out your project on Sierra (High Sierra doesn't change anything) and try to use Nix to provide the linker hack.

Long term solution is for haskell libraries not to use such long paths, now that stack/nix use separate folders and naming doesn't have to account for namespacing.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/commercialhaskell/stack/issues/2577#issuecomment-350543254, or mute the thread https://github.com/notifications/unsubscribe-auth/ABNscrZrcjU1uAvSLzIUBgwHMorxpRPFks5s-8dhgaJpZM4J546w .

asivitz commented 6 years ago

I am running into this as well. Would it be reasonable to use Nix's workaround inside Stack? I am not familiar with Stack's internals, but if anyone thinks that would be a good approach I can investigate it.

Also, perhaps this issue should be reopened?

bitemyapp commented 6 years ago

@asivitz Stack had already done the rpath shortening trick, I think this is what remains as a more permanent fix.

https://github.com/NixOS/nixpkgs/pull/27536

asivitz commented 6 years ago

Yikes, I think that's a little more involved than I hoped.

Btw, I tried to make a controlled reproduction case by generating a giant stack project with a bunch of sub-packages, but I couldn't cause the panic. I wonder what I'm missing? https://gist.github.com/asivitz/f4b983b2374a6155ac4faaf9b61aca59

bitemyapp commented 6 years ago

@asivitz I think they need to be dependencies rather than modules? Make a Stack project that depends on acme-everything and see if that trips it.

asivitz commented 6 years ago

They are dependencies actually. It's a top-level library that depends on 400 sub-packages. It's pretty weird because when I looked at the output for my failure it was around 300 deps, and this one has 400 (and longer filenames also) and it doesn't trip it.

bitemyapp commented 6 years ago

@asivitz Oh my bad, I only glanced at the script. I've found reproducing the failure with Stack to be inconsistent in this way too, FWIW.

Did you try executing it or using some Template Haskell in it? I think you can force a linker failure during compile-time if you use TH.

asivitz commented 6 years ago

@bitemyapp Nope! Didn't try that. I'll give it a shot, thanks!

bitemyapp commented 6 years ago

@asivitz AIUI this is the runtime linker in the Darwin kernel that is imposing the kilobyte limit on the paths. macOS doesn't actually permit static linking at all. Given that, the only way to trip it that I'm aware of is for TH to force runtime linking at compile time or to run the program.

asivitz commented 6 years ago

Ah, ok that's super helpful, thanks. I added some TH and am now able to reproduce it reliably. I updated the gist.

jship commented 6 years ago

We have worked around this issue by following in the Nix folks' footsteps and using a wrapper script around ld to recursively subdivide the dependencies into a tree of re-exporting delegate libraries.

The script, an example using @asivitz's repro, and more info about this issue is available here: https://github.com/Simspace/ld-wrapper-macos

Hope this helps!

borsboom commented 6 years ago

I wonder: would it be possible to modify the GHC settings file to use this wrapper for all builds (rather than having to add the flag to every project)? If so, it would be possible to have Stack use patched GHC bindists on macOS that includes this fix.

gelisam commented 6 years ago

Sure, but you have to add -fuse-ld=ld-wrapper-macos.sh to the C compiler flags section of the settings file, you can't simply set the ld command option.

edit: for stack, the path for the settings file looks like ~/.stack/programs/x86_64-osx/ghc-8.2.2/lib/ghc-8.2.2/settings

borsboom commented 6 years ago

So we could, in theory, fix this for macOS users by providing patched bindists that include the wrapper script and a modified configure to update settings to use it on macOS >= 10.12. I'm always hesitant about this sort of thing, because it's one more thing we have to support going forward and would be best fixed by the GHC team themselves, but in this case there are enough macOS users and this causes enough pain that it might be worth doing.

steshaw commented 6 years ago

I've changed my settings to have the following flags and it built my project on High Sierra. Thank you!

("C compiler link flags", " -m64 -fuse-ld=ld-wrapper-macos.sh")
brandon-leapyear commented 6 years ago

:sparkles: This is an old work account. Please reference @brandonchinn178 for all future communication :sparkles:

@steshaw I'm getting 

clang: error: argument unused during compilation: '-fuse-ld=ld-wrapper-macos.sh' [-Werror,-Wunused-command-line-argument]

I'm on stack 1.6.5. Have you seen this problem?

gelisam commented 6 years ago

Just ignore that warning instead of making it fail with -Werror; the warning comes from clang, who noticed that you gave it linking flags in a clang invocation which doesn't perform any linking. This in turn occurs because adding -fuse-ld=ld-wrapper-macos.sh to C compiler link flags causes ghc to add this flag to all of its invocations of clang, both those which perform linking and those which don't.

steshaw commented 6 years ago

@brandon-leapyear @gelisam I don't see that error. Perhaps it's because I updated C compiler link flags rather than C compiler flags in ~/.stack/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/settings.

I do get some warnings during the linking stage like this:

/Users/steshaw/.stack/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/include/rts/storage/ClosureMacros.h:552:32: error:
     warning: macro expansion producing 'defined' has undefined behavior [-Wexpansion-to-defined]
#if ZERO_SLOP_FOR_LDV_PROF && !ZERO_SLOP_FOR_SANITY_CHECK
                               ^
...
9 warnings generated.

Seems those can be safely ignored.