Support musl-based Linux distributions like Alpine Linux

[borsboom]

@dysinger has had success building Stack on Alpine using GHC 8.0.1 from here: https://github.com/mitchty/alpine-linux-ghc-bootstrap. Once there is a Stackage LTS release that uses GHC 8, we can consider adding official support for Alpine Linux.

[borsboom]

For Alpine support, we need GHC bindists. I'm putting out a call to the community for help in creating those: can anyone out there provide GHC 8.0.1 bindists for x86_64 and x86 (32-bit) at a minimum? Would also be nice to have GHC 7.10.3 bindists, and bindists for the armhf platform, but that's less important. This looks like a good starting point: https://github.com/mitchty/alpine-linux-ghc-bootstrap

[borsboom]

Possibly relevant: musl bindists of ghc 8.2.1. I have not tried them; anyone available to do so? (ping @mitchty)

[mitchty]

They may work post 8.0.2, but the last time I tried gentoo musl produced bindists they included TEXT relocations in shared libraries which means you can't link anything you build with them. Alpine Linux has hardening on that will prevent such things from linking.

If thats the case they probably won't work at all. Worth a shot to try. You could test with the alpine/latest docker image if you want.

[mitchty]

Managed to get some time last night to get Jenkins up and running and doing builds. While it takes forever and an age to build ghc on armhf (like an entire day), I can get bindists generated now.

Still need to brain on how I want to do this going forward but in essence my strategy is: patch the APKBUILD in aports to also build bindists, then snag those bindists out of the build directory.

For 32bit ghc however I still need to get the cross compile to work sanely and get a build vm setup that I can hook Jenkins up to test things. But I need to do that to validate things in alpine linux proper anyway.

But in about a week or so I should have some strategy for bindists. How do/should they get generated/updated? My thoughts are follow the standard APKBUILD, for each release update the bindist and have say a -latest link to point to the latest bindist release for each specific major version. Keeping the last... N releases.

Thoughts?

[borsboom]

I know nothing about how Alpine packages work, so I don't really have an opinion TBH. For the purposes of Stack, the only thing that matters is having a working working bindist for each released GHC version (it's OK if that's just starting with the latest GHC version, although having versions going back to 7.10.3 would be nice).

[decentral1se]

Any progress @mitchty!? This'd be really cool to get done.

[decentral1se]

Whoever ends up closing this, I think it also closes https://github.com/commercialhaskell/stack/issues/2717.

Just scanning through, but looks to be the case.

[mitchty]

Work is a bit hectic so this is a bit on the back burner. But what I have partially is the ability to generate apks similar to hvr's ppa putting things in /opt/ghc/$version.

While that builds I hacked into the apkbuild process the ability to build bindists as well. This isn't too onerous with x86_64 but armhf takes forever and a day, actually 3 days to build stuff.

it needs some cleanup but shouldn't be too hard to finish up. Once I can focus back on it a bit I'll finish it off.

[mitchty]

Heh, well I got it working, only to find a bug in the llvm3.7 package too. It needs to get recompiled for alpine linux 3.6. As it is you see this on make install:

Installing library in /usr/local/lib/ghc-8.0.2/ghci-8.0.2
"utils/ghc-cabal/dist-install/build/tmp/ghc-cabal-bindist" copy compiler stage2 ":" '' '/usr/local' '/usr/local/lib/ghc-8.0.2' '/usr/local/share/doc/ghc-8.0.2/html/libraries' 'v p dyn'
Installing library in /usr/local/lib/ghc-8.0.2/ghc-8.0.2
"/tmp/ghc-8.0.2/utils/ghc-pkg/dist/build/tmp/ghc-pkg" --force --global-package-db "/usr/local/lib/ghc-8.0.2/package.conf.d" update rts/dist/package.conf.install
/bin/sh: /tmp/ghc-8.0.2/utils/ghc-pkg/dist/build/tmp/ghc-pkg: not found
make[1]: *** [ghc.mk:985: install_packages] Error 127
make: *** [Makefile:51: install] Error 2

That is all due to missing symbols in libc++ that the older llvm3.7 was linked against that alpine linux 3.6 doesn't have. I swear I am a magnet for finding unrelated bugs in things.

[mitchty]

Good news everyone! I was wrong in the last comment. I think I got it working. You as a user need to:

Add the community apk repository to your alpine system (if necessary). Example here is for alpine linux 3.6.

echo "http://dl-cdn.alpinelinux.org/alpine/v3.6/community" >> /etc/apk/repositories
apk update
apk add alpine-sdk llvm3.7 perl binutils gmp-dev clang gcc libffi-dev

Then try this out: https://mitchty.net/ghc/8.0.2/ghc-8.0.2-x86_64-alpine-linux.tar.xz

It seems to work for me but i just did MINIMAL testing, aka compile hello world, ./configure --prefix=/tmp/ghc example:

/tmp/ghc-8.0.2 # echo 'main = putStrLn "hi"' > /tmp/hi.hs
/tmp/ghc-8.0.2 # /tmp/ghc/bin/ghc --make /tmp/hi.hs
Linking /tmp/hi ...
/tmp/ghc-8.0.2 # /tmp/hi
hi

I have 8.2.1 working as well but only for x86_64. It'll take a while to get that going but test away.

[mbj]

I'd prefer we use APK as primary way to package GHC for alpine.

@mitchty Some of your patches where merged upstream already. Can you publish your 8.2.1 apkbuild somewhere, I'd try to adapt it to 8.2.2 and publish my findings here.

[mitchty]

Its already been done, the issue is building on armhf segfaults in the stage2 builder. Thats the only reason there isn't an update for ghc in alpine linux right now, generally frowned on to remove a platform from the supported arches list. Debugging this is... trying as I need to negate out llvm, musl patches, ghc, the rts, etc...

Apkbuild is here: https://github.com/alpinelinux/aports/compare/master...mitchty:jenkins-8.2.2

[mitchty]

Note, that has the binary distribution stuff in it as well, you might want to rip that out if you don't want to build the binary distribution tar.xz files.

Essentially this is my jenkins branch that I use to test release candidates and to build the above binary distribution files. But for 8.2.2, I have 8.2.1 and 8.0.2 as well for x86_64, example:

https://mitchty.net/ghc/8.2.2/ghc-8.2.2-x86_64-alpine-linux.tar.xz

If anyone wants I can resurrect my apk signing keys and plop up the apks I build out of jenkins into that web server until I can fully debug the armhf segfault issue.

[mbj]

@mitchty Would alpine accept your apk if its not supporting armhf? And thx for the pointer. I can temporarily unblock myself now.

[mbj]

If anyone wants I can resurrect my apk signing keys and plop up the apks I build out of jenkins into that web server until I can fully debug the armhf segfault issue.

for now I'll build / package it by myself, but thanks for the offer. I think your time is better spend in getting the changes upstream ;) I do not want to reduce the chance of this happening soon.

[ketzacoatl]

In case this detail is helpful.. As of right now, running the static linux executable for stack, in an alpine 3.7 container, GHC build blows up with errors about cabal:

#  "/root/.stack/programs/x86_64-linux/ghc-ncurses6-nopie-8.2.2/lib/ghc-8.2.2/rts" = directory to install to
#
# The .dll case calls STRIP_CMD explicitly, instead of `install -s`, because
# on Win64, "install -s" calls a strip that doesn't understand 64bit binaries.
# For some reason, this means the DLLs end up non-executable, which means
# executables that use them just segfault.
/usr/bin/install -c -m 755 -d  "/root/.stack/programs/x86_64-linux/ghc-ncurses6-nopie-8.2.2/lib/ghc-8.2.2/rts"
for i in  rts/dist/build/libHSrts.a rts/dist/build/libHSrts_p.a rts/dist/build/libHSrts-ghc8.2.2.so rts/dist/build/libHSrts_l.a rts/dist/build/libHSrts_debug.a rts/dist/build/libHSrts_thr.a rts/dist/build/libHSrts_thr_debug.a rts/dist/build/libHSrts_thr_l.a rts/dist/build/libHSrts_thr_p.a rts/dist/build/libHSrts_debug-ghc8.2.2.so rts/dist/build/libHSrts_thr-ghc8.2.2.so rts/dist/build/libHSrts_thr_debug-ghc8.2.2.so rts/dist/build/libHSrts_l-ghc8.2.2.so rts/dist/build/libHSrts_thr_l-ghc8.2.2.so rts/dist/build/libffi.so rts/dist/build/libffi.so.6 rts/dist/build/libffi.so.6.0.4 rts/dist/build/libCffi.a rts/dist/build/libCffi_p.a rts/dist/build/libCffi_l.a rts/dist/build/libCffi_debug.a rts/dist/build/libCffi_thr.a rts/dist/build/libCffi_thr_debug.a rts/dist/build/libCffi_thr_l.a rts/dist/build/libCffi_thr_p.a; do case $i in *.a) /usr/bin/install -c -m 644  $i  "/root/.stack/programs/x86_64-linux/ghc-ncurses6-nopie-8.2.2/lib/ghc-8.2.2/rts"; true  "/root/.stack/programs/x86_64-linux/ghc-ncurses6-nopie-8.2.2/lib/ghc-8.2.2/rts"/`basename $i` ;; *.dll) /usr/bin/install -c -m 755  $i  "/root/.stack/programs/x86_64-linux/ghc-ncurses6-nopie-8.2.2/lib/ghc-8.2.2/rts" ; :  "/root/.stack/programs/x86_64-linux/ghc-ncurses6-nopie-8.2.2/lib/ghc-8.2.2/rts"/`basename $i` ;; *.so) /usr/bin/install -c -m 755  $i  "/root/.stack/programs/x86_64-linux/ghc-ncurses6-nopie-8.2.2/lib/ghc-8.2.2/rts" ;; *.dylib) /usr/bin/install -c -m 755  $i  "/root/.stack/programs/x86_64-linux/ghc-ncurses6-nopie-8.2.2/lib/ghc-8.2.2/rts";; *) /usr/bin/install -c -m 644  $i  "/root/.stack/programs/x86_64-linux/ghc-ncurses6-nopie-8.2.2/lib/ghc-8.2.2/rts"; esac; done
"utils/ghc-cabal/dist-install/build/tmp/ghc-cabal-bindist" copy libraries/ghc-prim dist-install ":" '' '/root/.stack/programs/x86_64-linux/ghc-ncurses6-nopie-8.2.2' '/root/.stack/programs/x86_64-linux/ghc-ncurses6-nopie-8.2.2/lib/ghc-8.2.2' '/root/.stack/programs/x86_64-linux/ghc-ncurses6-nopie-8.2.2/share/doc/ghc-8.2.2/html/libraries' 'v p dyn'  

utils/ghc-cabal/dist-install/build/tmp/ghc-cabal-bindist: exec: line 3: utils/ghc-cabal/dist-install/build/tmp/ghc-cabal: not found
make[1]: *** [ghc.mk:991: install_packages] Error 127
make: *** [Makefile:51: install] Error 2

Error: Error encountered while installing GHC with
         make install
         run in /root/.stack/programs/x86_64-linux/ghc-ncurses6-nopie-8.2.2.temp/ghc-8.2.2/

       The following directories may now contain files, but won't be used by stack:
         - /root/.stack/programs/x86_64-linux/ghc-ncurses6-nopie-8.2.2.temp/
         - /root/.stack/programs/x86_64-linux/ghc-ncurses6-nopie-8.2.2/
``

[borsboom]

@ketzacoatl Did you add setup-info configuration pointing to @mitchty's bindist for the linux64-ncurses6-nopie OS key? If not, Stack will try to install a standard glibc bindist, since it doesn't know about Alpine specifically.

I've not tried the bindist above on Alpine myself, so I'd be very interested in the results.

[ketzacoatl]

I have added the following to my stack.yaml:

resolver: lts-10.2

setup-info:
  ghc:
    linux64-alpine:
      8.2.2:
        url: https://mitchty.net/ghc/8.2.2/ghc-8.2.2-x86_64-alpine-linux.tar.xz

ghc-build: alpine

and then I run into the LLVM bug mitchty noted in a previous comment, even with the updates he noted. For example, this build is run in a docker image that has the packages installed as noted in mitchty's instructions, but fails with:

/bin/sh: /builds/web/haskell-data-sink-demo/.stack-root/programs/x86_64-linux/ghc-alpine-8.2.2.temp/ghc-8.2.2/utils/ghc-pkg/dist/build/tmp/ghc-pkg: not found
make[1]: *** [ghc.mk:992: install_packages] Error 127
make: *** [Makefile:51: install] Error 2

Error: Error encountered while installing GHC with
         make install
         run in /builds/web/haskell-data-sink-demo/.stack-root/programs/x86_64-linux/ghc-alpine-8.2.2.temp/ghc-8.2.2/

       The following directories may now contain files, but won't be used by stack:
         - /builds/web/haskell-data-sink-demo/.stack-root/programs/x86_64-linux/ghc-alpine-8.2.2.temp/
         - /builds/web/haskell-data-sink-demo/.stack-root/programs/x86_64-linux/ghc-alpine-8.2.2/

ERROR: Job failed: exit code 1

[naushadh]

Any updates on this issue? I too tried the suggestions presented above and ran into the same error @ketzacoatl commented right above. I would love to switch to alpine base docker image (instead of minideb/debian) to get that ultra light final app image.

[ketzacoatl]

IDK @naushadh.

@mitchty, do you have any advice or guidance, what are the reasonable next steps?

[vikrem]

Also trying with the 8.2.2 bindist and encountering the same issue :(

[naushadh]

@ketzacoatl : I managed to get a hello-world app built and run on an alpine docker image. The trick is to use a stockage resolver with a ghc version that you find a fully working bin-dist for. Since I couldn't find one for ghc-8.2.2, I instead went with lts-9.21 as there is a working official bin dist for ghc-8.0.2.

See sample project for how it all fits together.

[borsboom]

This is all not looking very optimistic for having official Alpine support any time soon, given the current (and soon to be previous, since GHC 8.4.1 should be out soon) doesn't appear to work at all.

[borsboom]

BTW, has anyone started working on a GHC 8.4 build on Alpine?

[mitchty]

Yep, I have it working on x86_64, but hitting an issue with 32 bit arm, which is what held up 8.2.1 and 8.2.2. The arm flavor of things has been.... problematic. At this point I think it might be easier to drop it in general.

[naushadh]

I'd be happy with x86-64 support for now.

[mitchty]

The bug I'm hitting on arm as a note.

https://ghc.haskell.org/trac/ghc/ticket/14739

Also affects builds of 32 bit ghc on other platforms such as nix, so its not just an alpine thing.

[chrisdone]

@mitchty Apparently GHC 8.4 dropped support for ARM? https://www.haskell.org/ghc/download_ghc_8_4_1.html#linux_armv7

[nh2]

I've filed https://github.com/commercialhaskell/stack/issues/4088 for getting statically linked stack binaries back.

[nh2]

I have now built a fully static stack binary with musl and nix (only stack v1.6.5 for now).

https://github.com/nh2/static-haskell-nix/blob/master/static-stack

These binaries should work fine on any Alpine right now as well. Any Alpine user want to try it out?

[tyhal]

Having an issue getting stack to attempt to use the alpine ghc-8.4.3 (stack v1.7.1)

No setup information found for ghc-8.4.3 on your platform.
This probably means a GHC bindist has not yet been added for OS key 'linux64-ncurses6'.
Supported versions: ghc-7.10.3, ghc-8.0.1, ghc-8.0.2, ghc-8.2.1, ghc-8.2.2, ghc-8.4.1, ghc-8.4.2

using 8.4.3 https://pkgs.alpinelinux.org/package/edge/community/x86_64/ghc

[borsboom]

@tylerhale Indeed, there are no GHC bindists for Alpine Linux.

[t3hmrman]

I think I just ran into this:

Step 7/13 : RUN curl -sSL https://get.haskellstack.org/ | sh
 ---> Running in 0c5a64a6f13e
Detected Linux distribution: alpine

fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz
(1/1) Installing xz (5.2.3-r0)
Executing busybox-1.26.2-r5.trigger
OK: 1077 MiB in 82 packages
Sorry, there is currently no 64-bit Alpine Linux binary available.
The command '/bin/sh -c curl -sSL https://get.haskellstack.org/ | sh' returned a non-zero code: 1

Someone emailed me about a Dockerfile in a blog post and i ran it only to find this error. While installing stack in this way inside a docker container is a good/bad thing I was surprised to find this error when the script was working before -- am I missing something obvious? If I just make sure to pick an older GHC version will the stack script find an older/previously-available binary distribution of itself?

[borsboom]

This means there's no official stack binary available for Alpine anymore. That's because the latest Stack version needs GHC 8.2.2 to build, and that version of GHC doesn't work on Alpine.

If you want to get the last version that did have an Alpine binary, use a command like this:

curl -L https://github.com/commercialhaskell/stack/releases/download/v1.6.5/stack-1.6.5-linux-x86_64-static.tar.gz | tar xz --wildcards --strip-components=1 -C ~/.local/bin '*/stack'

Or, if you need the latest stack, use the unofficial static binary that @nh2 built (see https://github.com/commercialhaskell/stack/issues/2387#issuecomment-397419789).

We do hope to have this rectified for the next stack release.

As a general Docker practice, it's not a good idea to rely on curl'd installer scripts, because you never know when they'll change and break your Dockerfile (like in this case).

[t3hmrman]

@borsboom Thanks for explaining it so clearly.

I did not realize that the latest Stack requires GHC 8.2.2 to build -- previously my project was using 8.0.2 which explains why it was working before.

As far as the general Docker practice, I totally hear you (though IMO apt get/apk add/dnf install and a curl installer script that installs binaries aren't too far apart in my mind) when I built the alpine image I was trying to get away from the relatively heavy fpco/* Stack base images (which have since gotten smaller I think), as well as attempting completely haskell static builds.

[borsboom]

I did not realize that the latest Stack requires GHC 8.2.2 to build -- previously my project was using 8.0.2 which explains why it was working before.

To be clear, Stack still fully supports GHC versions back to 7.8.4 for building projects. GHC 8.2.2 is just required to build itself.

[mitchty]

If we want I could build "unofficial" x86_64 alpine bindists for 8.0.2/8.2.2 and throw them up on my vps, they would require people to install the dependencies manually to use them. I could also throw up the dockerfiles I use to produce them as the ghc builds take too long for normal docker images to build. A caveat, building with multiple processes tends to make the builds non reproducible without doing make -j 1 for example.

I'll fire up my alpine linux vm and get cracking on that. I can contribute those dockerfiles upstream and hopefully get alpine linux on x86_64 as a non tier 2 platform for ghc.

I know this one worked the last time I tried it, I think.... at the bar so not sure. >.< https://mitchty.net/ghc/8.0.2/ghc-8.0.2-x86_64-alpine-linux.tar.xz

I don't remember if this worked or not though, if not I'll fix it so it does work. https://mitchty.net/ghc/8.2.2/ghc-8.2.2-x86_64-alpine-linux.tar.xz

[mitchty]

Sounds like all thats needed for official bindists is timely bindist generation around release time. I'll steal^Wborrow from @borsboom 's dockerfile and simplify it a bit and get some dockerfiles for 8.0.2, 8.2.2, and 8.4.3 and then finally the 8.6.x series which are their own level of fun I need to tackle for the apkbuilds anyway.

Anyone for/against? Any better ideas?

[nh2]

@mitchty Another alternative might be to work directly upstream with the GHC team (very nice people) to provide official GHC bindists for Alpine.

The point of the GHC devops group is to have non-core-ghc-devs help with infrastructure and releases. @mitchty, if you are up to the tasks of maintaining build scripts for Alpine on your own, you may find it an easy step to do it officially as part of this group instead.

[mitchty]

@nh2 Yep thats the plan, I call these "unofficial" for now in that I'll just get bindists for 8.0, 8.2, and 8.4 setup with the same patches as the "official" alpine linux apk's have. Then work on getting haskell.org official bindists as well.

Having an official x86_64 bindist would also help in making apk's at least without involving a glibc step again.

I'll have a look at that group as well, thanks!

[thomasjm]

Thanks for the hard work on this everyone, this looks awesome! Could someone tell me the current state of things? I see that Alpine has recent GHC packages now, like ghc-8.4.3-r0. There doesn't seem to be a current Alpine stack binary. What I'd really like to do is build a Haskell app on Alpine using stack build --docker with a publicly available Alpine image, is this feasible at the moment?

[hasufell]

mitchy.net seems down and no way of fetching the pre-built 8.2.2 GHC tarballs anymore.

[gliptak]

(Many) tarballs are also available at https://github.com/commercialhaskell/ghc/releases

Maybe above could be updated?

[hasufell]

I don't know what you mean. Those are not alpine tarballs.

[gliptak]

I was proposing that Alpine tarballs be added to publishing setup.

[thomasjm]

@mitchty , was there any luck working toward official bindists?

It sounds like a working version of GHC 8.2.2 on Alpine is a prerequisite for building stack itself on Alpine, and after that it should be straightforward to build other things with stack--is that right?

[borsboom]

We're working on using nix to build a static stack binary which should, in theory, work on Alpine Linux as well. Without Alpine Linux GHC bindists, however, Stack will not be able to manage different versions of GHC for you -- you'll have to install Alpine's ghc package and will be limited to building projects that use that version of GHC.

[thomasjm]

Thanks for the clarification @borsboom ! Is there a ticket I can follow for the static stack work? (I saw #2083 but it's a couple years old.) Installing ghc via package doesn't sound too bad.

[hasufell]

Installing ghc via package doesn't sound too bad.

It's very bad, since there is no ghc-8.2.2 and only one system-ghc you can have, not multiple at the same time. That simply doesn't work for anything remotely production ready.

[nh2]

and only one system-ghc you can have, not multiple at the same time

As far as I remember, system-ghc picks it from PATH, so you could do PATH=/path/to/desired/ghc/bin:$PATH stack build Also, I think you can use stack's feature to configure custom compilers, so these might be two workarounds for now.