Feature Request: Official ARM support

[blitzcode]

The current process of getting a Stack Haskell application build on ARM is rather... tricky:

https://github.com/blitzcode/hue-dashboard/#raspberry-pi

I'd love for Stack to have an official ARM bindist and be able to install GHC/LLVM with stack setup.

[borsboom]

The main thing we'd need to start to start generating ARM binaries for releases is a Vagrantfile that sets up a build environment similar to the ones found in https://github.com/commercialhaskell/stack/tree/master/etc/vagrant. I guess that Vagrantfile would have to launch a VM on an Arm-supporting VPS provider (or in an emulator, but that might be really slow).

From your link, it sounds like we may also want to add support for installing LLVM to stack setup. Although I guess if you already have LLVM, stack setup might already be able to take care of installing GHC (at least once we add a link to the correct GHC bindist for ARM to https://github.com/fpco/stackage-content/blob/master/stack/stack-setup-2.yaml).

[blitzcode]

I initially tried building on QEMU, but had little success with it. The Raspberry Pi with its 1GB of RAM might be problematic for reliable builds. As you suggest, a VPS provider that offers server-grade ARM systems with enough memory seems like the best option.

I filed a issue #1742 a while ago to suggest stack should install LLVM as it's often difficult to pick the right version and different GHC versions need different LLVMs. Wasn't there talk of GHC8 bundling the right LLVM anyway?

[quyse]

@blitzcode Just a few notes, I have set up an ARM machine on Scaleway few months ago for CI purposes, and it's doing great. I would advise to use Arch Linux distribution instead of Debian/Ubuntu, as it has up-to-date binary packages, in particular GHC 7.10.3 (on ARM too) and LLVM, which work together flawlessly. As I remember the setup was relatively straightforward, just install ghc, llvm35, cabal-install from packages, add some swap space (2Gb RAM will not be enough), compile stack from sources, and that's it. Don't do parallel builds, as it will run out of memory. Unfortunately I don't know other good ARM cloud providers, I would love to hear about any providing bigger ARM machines in terms of memory.

[blitzcode]

@quyse Yes, I was checking them out, seems really affordable. 2GB really wasn't enough? I got everything to build with the 1GB RPi and 100MB swap, I figure 2GB should be smooth sailing. Maybe only with -j1 / -j2, though. I'm a bit wary of Arch simply because I don't like the idea of being forced to upgrade and I want reproduceability. If 1 year from now the SD card in a RPi I deployed dies, I'd want to be able to set up the exact same system again. If Arch has already moved to GHC8 by then, I might be in trouble with my LTS 5.x code. At least that's my worry without ever having tried Arch. It seems otherwise much easier to get going than Debian Jessie with its dated packages...

[quyse]

@blitzcode I've checked again what I'm doing there, actually I'm not specifying -j1, so I guess it's actually -j4 by default (because of 4 cores), no wonder it was running out of memory :) With additional swap space it's fine though. -j1 is just too slow. Good point about Arch, yes, you are usually forced to upgrade. You can freeze your system's state by simply not upgrading, you also can make a backup of your pacman cache, so you can restore packages in the future without contacting distribution servers. But, yes, probably easier to stick with Debian in your case. I'm just not worrying too much about reproduceability at the moment, as I'm upgrading regularly to the latest LTS.

[borsboom]

I'm trying to build a stack binary on a Scaleway VM running Debian Jessie (to match the GHC bindist), but am having trouble with LLVM. I downloaded an LLVM bindist as recommended in here, but when I try to use it I get:

# opt --version
Illegal instruction

Any ideas where I can easily get an LLVM that will work?

[blitzcode]

@borsboom That's puzzling, last time I checked their servers were armv7 / 32 bit, same as the Raspberry Pi with Raspbian. So it's really strange that the LLVM bindist would not work.

Could you maybe do a lscpu, uname -a and maybe run file or whatever on the LLVM bindist and see if the architectures etc. match? I also sometimes hear that SoC vendors have a lot of discretion on what to include in their ARM cores, like NEON, 32/64 bit, etc, so maybe there's something different about the server-grade SoCs Scaleway uses. No idea.

[borsboom]

@blitzcode:

root@scw-476b18:~# uname -a
Linux scw-476b18 4.3.5-std-1 #1 SMP Fri Feb 19 11:52:18 UTC 2016 armv7l GNU/Linux

root@scw-476b18:~# lscpu
Architecture:          armv7l
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    1
Core(s) per socket:    4
Socket(s):             1
Model name:            ARMv7 Processor rev 2 (v7l)
CPU max MHz:           1333.0000
CPU min MHz:           666.5000

root@scw-476b18:~# /usr/local/bin/opt --version
Illegal instruction

root@scw-476b18:~# file /usr/local/bin/opt
/usr/local/bin/opt: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, BuildID[sha1]=f99b13805d70643bc8a3485026fb7ab9f6a37f04, stripped

Oddly enough, the file output matches that for ghc (which does run):

root@scw-476b18:~# /opt/ghc-7.10.3/lib/ghc-7.10.3/bin/ghc --version
The Glorious Glasgow Haskell Compilation System, version 7.10.3

root@scw-476b18:~# file /opt/ghc-7.10.3/lib/ghc-7.10.3/bin/ghc
/root/opt/ghc-7.10.3/lib/ghc-7.10.3/bin/ghc: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, BuildID[sha1]=c2002709f062e00b92536c9c6770e7fa9d96f7d8, not stripped

[varosi]

@borsboom, you need at least LLVM 3.5.2 or greater. There is a fix related to GHC in it.

[varosi]

@blitzcode, I'm running GHC 7.10.2 with LLVM 3.5.2 on ARMv7 (32bit) and it is working well.

[blitzcode]

@borsboom I'm terribly sorry, I said 3.5.2 in my document and then linked 3.5.1, dammit...

[mgsloan]

Note that another thing that will likely need to be resolved is support for byte alignment in store - https://github.com/fpco/store/issues/37 - as ARM cares about this.

[varosi]

Related: https://github.com/haskell-crypto/cryptonite/issues/90

[varosi]

This temporarily workarounds this problem:

cabal install --dependencies-only --constraint=cryptonite==0.15

[andreyk0]

Jumping here from https://github.com/commercialhaskell/stack/issues/1332

I've finally taken some time to untangle haskell/arm stuff from my personal files, here's the result: https://github.com/andreyk0/haskell-on-arm

Hope it helps somebody.

@varosi cryptonite used to be an issue for me at some point but haven't had any problems recently. Something got fixed, I imagine.

[varosi]

@andreyk0, fix is not yet pushed in Hackage and I still have this problem on my ARM.

[andreyk0]

@varosi hmm, could be my environment then. I don't remember everything but (it's in the readme in that repo and my images are on dockerhub) I've installed LLVM from the project's downloads page and hacked 7.10.2 to be a symlink to 7.10.3, so there's no 7.10.2 in my system.

[varosi]

with cryptonite-0.15 it is working fine on IMX.53 board. So I'm building project successfully with Stack. It'll be great if Stack have official distro for ARM and support stack setup.

[borsboom]

I did some more experimenting with this. I got Debian Jessie for armhf running in qemu (note: the LLVM bindists are not compatible with armel or with Wheezy for armhf which I tried first) and things seem to be working, but very slow. I'm building cabal-install and it's been running for about 10 hours now, and looks like it might be finished in the next hour or so. At that rate, Stack will take at least a couple of days to build, and I expect the integration tests to take another day or so to run. At that point I'll at least know whether Stack works properly on ARMv7.

The situation with LLVM bindists is annoying though; they only seem to support fairly specific versions of distros (unlike GHC bindists, which tend to work on a wide range). And since each GHC version needs a specific version of LLVM, we can't leave it up to the Linux distro to install LLVM for us. So we could officially support ARMv7 on Jessie armhf, but not necessarily on any other distributions or versions.

[blitzcode]

@borsboom Good to hear, but I'm surprised that the build times are so much worse than on the RPi. I've read several times about good results with QEMU. Hope you build Cabal with EXTRA_CONFIGURE_OPTS="" ./bootstrap.sh, it helps quite a bit!

In the meantime, Stack v1.1 has been working absolutely without issues for me on ARM after the initial setup pains.

Debian Wheezy only seems to have LLVM 3.1 available, Wheezy only goes to 3.5, had to download the LLVM binaries. Where on the LLVM page does it actually say with which distros the binaries have been tested?

[borsboom]

Thanks for the tip about EXTRA_CONFIGURE_OPTS; I didn't know that!

I didn't see anywhere on the LLVM page where it mentioned the distros. But when I tried clang+llvm-3.7.1-armv7a-linux-gnueabihf (I'm using GHC 8.0.1, so the newer LLVM is required) on Wheezy, this happened:

manny@debarm:~$ clang
clang: /usr/lib/arm-linux-gnueabihf/libstdc++.so.6: version `GLIBCXX_3.4.18' not found (required by clang)
clang: /lib/arm-linux-gnueabihf/libc.so.6: version `GLIBC_2.15' not found (required by clang)

[blitzcode]

Oh, it's gated by glibc, that makes sense. That's an unfortunate problem...

Once you have Cabal be sure to use --disable-library-stripping --disable-executable-stripping and / or set executable-stripping: False + library-stripping: False in Cabal's ~/.cabal/config. I had to start over and rebuild all of Stack's dependencies on my first try :/

[varosi]

I have tested latest cryptonite-0.17 on ARMv7 and the compilation issue from 0.16 is fixed (haskell-crypto/cryptonite#90) there. So Stack will be building alright now.

[andreyk0]

@varosi yeah, I must've hacked flags long time ago and forgot about it, sorry for the noise, you were right. Had the same problem with a clean ghc8.0.1 build, luckily it's fixed now!

[andreyk0]

@borsboom " ARMv7 on Jessie armhf" is probably the way to go, too much pain to get this stuff to work anywhere else for now. Regarding QEMU build times: I only used cabal-install once, about a year ago to build stack itself. Since then using stack to build packages and newer versions of stack itself. Most packages and my tiny projects compile in a few minutes, which is reasonable. There are outliers though, e.g. compiling regex-tdfa the other day with ghc8.0.1 took over 2h. Don't know why. At least stack will reuse the binaries, which helps a lot.

[borsboom]

Replying to myself here, regarding https://github.com/commercialhaskell/stack/issues/2103#issuecomment-220727107: I've seen some issues in other projects that need ARM CPUs with NEON support and have the same Illegal Instruction error on Scaleway (where the CPU doesn't have NEON support), so I'm guessing that's what is going on with the LLVM bindist. Annoying, because this would be much faster/easier if I could build on Scaleway.

[borsboom]

According to this:

The LLVM backend is now included in GHC HEAD. Just grab the git HEAD version of GHC and build it.

So Stack having to install LLVM itself may not be necessary with future versions of GHC. Still needed for GHC 8.0.1 and 7.10.* though.

aside: I've successfully built Stack on ARMv7, and am now rebuilding Stack using itself, after which I'll run the integration tests.

[borsboom]

... unless that's just referring to GHC's own LLVM code generation backend, and not the LLVM toolchain itself... which upon re-reading again is probably the case.

[borsboom]

Hooray, the integration tests passed! Now working on an official Stack bindist for ARMv7 so others won't have to go through the painful bootstrap process. Note however, that this does not mean Stack has official support for ARM yet. That still needs a few updates so that stack setup works. If anyone on this thread wants to help with that, I'd be very happy!

[blitzcode]

That's amazing to hear! Looking forward to deleting my backups of stack/cabal binaries, didn't want to go through this process a second time either.

[borsboom]

ARM bindist is now available at https://github.com/commercialhaskell/stack/releases/tag/v1.1.2. I'll endeavour to build them for future releases as well, although there will be some lag because it takes so long to build.

[andreyk0]

@borsboom awesome! Thanks a lot! It works.

[borsboom]

Over in this reddit comment, @thoughtpolice wrote:

I probably wouldn't bother making Stack support LLVM installation, although it might be convenient for now. See this plan which I'm hoping to finish up. In the future, Stack (or anyone else including Debian) hopefully should not have to worry installing the right LLVM, only GHC: which will instead come with a copy of the two needed LLVM tools (opt and llc), for private use.

So this should hopefully be much easier with future GHC versions. So if someone wants to put in the effort to have Stack install LLVM itself I'd be very happy to give advice and review code, but I don't think I'll be doing it myself for the time being.

[varosi]

Raspberry Pi 3 on directory without stack.yaml:

> pi@haskellbox:~ $ stack --version
> Version 1.1.2, Git revision 6bca6d8eaf542c5bd4b5c83d23670137249f2145                              (3663 commits) arm hpack-0.14.0
> pi@haskellbox:~ $ stack ghci
> Run from outside a project, using implicit global project config
> Using latest snapshot resolver: lts-6.4
> Writing implicit global project config file to: /home/pi/.stack/globa                             l-project/stack.yaml
> Note: You can change the snapshot via the resolver field there.
> Downloaded lts-6.4 build plan.
> Fetching package index ...remote: Counting objects: 218984, done.
> remote: Compressing objects: 100% (176510/176510), done.
> remote: Total 218984 (delta 58498), reused 172587 (delta 40275), pack                             -reused 0
> Receiving objects: 100% (218984/218984), 51.30 MiB | 4.51 MiB/s, done                             .
> Resolving deltas: 100% (58498/58498), completed with 1 local objects.
> From https://github.com/commercialhaskell/all-cabal-hashes
>  * [new tag]         current-hackage -> curFetched package index.
> Populated index cache.
> Did not find .cabal file for type-eq-0.5 with Git SHA of 44ef3dae4dac                             dc2729a2fde3ced964bd52ae05c7
> Right Nothing
> Did not find .cabal file for language-ecmascript-0.17.1.0 with Git SH                             A of 3f4043479db04f39fe3734b61ebfee6384a02596
> Right Nothing
> Did not find .cabal file for ixset-typed-0.3 with Git SHA of e2f90193                             88d77b3fb23b2fb6395564a6cd5bd7d9
> Right Nothing
> Did not find .cabal file for fay-builder-0.2.0.5 with Git SHA of 45d2                             4fd3663ae74696726d87a783469679ad6472
> Right Nothing
> Did not find .cabal file for fay-0.23.1.12 with Git SHA of 8dbcdfd039                             559115ca6928f4140cc880b1a4e6b3
> Right Nothing
> Did not find .cabal file for HDBC-2.4.0.1 with Git SHA of df385b10294                             c417ae6f2667b9cbc7c020393722f
> Right Nothing
> Error parsing targets: The specified targets matched no packages.
> Perhaps you need to run 'stack init'?
> Warning: build failed, but optimistically launching GHCi anyway
> Configuring GHCi with the following packages:
> GHCi, version 7.10.3: http://www.haskell.org/ghc/  :? for help
> Ok, modules loaded: none.
> 

(EDIT by @Blaisorblade: improved line breaks).

[borsboom]

@varosi do you actually end up with a Prelude> prompt? It seems to work for me on qemu, anyway. The "Did not find .cabal file" warnings are a known bug that affects all platforms.

[andreyk0]

@varosi @borsboom I don't think it's QEMU related, I've seen these odd messages too after the upgrade, guessed it had something to do with the stack version change. Ran 'git clean -fdx .' in the project and re-run stack build, everything worked fine after that.

[varosi]

@borsboom, it show Prelude>, yes. And it is working fine. It's strange that on the second run now it doesn't show those warnings.

[borsboom]

Over in https://github.com/commercialhaskell/stack/issues/257#issuecomment-229538337, @GordonBGood said:

Even easier is just to put the required versions of the LLVM programs "llc" and "opt" in the exec folder of ghc, which for a normal HaskellPlatform Linux install is /usr/local/Haskell/ghc-8.0.1-x86_64/lib/ghc-8.0.1/bin. GHC will then find them there when it needs them for -fllvm builds and won't look elsewhere. That's what I currently do and it works. This way, I don't have to mess with -pgmlc and -pgmlo options to ghc in the calling script.

So that might be convenient if stack setup just extracts llc and opt from the LLVM bindists and puts them right in GHC's bin/ directory.

[GordonBGood]

Over in #257 (comment), @GordonBGood said:

Even easier is just to put the required versions of the LLVM programs "llc" and "opt" in the exec folder of ghc, which for a normal HaskellPlatform Linux install is /usr/local/Haskell/ghc-8.0.1-x86_64/lib/ghc-8.0.1/bin. GHC will then find them there when it needs them for -fllvm builds and won't look elsewhere. That's what I currently do and it works. This way, I don't have to mess with -pgmlc and -pgmlo options to ghc in the calling script.

So that might be convenient if stack setup just extracts llc and opt from the LLVM bindists and puts them right in GHC's bin/ directory.

Further experimentation shows that may not be enough, although it is part of the solution. The rest is to modify the settings file in the directory pointed to by "ghc --print-libdir" (which points to the above lib dir one level up from the bindir) to give the llc and llo entries the full path to the bindir where these files were placed (which may as well be the haskell platform bindir).

[tmspzz]

Hello,

stack arm binary installed correctly form me but stack setup was complaining about missing llvm. Despite that it managed to install correctly ghc-8.0.1 and ghci.

However, when using stack build on a project, ghc would complain about missing opt and llvm 3.7.

I am using raspbian jessie on my Raspberry Pi 3 so I did:

$ sudo apt-get install llvm-3.7 
$ ln -s /usr/bin/opt-3.7 /usr/bin/opt
$ ln -s /usr/bin/llc-3.7 /usr/bin/llc

this allowed stack build to go a little further but then I encountered some errors like the following Error: selected processor does not support ARM ...

apparently when compiling ghc needs the following option ghc -opta-march=armv7-a

Unfortunately seems like ghc-options is not working correctly so doing stack build --ghc-options="-opta-march=armv7-a" does nothing.

As a workaround I deleted the symbolic link to ghc at ~/.stack/programs/arm-linux/ghc-8.0.1/bin/ghc and created a new one pointing to a script as suggested here

Finally my project build but when I executed I got greeted by hello-exe: unknown RTS option: -N

So I edited my projects cabal file and removed the rtsoptions.

Everything seems to build now. Happy hacking.

Edit: @borsboom since I got you attention with a related pull request maybe something can be done here too? Thanks

[tmspzz]

I put together this script (for raspbian) if it's of any help.

[peterbecich]

@blender thanks! Your script is helpful. Have you by any chance tried to use Intero for Emacs on the Raspberry Pi?

[varosi]

Hello! Are there plans to put ARM bindist of Stack on the site officially?

[borsboom]

@varosi: We'd certainly like to, but before Stack officially supports ARM we need a better support for different LLVM versions. Currently each GHC version needs a specific version of LLVM to be installed, and right now it's up to the user to do so (usually using their distro's package manager), but that means Stack can't work with snapshots that use different GHC versions that need different LLVM versions.

There are two ways to fix this, so contributing to one or the other would be the way to get ARM officially supported:

1. Best: Finish https://ghc.haskell.org/trac/ghc/ticket/10074, which will bundle LLVM with GHC. This had originally been targeted for ghc-8, but obviously that didn't happen. There seems to be some renewed activity on that issue in the last week, so now would be a good time to jump in!

2. Workaround: add support to Stack for managing LLVM alongside GHC in ~/.stack/programs. It would have to download and install the correct LLVM bindist. Note that I haven't had luck using the official LLVM ARM bindists because they seem to have different CPU requirements than the GHC ARM bindists, so this will probably also mean building and distributing our own LLVM bindists.

In the mean time, we are making a best effort to provide ARM builds of Stack, but they tend to lag behind the "official" builds (mainly because I'm building them in QEMU on my laptop, which takes several days).

[varosi]

Option 1 seems the best, but issue is targeted far for 8.4 if it isn't postponed even more.

[robinvd]

@borsboom Where can i find these arm binaries? I tried the bash install script but it just downloads the 32bit linux version.

[tmspzz]

@robinvd @borsboom yep, seems like https://www.stackage.org/stack/linux-arm is gone

[bogdan-manole]

I used the previous version (1.3.4)

https://github.com/commercialhaskell/stack/releases/download/v1.3.2/stack-1.3.2-linux-arm.tar.gz https://github.com/commercialhaskell/stack/releases/download/v1.3.2/stack-1.3.2-linux-arm.tar.gz.

Extract the binary (stack) and copy it to /usr/bin/

It, worked

On Mar 22, 2017, at 01:48, Tommaso Piazza notifications@github.com wrote:

@robinvd https://github.com/robinvd @borsboom https://github.com/borsboom yep, seems like https://www.stackage.org/stack/linux-arm https://www.stackage.org/stack/linux-arm is gone

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/commercialhaskell/stack/issues/2103#issuecomment-288255007, or mute the thread https://github.com/notifications/unsubscribe-auth/ADB3PybHyt9hpVD6GCAdNh9UtpiAsMnXks5roGHHgaJpZM4IX_re.

[tmspzz]

True but now, the script at https://get.haskellstack.org/ is broken :(

[borsboom]

Is there anyone out there who can build a 1.4.0 armv7 binary? My ARM building setup is borked and I hate listening to my CPU fan running at full tilt for the four days it takes to build Stack from scratch with qemu (in the unlikely event it works the first time, which is rare...)