Open nh2 opened 6 years ago
Here is a full log of me installing Arch Linux in a vagrant
VM, installing stack and Docker, and trying to use Arch's own stack
(obtained via pacman
) to run stack --docker build
.
This does not capture the problem from the issue description precisely, as here the problem is not glibc, but the fact that Arch links stack fully-dynamically (so we get the error /tmp/stack: error while loading shared libraries: libHSstack-1.7.1-7u6zix21rTfHmao1C2GDj1-ghc8.4.3.so: cannot open shared object file: No such file or directory
).
But it demonstrates the concept; a glibc mismatch would fail similarly.
Quick command overview:
vagrant init archlinux/archlinux
vagrant up # ignoring shared folder mount error due to guest additions version mismatch
vagrant ssh
sudo pacman -Syy
sudo pacman -S stack docker
systemctl start docker.service
sudo usermod -aG docker vagrant
exit
vagrant ssh
stack new stack-docker-test
cd stack-docker-test
stack docker pull
stack --docker build
Details and output:
Here is a workaround:
From https://docs.haskellstack.org/en/stable/docker_integration/
# Location of a Docker container-compatible 'stack' executable with the
# matching version. This executable must be built on linux-x86_64 and
# statically linked.
# Valid values are:
# host: use the host's executable. This is the default when the host's
# executable is known to work (e.g., from official linux-x86_64 bindist)
# download: download a compatible executable matching the host's version.
# This is the default when the host's executable is not known to work
# image: use the 'stack' executable baked into the image. The version
# must match the host's version
# /path/to/stack: path on the host's local filesystem
stack-exe: host
executable must be built on linux-x86_64 and statically linked
is a bit misleading:
If we have a stack release available for download thats compatible with the docker image the user has chosen, then download
will work fine even when the downloaded version is not statically linked.
For example, this works:
vagrant init archlinux/archlinux
vagrant up
vagrant ssh
sudo pacman -Syy
sudo pacman -S stack docker
systemctl start docker.service
sudo usermod -aG docker vagrant
exit
vagrant ssh
stack new stack-docker-test
cd stack-docker-test
stack docker pull
cat >> stack.yaml <<EOF
docker:
stack-exe: download
EOF
stack --docker build
I'll update the TODO list in the issue description to reflect that.
Nevertheless, static binaries still seem desirable, so that they work no matter what docker image the user has specified in stack.yaml.
@nh2 Can this be closed? As you know, since you did most of the work to make it happen, we're now releasing static stack
binaries again.
@borsboom I ticked off
- [x] get statically linked
stack
back in order to makestack --docker
work everywhere.
but I think we should still do the other points in the issue description (showing a warning when a dynamic stack
is re-executing itself in the docker container.
Do you want me to do that?
Before re-executing itself in a container, Stack does a compatibility check where it tries to run itself with stack --version
in a temporary container (and caches the result so this only has to be done once per exe/docker image combination). Only if that succeeds will it re-execute itself in the container (otherwise it will download a statically linked stack
and execute that), so I don't think we need any messages around that.
My suspicion is that the Received ExitFailure 127 when running
came from the compatibility check not catching the failure properly, after some changes were made to underlying process handling functions that caused them to throw different exceptions. I believe I fixed this in https://github.com/commercialhaskell/stack/commit/e65f4df3e79456ec971ae7a9c2014fa2632f08a5. Can you confirm?
In terms of the docs, I don't think
This executable must be built on linux-x86_64 and statically linked.
is quite right. The requirement is that the executable is compatible (in terms of platform and shared libraries) with the Docker image you're using. Dynamically linked is fine if the image has the right shared library versions installed. Of course, statically linked is going to be the most compatible, so that's preferred, but it's not a must.
@borsboom
My suspicion is that the
Received ExitFailure 127 when running
came from the compatibility check not catching the failure properly, after some changes were made to underlying process handling functions that caused them to throw different exceptions. I believe I fixed this in e65f4df.
Ah, that makes sense.
Can you confirm?
Maybe: I can't very easily check the version at your commit for my scenario ("downstream packagers don't seem to know that they break stack --docker
when they link stack fully dynamically"), because for my Arch repro I'd have to build the latest master of stack fully dynamically the way that Arch does, for which I probably have to update dependencies as well, which may take many hours.
I'll have a quick shot though at cherry-picking e65f4df3e79456ec971ae7a9c2014fa2632f08a5 on top of the version of stack in Arch that produced the failure.
In terms of the docs, I don't think
This executable must be built on linux-x86_64 and statically linked.
is quite right.
Yes, I agree. That's what it currently says in our docs though. With
- [ ] improve the docs saying
executable must be built on linux-x86_64 and statically linked
I didn't mean we should add the sentence executable must be built on linux-x86_64 and statically linked
to the docs (it is already there), but that we should change it to no longer say that, instead saying what you explained above.
I've opened https://github.com/commercialhaskell/stack/pull/4858 with the documentation change.
Summary
stack --docker
doesn't work reliably across Linux distributions becausestack
is no longer statically linked.Why we don't have statically linked executables at the moment
In the past, we provided statically linked stack executables. Those worked fine with
stack --docker
.We used Alpine Linux to create these stack executables. Unforunately, Alpine Linux doesn't work with GHC >= 8.2 (see https://github.com/commercialhaskell/stack/issues/2387, https://github.com/channable/vaultenv/issues/43#issuecomment-390116575, and https://github.com/alpinelinux/aports/pull/4255).
Stack 1.7 requires GHC >= 8.2. So we don't have staically linked executables for that.
We needed Alpine Linux to create the executables because it provided a full
musl
based toolchain (musl
is a replacement forglibc
, andglibc
pretty much forbids static linking)Why static linking is needed for
stack --docker
to workWhen you use
stack --docker build
, the outer stack executable is re-executed inside the docker container.If that executable is dynamically linked, and the outer
libc
is a different version than the innerlibc
, then stack will fail to run in the container.Right now, the linked-dynamicall-against-libc generic
stack
installer we provide works on many Linux distributions, and it works in docker images such asfpco/stack-build:lts-11.13
which is whatstack docker pull
pulls. But it doesn't work on all distributions, and for those where it doesn't,stack --docker
support is completely broken right now. Further, as soon as some distro (like Arch) will advance to an incompatible libc earlier than other distros (e.g. Ubuntu), the generic installer will also stop working nicely.How it fails
It doesn't fail with a nice error message, leaving the user in doubt:
Proposed solution
We need to
stack
back in order to makestack --docker
work everywhere.stack
doesn't really supportstack --docker
.stack
execution so that users don't get confused by the nonexisting error message (see above)stack --docker
when they link stack fully dynamically (see my next comment for how Arch Linux gets it wrong)stack --version
inside Docker doesn't work, see https://github.com/commercialhaskell/stack/issues/4087#issuecomment-500622630executable must be built on linux-x86_64 and statically linked
(see comment below)4858
Scope
This ticket is specifically for
stack --docker
problems; I'll file a separate issue (https://github.com/commercialhaskell/stack/issues/4088) about getting statically linkedstack
back, as that is desiable also for other reasons than Docker.