Git dependencies fail when they have some kinds of relative symlinks

[michaelpj]

General summary/comments (optional)

We have are trying to use https://github.com/input-output-hk/plutus as a git dependency. New versions of Stack fail with a tarball error.

This is an offshoot from https://github.com/commercialhaskell/stack/issues/4913 for the case that I think is definitely a bug.

Steps to reproduce

Add

extra-deps:
- git: https://github.com/input-output-hk/plutus
  commit: dbc2646112a87c710a8dd6f80b63458ca37a06c4

to your stack.yaml and stack build.

Expected

It works.

Actual

Unsupported tarball from https://github.com/input-output-hk/plutus/archive/dbc2646112a87c710a8dd6f80b63458ca37a06c4.tar.gz: Symbolic link dest not found from plutus-dbc2646112a87c710a8dd6f80b63458ca37a06c4/plutus-book/src/Game/Guess.lhs to ../../doc/game/guess.adoc, looking for plutus-dbc2646112a87c710a8dd6f80b63458ca37a06c4/plutus-book/src/../doc/game/guess.adoc.

Note that this link is not broken.

The link does have two levels of .. in it, and if we look at what is actually being checked, the final path appears to still have one of the ..s in it, possibly this is related.

Stack version

2.1.1.1 x86_64 hpack-0.31.2

Method of installation

Nix.

[judah]

This bug is also preventing using https://github.com/google/proto-lens from being used as a git dependency, which used to work fine in stack-1.*. (The proto-lens repo contains a couple of symlinks into its submodule of the google/protobuf repo.)

$ mkdir temp
$ cd temp
$ stack init --force
$ cat > stack.yaml <<EOF
resolver: lts-13.23
extra-deps:
- git: https://github.com/google/proto-lens
  commit: b5802fcf848ac04386f48cdb079fe8c2710e2d40
EOF
$ stack build proto-lens
Cloning b5802fcf848ac04386f48cdb079fe8c2710e2d40 from https://github.com/google/proto-lens
Unsupported tarball from /private/var/folders/dk/xxrqcv1n5_5_gzkyzc94r5_80000gn/T/with-repo-archive85629/foo.tar: Symbolic link dest not found from proto-lens-protobuf-types/proto-src to ../google/protobuf/src/, looking for google/protobuf/src.
This may indicate that the source is a git archive which uses git-annex.
See https://github.com/commercialhaskell/stack/issues/4579 for further information.

[hsyl20]

This bug also breaks building packages from a GHC repo:

extra-deps:
  - git: https://github.com/hsyl20/ghc.git
    commit: 71b7458ba662263ed7e31f61876936876f41aa15
    subdirs:
       - libraries/Cabal/Cabal

Cloning 71b7458ba662263ed7e31f61876936876f41aa15 from https://github.com/hsyl20/ghc.git
Unsupported tarball from /tmp/with-repo-archive24598/foo.tar: Symbolic link dest not found from nofib/imaginary/bernouilli/NofibUtils.hs to ../../common/NofibUtils.hs, looking for nofib/imaginary/../common/NofibUtils.hs.
This may indicate that the source is a git archive which uses git-annex.
See https://github.com/commercialhaskell/stack/issues/4579 for further information.

[vlatkoB]

I'm testing upgrade to stack 2.3.1 and encountered this error.

It is a repo with several sub-repos. The main repo has a directory with some data, and all sub-repo's "test" directories contain a link to that main test data storage. So, it is actually a valid, rebuildable project, as all the data will always be present. Link is relative and its target is inside the repo. I'm lucky as it is my repo, so easy to fix.

However, I'm more concerned with what would happen when a package is not under my control. A decision to not solving this, for whatever reason, justified or not, would leave me in a situation where I'd have to choose between (a) use stack 1.*, (b) switch to cabal or (c) clone all such "bad" repos locally. I personally do not like any of the solutions.

I agree with @snoyberg's comment that we should strive for using only "correctly defined Haskell package" so that our project can be successfully rebuilt in the future, but that is not always possible.

Maybe the solution would be to add a switch to stack.yaml, something like:

ignore-symlink-error: true

so that we, by ourselves, can make a decision of whether we should use such package(s) in our projects or not.

Can some form of this solution be accepted?

[michaelpj]

I agree with @snoyberg's comment that we should strive for using only "correctly defined Haskell package" so that our project can be successfully rebuilt in the future, but that is not always possible.

To be clear, I think this one is just a bug. The case with broken symlinks is arguably invalid in some sense.

[gergoerdi]

I've found that this breaks packages even if the symlinks are files which are completely inconsequential. For example, I have a repo where Stack complains about the symlink docs/general/LICENSE to ../../LICENSE, but docs is not even listed as one of the subdirs in my Stack configuration for this dependency.

[yfyf]

Any tips on how to work-around this at least temporarily? It's of course possible to fork upstream repositories and manually eliminate symlinks, but that's hardly ideal.

One more project which is impossible to use because of this is ADL: https://github.com/timbod7/adl

It contains a (valid) symlink from haskell/compiler/lib/adl/sys to adl/stdlib/sys/, which causes stack to choke:

Unsupported tarball from /tmp/with-repo-archive13147/foo.tar:
Symbolic link dest not found from
haskell/compiler/lib/adl/sys to ../../../../adl/stdlib/sys, looking for
haskell/compiler/lib/../adl/stdlib/sys.

it's odd because it seems that instead of following ../../../../<..>, stack is only using ../../<..>.