Open gergoerdi opened 3 years ago
@gergoerdi would you mind creating a simple project to reproduce this? In that case I could take a look into what's going on here. Thanks for reporting this anyway.
Sure, I will try to make a small self-contained way of reproducing it (hopefully by tomorrow). In the meantime, https://github.com/gergoerdi/clash-issue-1536/tree/memory-map-th should be the full code, including the stack.yaml
and clash-HEAD.yaml
files.
Minimizing preproduction will help a lot, thank you.
OK I have uploaded a fully self-contained, reproducing repo to https://github.com/gergoerdi/stack-issue-5473
stack build
cp clash-HEAD2.yaml clash-HEAD.yaml
stack build
Yeah, great, reproduced it locally
@gergoerdi it looks like you've found a hole in Pantry design :)
the problem is that clash-ghc
directory has the same contents for both commits but its dependencies differ. As contents doesn't change (and the deps have the same versions) Stack reuses already built version which results in the failure you're getting.
Not sure yet what to do about this but there is a workaround which doesn't require nuking ~/.stack
- use stack exec -- ghc-pkg unregister clash-ghc
So wouldn't a straightforward fix here be to include, in the identifier of a dependency, a hash of (the hash of) all its dependencies?
Nix-like way (when all inputs in theory determine the resulting output) makes sense but that's not quite how Stack works. In principle when checking already compiled libraries deps are also taken into account but it looks like in this case for some reason package identifiers (from ghc-pkg) for cabal-lib
and cabal-prelude
stay the same. So probably there's some lurking bug - I'll try to dig deeper into it.
@qrilka Thank you for looking into this. I'm hitting this bug as well, and I reported it as #5501 (before I found this issue).
I've experienced this a couple of times now, and I very often update git dependencies, so it happens fairly rarely. I was quite happy that I was able to reproduce it in a Docker container (see #5501). This reproduction may provide an extra data point in figuring out what's going on. But I'm happy to see that this issue has a more minimal reproduction.
For both reproductions, the bug is triggered when a git dependency commit hash is changed to a version where an export is added to an existing module. This is the diff between the dependency versions that causes the error for this issue's reproduction: https://github.com/gergoerdi/clash-compiler/compare/d83b5cc58816654c84537b0b049b32ce899e179e...62411778020b09433744d267acd5aa3deb85b04f, and this is the diff between the dependency version that causes the error for my reproduction: https://github.com/runeksvendsen/bellman-ford/compare/a089bfe9d449afbc1907ded5223bba23a5de9e8a...45bfb65d16a635fadb4097cdb8f3ec9281784083. In both cases the linker error refers to the added exports (modules Clash.Annotations.Primitive
and Data.Graph.Digraph
respectively).
@runeksvendsen as I've written above it looks like Stack doesn't see the update and reusing old library clearly causes problems. Unfortunately I didn't have time yet to look more into it.
After coming back to this ticket now I see that stack exec -- ghc-pkg unregister clash-ghc
I advised above doesn't work - after unregistering Stack picks up incorrect version as precompiled. Will dig more into why that happens.
@gergoerdi I had another look into the details of this and it appears that for some reason Cabal
produces the same package-id for 2 different source trees (and having the same package version in cabal file) and it seems like it doesn't contradict the docs in GHC sources - https://gitlab.haskell.org/ghc/ghc/-/blob/master/compiler/GHC/Unit.hs#L42 but at the same time Cabal's definition is slightly different (UnitId is basically ComponentId if we ignore Backback for which Stack doesn't have support yet) - https://hackage.haskell.org/package/Cabal-3.4.0.0/docs/Distribution-Types-ComponentId.html#t:ComponentId
Stack by itself relies on package ids reported by Cabal so I'm not sure what could be a fix for this scenario.
@snoyberg do you know what definition of package id made Stack rely on uniqueness of package ids?
I have no idea. Package ID is one of those concepts that seems to have changed significantly over time, and I have no idea what we can or cannot rely upon in the Cabal or GHC worlds anymore.
Looks like I just ran into this as well, ironically while building a repro for a different bug in a different library: https://github.com/chreekat/r2-repro/
I say "Uncomment to try the fix", but that doesn't work because I just get a bunch of linker errors. :P
On Stack 2.5.1, I am using the following resolver file:
After a full
stack build
in my project, I then want to change the version of Clash used, so I change the git commit hash to62411778020b09433744d267acd5aa3deb85b04f
. Subsequently,stack build
fails at link time with an error message referencing a symbol that has been removed from Clash between the two Git commits:I tried removing
.stack-work
and*/.stack-work
, but the problem persists. It seems like I would need to nuke~/.stack/
, just like in the bad old Cabal days.