search

commercialhaskell / stack

The Haskell Tool Stack
http://haskellstack.org
BSD 3-Clause "New" or "Revised" License
4k stars 843 forks source link

"Right Nothing" printed to the console during setup #2175

Closed silky closed 8 years ago

silky commented 8 years ago

with

>stack --version
Version 1.1.3, Git revision 8f7e62fbe3620a4b0ae5eb833cebfa50820f5050 x86_64 hpack-0.14.0

when i attempt a setup i see:

⚡ 05:37 PM noon ∈ ~>stack --resolver nightly-2016-05-13 setup
Run from outside a project, using implicit global project config
Using resolver: nightly-2016-05-13 specified on command line
Downloaded nightly-2016-05-13 build plan.    
Did not find .cabal file for servant-yaml-0.1.0.0 with Git SHA of d7c8a8eaf9e7c215433f8f39daac7fc6cf116325
Right Nothing
Did not find .cabal file for reducers-3.12.1 with Git SHA of bcebf11be1770ac480d9d0ee6238044f143a57f8
Right Nothing
Did not find .cabal file for math-functions-0.1.7.0 with Git SHA of 5870cda6259e8a838c8cb22043b9d3cbf55d9a8e
Right Nothing
Did not find .cabal file for hw-prim-0.0.0.10 with Git SHA of 3658e799df0b7fceb045899b7a3d686b960b5d01
Right Nothing
Did not find .cabal file for hspec-megaparsec-0.1.1 with Git SHA of 116c887f005a6e599dc9b6ce5627f1b69782f44b
Right Nothing
Did not find .cabal file for dixi-0.6.9.1 with Git SHA of 17cba908e23588d9e5b67541f53afd8590c5c30f
Right Nothing
Did not find .cabal file for cron-0.4.0 with Git SHA of d485877cb0e43ee1b8fcdaa0a7224acbd753186c
Right Nothing
stack will use a locally installed GHC
For more information on paths, see 'stack path' and 'stack exec env'
To use this GHC and packages outside of a project, consider using:
stack ghc, stack ghci, stack runghc, or stack exec

it's mildly unfortunate that Right Nothing is printed here.

introduced a few weeks ago by https://github.com/commercialhaskell/stack/commit/f483e820d23707ec4ddd3fcdf38385e36f769203#diff-2068556e565acc62de252ac25ed550c4R288

mgsloan commented 8 years ago

Good point, fixed!

mgsloan commented 8 years ago

Then again, perhaps the root issue is not addressed - why are these SHAs missing?

silky commented 8 years ago

i'm not too sure about that one.

let me know if you want more information from me.

mgsloan commented 8 years ago

Oh yeah, to be clear that wasn't a request for more information. More like "this needs further investigation" :)

The failure case here is just a fallback to the old behavior where cabal metadata wasn't fixed to a particular revision. So thankfully this shouldn't be very problematic in practice. Good to fix, though!

schell commented 8 years ago

I seem to be running into a similar issue, though the output is a bit different - notably the "Left ..." result:

> stack --version
Version 1.1.2, Git revision cebe10e845fed4420b6224d97dcabf20477bbd4b (3646 commits) x86_64 hpack-0.14.0

> stack build
Downloading nightly-2016-05-20 build plan ...                                             Downloaded nightly-2016-05-20 build plan.
Updating package index Hackage (mirrored at https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz) ...                                                                                                                Downloading package index from https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz
Updating package index Hackage (mirrored at https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz) ...
                                                                                                                Populating index cache ...                          Populated index cache.
Did not find .cabal file for zot-0.0.3 with Git SHA of f9092bae03d0044d2b4e3f33609c9544e8c46b19
Left /root/.stack/indices/Hackage/git-update/all-cabal-hashes/.git/objects/pack: listDirectory: does not exist (No such file or directory)
Did not find .cabal file for zlib-lens-0.1.2.1 with Git SHA of b5797267b30c16afcccbd42fa0ce4b9fa58a7ca9
Left /root/.stack/indices/Hackage/git-update/all-cabal-hashes/.git/objects/pack: listDirectory: does not exist (No such file or directory)
Did not find .cabal file for zlib-enum-0.2.3.1 with Git SHA of 5bb46141c62ea8a473a2cfac9293a018c804189a
...

I'm using stack on bitbucket's new "pipelines" feature which executes stack from the haskell:7.10.3 docker image.

schell commented 8 years ago

The result of using lts-5.2 is similar but different:

stack build
Downloading lts-5.2 build plan ...                                  Downloaded lts-5.2 build plan.
Updating package index Hackage (mirrored at https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz) ...                                                                                                                Downloading package index from https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz
Updating package index Hackage (mirrored at https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz) ...                                                                                                                Populating index cache ...                          Populated index cache.
/opt/atlassian/bitbucketci/agent/build/.stack-work/downloaded/NC9VHrq5ky0p/: getDirectoryContents: does not exist (No such file or directory)
simonmichael commented 8 years ago

Another data point: I saw a bunch of these messages after upgrading stack (from 1.1.0 x86_64 hpack-0.13.0 to 1.1.2 x86_64 hpack-0.14.0) and running stack install. I did stack clean and they went away.

sjakobi commented 8 years ago

I see the Did not find .cabal file messages every time I build a project with a different (dev) version of stack.

Edit: I actually see these messages also for other commands, like stack path.

DanBurton commented 8 years ago

I've seen these messages lately as well, seemingly related to using the latest stack in environments that don't have a ~/.stack directory yet. For example:

(Modified instructions from trying to reproduce this bug)

docker run -it danburton/stackage-build-server:2016-07-11
cd ~
stack unpack gi-gtk-3.0.4
cd gi-gtk-3.0.4
stack init --resolver nightly-2016-06-08
Right Nothing
Did not find .cabal file for text-show-3.2.2 with Git SHA of dd7d87942bbe93e92db8be47a37172ac12be63fa
Right Nothing
Did not find .cabal file for socks-0.5.5 with Git SHA of a4fe3dade53c652f5ef4d04218d17a989e8414e1
Right Nothing
Did not find .cabal file for servant-yaml-0.1.0.0 with Git SHA of 2ef4721a30ef0c982f26f67688cfa09a307a62fc
Right Nothing
Did not find .cabal file for servant-swagger-1.1 with Git SHA of a35531ce5ae19faf483b4a7c5ceb73b67b0612e6
Right Nothing
Did not find .cabal file for servant-lucid-0.7.1 with Git SHA of cf726c02fbee0a185c0b861fcd8cbf1863c7b414

Last month when I ran this with snoyberg/stackage:nightly, these things were not printed to the console.

yacinehmito commented 8 years ago

I still experience the same bug.

rainbyte commented 8 years ago

I'm experiencing the same issue inside a docker container:

When I run the cmd

stack install --stack-root $PWD/.stack-root/ --work-dir .stack-work-docker/ --local-bin-path artifacts/

It shows this

Downloading lts-6.10 build plan ...
Downloaded lts-6.10 build plan.
Updating package index Hackage (mirrored at https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz) ...
Downloading package index from https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz
Populating index cache ...
Populated index cache.
Did not find .cabal file for zot-0.0.3 with Git SHA of f9092bae03d0044d2b4e3f33609c9544e8c46b19
Left /src/.stack-root/indices/Hackage/git-update/all-cabal-hashes/.git/objects/pack: listDirectory: does not exist (No such file or directory)
Did not find .cabal file for zlib-lens-0.1.2.1 with Git SHA of b5797267b30c16afcccbd42fa0ce4b9fa58a7ca9
Left /src/.stack-root/indices/Hackage/git-update/all-cabal-hashes/.git/objects/pack: listDirectory: does not exist (No such file or directory)

And after various similar lines, I get

Left /src/.stack-root/indices/Hackage/git-update/all-cabal-hashes/.git/objects/pack: listDirectory: does not exist (No such file or directory)
Did not find .cabal file for AC-Vector-2.3.2 with Git SHA of 32d97a60969fe6912b33597e7399c407b5f2e39a
Left /src/.stack-root/indices/Hackage/git-update/all-cabal-hashes/.git/objects/pack: listDirectory: does not exist (No such file or directory)
/src/.stack-work-docker/downloaded/9rVosC0PeQVv/: getDirectoryContents: does not exist (No such file or directory)
lehtoj commented 8 years ago

I am new to Haskell and today I attempted to install Haskell for Windows using Stack. However, I get similar errors as in this issue. Number of errors is too long to be copied here, so I provide you with a link instead.

While I have clearly run stack install outside a project, I get same errors on the first time I run stack setup.

Version 1.1.2, Git revision c6dac65e3174dea79df54ce6d56f3e98bc060ecc (3647 commits) x86_64 hpack-0.14.0 Downloaded from https://www.stackage.org/stack/windows-x86_64-installer

mgsloan commented 8 years ago

I was working on getting https://github.com/lukexi/rumpus built on windows yesterday, and also ran into this. I think it just happens when you don't have git. We need to identify when we shouldn't expect to find any SHAs for snapshot packages.

Blaisorblade commented 8 years ago

What I got from looking at the code and hadn't got from reading here: master won't print Right Nothing or Left ..., but that fix is not yet released. The "Did not find .cabal file" message is still there and not fully explained.

lehtoj commented 8 years ago

Another observation is that when I use Stack to download contents of stack_root, mirror is https://s3.amazonaws.com/hackage.fpcomplete.com, size ends up being 262MB. But when someone else in another country uses Stack, download mirror is Github and contents end up being 451MB. Is the size difference explained by missing cabal/SHA files, or is there something else amiss?

Blaisorblade commented 8 years ago

@KeeperB5 interesting catch! Do you have the two actual URLs for one to look at? If mirrors aren't mirrors, debugging this issue isn't going to be fun.

lehtoj commented 8 years ago

Well, my index is fetched from https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz, but I believe the GitHub mirror had https://github.com/commercialhaskell/all-cabal-hashes.

How does Stack pick a mirror and do we have any way to manually select a mirror?

sjakobi commented 8 years ago

I've had the "Did not find .cabal file" message pop up in the middle of a build while trying to reproduce https://github.com/commercialhaskell/stack/issues/2422:

~/t/postgres-tmp> stack build --trace
...
attoparsec-0.13.0.2: copy/register
Did not find .cabal file for aeson-0.11.2.0 with Git SHA of 2057e715a488cd9d2456fdb1c6c4b0d6142c539e
aeson-0.11.2.0: configure
aeson-0.11.2.0: build
aeson-0.11.2.0: copy/register
...

I can't rule out that I had just started using another stack version in another thread.

@mgsloan:

I was working on getting https://github.com/lukexi/rumpus built on windows yesterday, and also ran into this. I think it just happens when you don't have git.

Are you saying you got the "Right Nothing" lines with stack-HEAD? Or do you only see the "Did not find .cabal file" lines?

The "Did not find .cabal file" lines certainly appear on Linux with git installed.

sjakobi commented 8 years ago

BTW I currently think this issue – while not a dangerous bug – is sufficiently widespread and irritating that it should be fixed before a new release.

Blaisorblade commented 8 years ago

@sjakobi Is this issue in part expected when mixing stack versions? I see more messages when I ran stack exec -- stack foo, where the outer stack is 1.1.2. I'd really wish for RTFM as an answer, but I haven't seen the manual I'd need and get too little of the architecture ;-).

simonmichael commented 8 years ago

I don't think this is windows-specific. I've seen it many times and have been using only gnu/linux and mac.

Blaisorblade commented 8 years ago

Windows-label removed.

sjakobi commented 8 years ago

Is this issue in part expected when mixing stack versions?

I can reproduce it reliably by switching between stack-1.1.2 and stack-HEAD between rebuilds:

~/t/postgres-tmp> /usr/bin/stack build -v
Version 1.1.2, Git revision cebe10e845fed4420b6224d97dcabf20477bbd4b (3646 commits) x86_64 hpack-0.14.0
...
2016-08-09 15:57:00.072316: [debug] Checking for project config at: /home/simon/tmp/postgres-tmp/stack.yaml @(stack_K1e6VSKnzs1HNYmTJGAgTQ:Stack.Config src/Stack/Config.hs:811:9)
2016-08-09 15:57:02.076435: [debug] Trying to decode /home/simon/.stack/indices/Hackage/00-index.cache @(stack_K1e6VSKnzs1HNYmTJGAgTQ:Data.Binary.VersionTagged src/Data/Binary/VersionTagged.hs:55:5)
2016-08-09 15:57:02.076750: [debug] Failure decoding /home/simon/.stack/indices/Hackage/00-index.cache @(stack_K1e6VSKnzs1HNYmTJGAgTQ:Data.Binary.VersionTagged src/Data/Binary/VersionTagged.hs:59:13)
Populated index cache.    
2016-08-09 15:57:09.196985: [warn] Did not find .cabal file for servant-blaze-0.7.1 with Git SHA of a611898708511c515ae69860506efb2da87bbfc9
Right Nothing @(stack_K1e6VSKnzs1HNYmTJGAgTQ:Stack.Fetch src/Stack/Fetch.hs:289:17)
2016-08-09 15:57:11.214682: [warn] Did not find .cabal file for diagrams-lib-1.3.1.3 with Git SHA of e47d8a2b5f237ed5da84b35290640587d43bbccf
Right Nothing @(stack_K1e6VSKnzs1HNYmTJGAgTQ:Stack.Fetch src/Stack/Fetch.hs:289:17)
2016-08-09 15:57:12.216469: [warn] Did not find .cabal file for aeson-0.11.2.0 with Git SHA of 2057e715a488cd9d2456fdb1c6c4b0d6142c539e
Right Nothing @(stack_K1e6VSKnzs1HNYmTJGAgTQ:Stack.Fetch src/Stack/Fetch.hs:289:17)
...
~/t/postgres-tmp> stack build -v
Version 1.1.3, Git revision 49d96c9cbbd34ab33af02f3957b33fdd9575a7f8 (dirty) (3945 commits) x86_64 hpack-0.14.0
...
2016-08-09 15:57:20.471192: [debug] Trying to decode /home/simon/.stack/indices/Hackage/00-index.cache @(stack_BSx6FTAIDJFC2EPDshToqJ:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:68:5)
2016-08-09 15:57:20.480004: [debug] Error while decoding /home/simon/.stack/indices/Hackage/00-index.cache: PeekException {peekExBytesFromEnd = 21369809, peekExMessage = "Attempted to read too many bytes for Data.ByteString.ByteString. Needed -5665116520842264576, but only 21369809 remain."} (this might not be an error, when switching between stack versions) @(stack_BSx6FTAIDJFC2EPDshToqJ:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:96:18)
2016-08-09 15:57:20.480258: [debug] Failure decoding /home/simon/.stack/indices/Hackage/00-index.cache @(stack_BSx6FTAIDJFC2EPDshToqJ:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:75:13)
Populated index cache.    
2016-08-09 15:57:29.767894: [debug] Encoding /home/simon/.stack/indices/Hackage/00-index.cache @(stack_BSx6FTAIDJFC2EPDshToqJ:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:51:5)
2016-08-09 15:57:30.705207: [debug] Finished writing /home/simon/.stack/indices/Hackage/00-index.cache @(stack_BSx6FTAIDJFC2EPDshToqJ:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:55:5)
2016-08-09 15:57:31.552932: [warn] Did not find .cabal file for servant-blaze-0.7.1 with Git SHA of a611898708511c515ae69860506efb2da87bbfc9 @(stack_BSx6FTAIDJFC2EPDshToqJ:Stack.Fetch src/Stack/Fetch.hs:295:17)
2016-08-09 15:57:31.553089: [debug] Right Nothing @(stack_BSx6FTAIDJFC2EPDshToqJ:Stack.Fetch src/Stack/Fetch.hs:301:17)
2016-08-09 15:57:33.686882: [warn] Did not find .cabal file for diagrams-lib-1.3.1.3 with Git SHA of e47d8a2b5f237ed5da84b35290640587d43bbccf @(stack_BSx6FTAIDJFC2EPDshToqJ:Stack.Fetch src/Stack/Fetch.hs:295:17)
2016-08-09 15:57:33.687049: [debug] Right Nothing @(stack_BSx6FTAIDJFC2EPDshToqJ:Stack.Fetch src/Stack/Fetch.hs:301:17)
2016-08-09 15:57:34.721612: [warn] Did not find .cabal file for aeson-0.11.2.0 with Git SHA of 2057e715a488cd9d2456fdb1c6c4b0d6142c539e @(stack_BSx6FTAIDJFC2EPDshToqJ:Stack.Fetch src/Stack/Fetch.hs:295:17)
2016-08-09 15:57:34.721781: [debug] Right Nothing @(stack_BSx6FTAIDJFC2EPDshToqJ:Stack.Fetch src/Stack/Fetch.hs:301:17)
...

Deleting the build plan cache and and the index cache also triggers it reliably:

~/t/postgres-tmp> rm /home/simon/.stack/build-plan-cache/x86_64-linux/nightly-2016-07-30.cache
~/t/postgres-tmp> rm /home/simon/.stack/indices/Hackage/00-index.cache
~/t/postgres-tmp> stack build -v
Version 1.1.3, Git revision 49d96c9cbbd34ab33af02f3957b33fdd9575a7f8 (dirty) (3945 commits) x86_64 hpack-0.14.0
...
2016-08-09 16:06:02.438975: [debug] Stack was not built with libgmp4 @(stack_BSx6FTAIDJFC2EPDshToqJ:Stack.Config src/Stack/Config.hs:338:14)
2016-08-09 16:06:02.439148: [debug] Trying to decode /home/simon/.stack/build-plan-cache/x86_64-linux/nightly-2016-07-30.cache @(stack_BSx6FTAIDJFC2EPDshToqJ:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:68:5)
2016-08-09 16:06:02.439289: [debug] Exception ignored when attempting to load /home/simon/.stack/build-plan-cache/x86_64-linux/nightly-2016-07-30.cache: /home/simon/.stack/build-plan-cache/x86_64-linux/nightly-2016-07-30.cache: openBinaryFile: does not exist (No such file or directory) @(stack_BSx6FTAIDJFC2EPDshToqJ:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:86:9)
2016-08-09 16:06:02.439415: [debug] Failure decoding /home/simon/.stack/build-plan-cache/x86_64-linux/nightly-2016-07-30.cache @(stack_BSx6FTAIDJFC2EPDshToqJ:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:75:13)
2016-08-09 16:06:02.439699: [debug] Decoding build plan from: /home/simon/.stack/build-plan/nightly-2016-07-30.yaml @(stack_BSx6FTAIDJFC2EPDshToqJ:Stack.BuildPlan src/Stack/BuildPlan.hs:499:5)
2016-08-09 16:06:04.474490: [debug] Trying to decode /home/simon/.stack/indices/Hackage/00-index.cache @(stack_BSx6FTAIDJFC2EPDshToqJ:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:68:5)
2016-08-09 16:06:04.474691: [debug] Exception ignored when attempting to load /home/simon/.stack/indices/Hackage/00-index.cache: /home/simon/.stack/indices/Hackage/00-index.cache: openBinaryFile: does not exist (No such file or directory) @(stack_BSx6FTAIDJFC2EPDshToqJ:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:86:9)
2016-08-09 16:06:04.474906: [debug] Failure decoding /home/simon/.stack/indices/Hackage/00-index.cache @(stack_BSx6FTAIDJFC2EPDshToqJ:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:75:13)
Populated index cache.    
2016-08-09 16:06:14.096936: [debug] Encoding /home/simon/.stack/indices/Hackage/00-index.cache @(stack_BSx6FTAIDJFC2EPDshToqJ:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:51:5)
2016-08-09 16:06:15.267244: [debug] Finished writing /home/simon/.stack/indices/Hackage/00-index.cache @(stack_BSx6FTAIDJFC2EPDshToqJ:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:55:5)
2016-08-09 16:06:16.121019: [warn] Did not find .cabal file for servant-blaze-0.7.1 with Git SHA of a611898708511c515ae69860506efb2da87bbfc9 @(stack_BSx6FTAIDJFC2EPDshToqJ:Stack.Fetch src/Stack/Fetch.hs:295:17)
2016-08-09 16:06:16.121174: [debug] Right Nothing @(stack_BSx6FTAIDJFC2EPDshToqJ:Stack.Fetch src/Stack/Fetch.hs:301:17)
2016-08-09 16:06:18.133818: [warn] Did not find .cabal file for diagrams-lib-1.3.1.3 with Git SHA of e47d8a2b5f237ed5da84b35290640587d43bbccf @(stack_BSx6FTAIDJFC2EPDshToqJ:Stack.Fetch src/Stack/Fetch.hs:295:17)
2016-08-09 16:06:18.133983: [debug] Right Nothing @(stack_BSx6FTAIDJFC2EPDshToqJ:Stack.Fetch src/Stack/Fetch.hs:301:17)
2016-08-09 16:06:19.132195: [warn] Did not find .cabal file for aeson-0.11.2.0 with Git SHA of 2057e715a488cd9d2456fdb1c6c4b0d6142c539e @(stack_BSx6FTAIDJFC2EPDshToqJ:Stack.Fetch src/Stack/Fetch.hs:295:17)
2016-08-09 16:06:19.132364: [debug] Right Nothing @(stack_BSx6FTAIDJFC2EPDshToqJ:Stack.Fetch src/Stack/Fetch.hs:301:17)
...
Blaisorblade commented 8 years ago

I've figured out something, though on one example. First, the issue with Left path from @schell appears something else and should be filed as a separate issue — the repository there is probably corrupted somehow. I'll focus on the rest here.

Regarding the architecture, I've got some basics from docs at https://github.com/commercialhaskell/all-cabal-hashes and sources. Basically stack expects not only to find cabal files from there, but also to find them with specific hashes. The files are there, but the hashes do not exist in the repo as checked out. However, they do exist if you check out the full repo without --depth 1, and are variants of the files that are actually there. If the hashes come (for instance) from some snapshot package set (I don't know yet), this suggests an architectural conflict between "assuming the Git repo is a persistent data structure and old hashes stay available" and "fetch only the latest version, making the repo ephemeral".

There seems also to be also something else—stack looks for SHA for updated cabal files, but they're not in the main history of the repo (not sure where they're linked in). This appears a potential issue in the generation of all-cabal-hashes.

Consider the first message in this issue.

Did not find .cabal file for servant-yaml-0.1.0.0 with Git SHA of d7c8a8eaf9e7c215433f8f39daac7fc6cf116325

The file is there:

https://github.com/commercialhaskell/all-cabal-hashes/blob/hackage/servant-yaml/0.1.0.0/servant-yaml.json but has hash 2ba4ac33ef38ad300eae27a6474612f7f43c052c. The version that stack is looking for has x-revision: 6 and wider bounds. The version on Hackage has yet wider bounds AFAICS for base and servant (I can't compare the testsuite bounds though). http://hackage.haskell.org/package/servant-yaml

$ diff -u -i <(git show 2ba4ac33ef38ad300eae27a6474612f7f43c052c|dos2unix) <(git show d7c8a8eaf9e7c215433f8f39daac7fc6cf116325|dos2unix)
--- /dev/fd/63  2016-08-09 16:09:03.000000000 +0200
+++ /dev/fd/62  2016-08-09 16:09:03.000000000 +0200
@@ -4,6 +4,7 @@

 name:           servant-yaml
 version:        0.1.0.0
+x-revision: 6
 synopsis:       Servant support for yaml
 description:    Servant support for yaml
 category:       Web
@@ -32,7 +33,7 @@
       base        >=4.7      && <4.9
     , bytestring  >=0.10.4.0 && <0.11
     , http-media  >=0.6.2    && <0.7
-    , servant     >=0.4.4.5  && <0.5
+    , servant     >=0.4.4.5  && <0.8
     , yaml        >=0.8.12   && <0.9
   exposed-modules:
       Servant.Yaml
@@ -48,12 +49,12 @@
       base        >=4.7      && <4.9
     , bytestring  >=0.10.4.0 && <0.11
     , http-media  >=0.6.2    && <0.7
-    , servant     >=0.4.4.5  && <0.5
+    , servant     >=0.4.4.5  && <0.7
     , yaml        >=0.8.12   && <0.9
     , servant-yaml
-    , servant-server >=0.4.4.5  && <0.5
-    , base-compat    >=0.6.0    && <0.9
-    , aeson          >=0.8.0.2  && <0.11
-    , wai            >=3.0.3.0  && <3.1
-    , warp           >=3.0.13.1 && <3.2
+    , servant-server >=0.4.4.5  && <0.8
+    , base-compat    >=0.6.0    && <0.10
+    , aeson          >=0.8.0.2  && <0.12
+    , wai            >=3.0.3.0  && <3.3
+    , warp           >=3.0.13.1 && <3.3
   default-language: Haskell2010
Blaisorblade commented 8 years ago

To be sure: is x-revision a marker for updates of Cabal files on Hackage directly? I've heard about the feature and I assume this is what's going on, but I'm not 100% positive.

EDIT: yes, there are references in https://github.com/haskell/hackage-server/issues/316, which also explains why I needed dos2unix to get the above diff.

Blaisorblade commented 8 years ago

The repo is generated by https://github.com/commercialhaskell/all-cabal-hashes/blob/hackage/update.sh https://github.com/commercialhaskell/all-cabal-hashes-tool/blob/master/all-cabal-hashes-tool.hs. I'm also hunting down the commit mentioning d7c8a8eaf9e7c215433f8f39daac7fc6cf116325 using the Perl script in this StackOverflow answer, but this will take a while.

sjakobi commented 8 years ago

Great work, @Blaisorblade! :+1:

Here's an older issue pointing out that we need to handle hackage revisions: https://github.com/commercialhaskell/stack/issues/2217

Blaisorblade commented 8 years ago

@sjakobi Thanks! New findings:

  1. the repo history is fully linear and there are no force-pushes in sight, and the repo does store newer hackage revisions. But I guess the source of the problematic hashes is not updated in sync with hackage. I'm tempted to say, the hash source must either be in the same repo, or kept in sync with it closely.
  2. the JSON files (with checksums) aren't updated. Hence I suppose the tarballs can't contain updated cabal files and pass verification. However, stack unpack produces up-to-date cabal files, so maybe you already handle this somehow (by replacing the cabal files I guess?).
  3. I'm guessing the fallback code patches up the discrepancy and stores the result in the cache; what I know is, switching version triggers the message right after failing to decode the cache.

History of cabal files within all-cabal-hashes (points 1, 2)

Inside a fresh checkout of all-cabal-hashes (not stack's one with --depth 1), we can see all the revisions (for the first example) with

git log --stat origin/hackage -- servant-yaml/0.1.0.0/servant-yaml.cabal servant-yaml/0.1.0.0/servant-yaml.json

Why switching versions triggers these messages (point 3)

The message 

2016-08-09 21:54:32.627326: [debug] Failure decoding /Users/pgiarrusso/.stack/indices/Hackage/00-index.cache @(stack_K6bD2zvmjMRCxJO1RtHorG:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:75:13)
Populated index cache.
2016-08-09 21:54:36.495092: [debug] Encoding /Users/pgiarrusso/.stack/indices/Hackage/00-index.cache @(stack_K6bD2zvmjMRCxJO1RtHorG:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:51:5)
2016-08-09 21:54:36.676968: [debug] Finished writing /Users/pgiarrusso/.stack/indices/Hackage/00-index.cache @(stack_K6bD2zvmjMRCxJO1RtHorG:Data.Store.VersionTagged src/Data/Store/VersionTagged.hs:55:5)
2016-08-09 21:54:36.863792: [warn] Did not find .cabal file for yesod-core-1.4.20.2 with Git SHA of 5350ec333aa9ed6ae24e46142e33d2f851ace553 @(stack_K6bD2zvmjMRCxJO1RtHorG:Stack.Fetch src/Stack/Fetch.hs:295:17)
rainbyte commented 8 years ago

@Blaisorblade, as I've commented before, I'm experiencing the Left path issue when running stack install inside a docker container, should I report this as a new issue then?

I've seen that behaviour in one of my projects: https://github.com/rainbyte/openwhisk-wrapper

Edit: the issue appears also in an internal project, but I cannot expose the code

Blaisorblade commented 8 years ago

@rainbyte please do report a new issue; I currently assume my investigation doesn't apply to your scenario.

Blaisorblade commented 8 years ago

Further experiments (below) suggest the hashes are part of the snapshot, which makes sense to ensure reproducibility. Hence, we get a log entry for each cabal file in the snapshot considered which was updated since then. Furthermore, I understand the fallback uses a newer cabal file, limiting build reproducibility in the sense of #2217.

Under these circumstances, I'd propose to clone the entirety of all-cabal-hashes as a quick fix (currently 215M). This is not fine long-term since bandwidth is limited and since the repo will grow indefinitely, but a better fix wouldn't be immediate.

I imagine one could add tags for each snapshot, and stack could fetch those tags when needed, but the tags would be needed remotely.

Experiment Results

Using lts-6.0, we request for instance a package at revision 5, while git log reveals we're at revision 7. Trying out the snapshot in the OP shows the same hash (revision 6) for servant-yaml, while we're now at revision 8. Generally, all old messages reappear, and there are new ones.

Did not find .cabal file for opaleye-0.4.2.0 with Git SHA of 9b5608ac701b60a82f5cf985887d0d1450757864
Right Nothing
$ git show 9b5608ac701b60a82f5cf985887d0d1450757864|grep x-revi
x-revision: 5
$ git log -p current-hackage -- opaleye/0.4.2.0/opaleye.cabal
[...]
-x-revision: 6
+x-revision: 7
Blaisorblade commented 8 years ago

@rainbyte In your new issue, can you also add whether stack update fixes it? I've just gotten that after rm -rf ~/.stack/indices/Hackage/git-update—somehow, stack did not figure out it needed to refetch the repo.

Blaisorblade commented 8 years ago

@KeeperB5: mirroring turns out to not be the problem, but here are some answers. But since you were installing from scratch on Windows, here's the short of it: TL;DR.: Install Git and ensure it is on the PATH so Stack can use it to update the index faster. Otherwise, Stack will fall back to downloading an index snapshot via HTTP from scratch each time.

Below's more info than you'd want, saving it for me.

Well, my index is fetched from https://s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz, but I believe the GitHub mirror had https://github.com/commercialhaskell/all-cabal-hashes.

Those are the correct URLs, and they're related but not exactly mirrors, so the size difference is expected. https://github.com/commercialhaskell/all-cabal-hashes has extra metadata on top of https://github.com/commercialhaskell/all-cabal-files, which is a mirror in Git format of the tarball (which is a mirror of the Hackage index on the Amazon cloud).

How does Stack pick a mirror and do we have any way to manually select a mirror?

Basically, mirroring is not geographically based (except whatever optimizations Amazon does internally in S3). Make sure your stack can find your Git and you don't change your config, and stack will use git to update the index incrementally and faster (since you can download only the changes rather than a new snapshot), as described in these docs.

The relevant config is described here, https://docs.haskellstack.org/en/latest/yaml_configuration/#package-indices, but I'd avoid changing it usually.

With the default config, on both master and 1.1.2, the HTTP download (which is slower) is not used and the Git one is preferred, as long as Git is installed. So either stack doesn't find your Git or you changed your config.

Sources

Updating logic: https://github.com/commercialhaskell/stack/blob/aac2c81dd020e399b49894c566075a495240baaf/src/Stack/PackageIndex.hs#L214-L223

Default config (matching docs): https://github.com/commercialhaskell/stack/blob/aac2c81dd020e399b49894c566075a495240baaf/src/Stack/Config.hs#L219-L228

lehtoj commented 8 years ago

I've been using Visual Studio and SourceTree which don't need Git to be installed separately. Anyhow, I installed Git for Windows, deleted my stack_root directory, ran stack setup. Now it seems to work as intended, no more errors.

Perhaps install documentation should mention requirement for Git?

Blaisorblade commented 8 years ago

Perhaps install documentation should mention requirement for Git?

2469

I've been using Visual Studio and SourceTree which don't need Git to be installed separately. Anyhow, I installed Git for Windows, deleted my stack_root directory, ran stack setup. Now it seems to work as intended, no more errors.

Note these were warnings. Should you get further warnings of this kind, you can mostly ignore them—and we'll have a fix by next release. The real advantage with Git is that index updates will be much faster.

lehtoj commented 8 years ago

Noted. :) But the log does not really specify either way, Usually "The system cannot find the path specified." is treated as an error, so I just followed the suit since I didn't know better.

dpwiz commented 8 years ago

Receiving objects: 0% (3467/576096), 1.97 MiB | 19.00 KiB/s

Ouch! I never experienced the problems that prompted this fix but can't proceed now. You'd better add something like --do-no-harm or --fix-2175 next time. Until then, gotta return to previous stack... :pensive:

Blaisorblade commented 8 years ago

@wiz Is that your usual connection speed?

dpwiz commented 8 years ago

I hope not. It downloaded eventually, but in a pinch that got me nervous.

dpwiz commented 8 years ago

Now there's real pain. My CI have this restricted uplink and should start from scratch each time :pensive:

sjakobi commented 8 years ago

@wiz: Can't you cache the ~/.stack directory?