bootstrap.hs doesn't work with stack 2.3.1 (Cannot complete repo information for a non SHA1 commit due to non-reproducibility)

[blackgnezdo]

gnezdo@os-ws:~/s$ git clone https://github.com/google/proto-lens.git
Cloning into 'proto-lens'...
remote: Enumerating objects: 61, done.
remote: Counting objects: 100% (61/61), done.
remote: Compressing objects: 100% (52/52), done.
remote: Total 4590 (delta 9), reused 31 (delta 7), pack-reused 4529
Receiving objects: 100% (4590/4590), 1.42 MiB | 0 bytes/s, done.
Resolving deltas: 100% (2592/2592), done.
gnezdo@os-ws:~/s$ cd ./proto-lens/
gnezdo@os-ws:~/s/proto-lens$ git submodule update --init --recursive
Submodule 'google/protobuf' (https://github.com/google/protobuf) registered for path 'google/protobuf'
Cloning into '/home/gnezdo/s/proto-lens/google/protobuf'...
Submodule path 'google/protobuf': checked out '48cb18e5c419ddd23d9badcfe4e9df7bde1979b2'
Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'google/protobuf/third_party/benchmark'
Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'google/protobuf/third_party/googletest'
Cloning into '/home/gnezdo/s/proto-lens/google/protobuf/third_party/benchmark'...
Cloning into '/home/gnezdo/s/proto-lens/google/protobuf/third_party/googletest'...
Submodule path 'google/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
Submodule path 'google/protobuf/third_party/googletest': checked out 'c3bb0ee2a63279a803aaad956b9b26d74bf9e6e2'
gnezdo@os-ws:~/s/proto-lens$ stack --version
Version 2.3.1, Git revision de2a7b694f07de7e6cf17f8c92338c16286b2878 (8103 commits) x86_64 hpack-0.33.0
gnezdo@os-ws:~/s/proto-lens$ runghc bootstrap.hs 
Cannot complete repo information for a non SHA1 commit due to non-reproducibility: Git repo at https://github.com/google/proto-lens, commit master in subdirectory proto-lens
bootstrap.hs: readCreateProcess: stack "--stack-yaml=stack-bootstrap.yaml" "path" "--local-install-root" (exit 1): failed

[jinwoo]

The error message is related to this: https://github.com/commercialhaskell/stack/issues/4882

[jinwoo]

After https://github.com/google/proto-lens/pull/392, we go one step further and get this error:

$ stack runghc -- bootstrap.hs 
Cloning 0bef8c2f3da645f068b8a26ac168c1da41608182 from https://github.com/google/proto-lens
Unsupported tarball from /tmp/with-repo-archive212759/foo.tar: Symbolic link dest not found from proto-lens/proto-lens-imports/google to ../../google/protobuf/src/google, looking for proto-lens/../google/protobuf/src/google.
This may indicate that the source is a git archive which uses git-annex.
See https://github.com/commercialhaskell/stack/issues/4579 for further information.
bootstrap.hs: readCreateProcess: stack "--stack-yaml=stack-bootstrap.yaml" "path" "--local-install-root" (exit 1): failed

Apparently related to https://github.com/commercialhaskell/stack/issues/4579.

[jinwoo]

I tried various things but couldn't figure out how to fix this symbolic link error. The new stack doesn't seem to give an escape hatch for ignoring it. @judah any idea?

[judah]

Oof, sorry, I forgot about this problem. I encountered it a while ago here: https://github.com/commercialhaskell/stack/issues/5004#issuecomment-527197457

Previously I just used stack-1.* when bootstrapping, but that won't work anymore due to the changes to proto-lens-runtime's reexport-modules. I suppose one fairly manual approach would be:

• Delete package.yaml (so proto-lens-runtime.cabal is the source of truth)
• Run the bootstrapping script with stack-1.*
• Add back that package.yaml

That might be worth trying to unblock the release. I'll see if I can think of something better.

[judah]

Actually, currently we're in a good place because we're not making any backwards-incompatible changes to the API. So I was able to regenerate it by removing the git-related logic altogether and using the in-tree packages for bootstrapping. It's not a great long-term solution, but it should work for the upcoming release:

diff --git a/bootstrap.hs b/bootstrap.hs
index 1d55ed4..9a00f37 100755
--- a/bootstrap.hs
+++ b/bootstrap.hs
@@ -47,9 +47,6 @@ main = do
     --    overwriting the previous versions.
     callProcess "git" ["checkout", bootstrapCommit, "--",
                        bootstrapModuleRoot </> "Proto"]
-    [sha] <- lines <$> readProcess "git" ["rev-parse", bootstrapCommit] ""
-    -- Append the bootstrapping commit hash to the yaml file.
-    appendFile bootstrappingYaml ("  commit: " ++ sha ++ "\n")
     [installRoot] <- lines <$> readProcess stack
                     [useBootstrappingYaml, "path", "--local-install-root"] ""
     let protocGenHaskell = installRoot </> "bin/proto-lens-protoc"
diff --git a/stack-bootstrap.yaml b/stack-bootstrap.yaml
index a30c62c..6edfe97 100644
--- a/stack-bootstrap.yaml
+++ b/stack-bootstrap.yaml
@@ -1,15 +1,11 @@
-resolver: lts-12.7
+resolver: lts-15.15

 packages:
+- proto-lens
 - proto-lens-protoc
+- proto-lens-runtime

 # Build the current HEAD proto-lens-protoc against older revisions of proto-lens
 # and proto-lens-descriptors that are consistent with each other.
 extra-deps:
 - ghc-source-gen-0.4.0.0
-- git: https://github.com/google/proto-lens
-  subdirs:
-  - proto-lens
-  - proto-lens-runtime
-  # A line like below will be appended by bootstrap.hs.
-  # commit: 0bef8c2f3da645f068b8a26ac168c1da41608182