search

commercialhaskell / stack

The Haskell Tool Stack
http://haskellstack.org
BSD 3-Clause "New" or "Revised" License
4k stars 843 forks source link

Stack rebuilds pretty-show and few more packages with no apparent reason #4084

Closed gromakovsky closed 6 years ago

gromakovsky commented 6 years ago

General summary/comments (optional)

If I build one package, modify one file there and built it again, for some reason more than one package is rebuilt.

Steps to reproduce

Can be reproduced using this code: https://github.com/serokell/cardano-sl/tree/vanya (serokell/cardano-sl repository, vanya branch).

  1. Build the cardano-sl-txp package:

    stack build --fast cardano-sl-txp

  2. Modify some file:

    echo "\n -- aa" >> txp/src/Pos/Txp/Base.hs

  3. Build cardano-sl-txp again:

    stack build --fast cardano-sl-txp

Expected

Only one file is recompiled and the command finishes quickly.

Actual

Stack decides to unregister cardano-sl-core, pretty-show, cardano-sl-crypto-test and some other packages. Then it builds these packages:

cardano-sl-core-1.2.0: unregistering
cardano-sl-crypto-test-1.1.0: unregistering
cardano-sl-db-1.2.0: unregistering
cardano-sl-infra-1.2.0: unregistering
cardano-sl-txp-1.2.0: unregistering
hedgehog-0.6: unregistering
pretty-show-1.6.16: unregistering
pretty-show-1.6.16: configure
pretty-show-1.6.16: build
pretty-show-1.6.16: copy/register
hedgehog-0.6: configure (lib)
hedgehog-0.6: build (lib) 
hedgehog-0.6: copy/register
cardano-sl-crypto-test-1.1.0: configure (lib)
cardano-sl-crypto-test-1.1.0: build (lib) 
cardano-sl-crypto-test-1.1.0: copy/register
cardano-sl-core-1.2.0: configure (lib)    
cardano-sl-core-1.2.0: build (lib)        
cardano-sl-core-1.2.0: copy/register
cardano-sl-db-1.2.0: configure (lib)
cardano-sl-db-1.2.0: build (lib)   
cardano-sl-db-1.2.0: copy/register
cardano-sl-infra-1.2.0: configure (lib)
cardano-sl-infra-1.2.0: build (lib) 
cardano-sl-infra-1.2.0: copy/register
cardano-sl-txp-1.2.0: configure (lib)
Configuring cardano-sl-txp-1.2.0... 
cardano-sl-txp-1.2.0: build (lib) 
Preprocessing library for cardano-sl-txp-1.2.0..
Building library for cardano-sl-txp-1.2.0..
[23 of 37] Compiling Pos.Txp.Base     ( src/Pos/Txp/Base.hs, .stack-work/dist/x86_64-linux-tinfo6/Cabal-2.0.1.0/build/Pos/Txp/Base.o )
cardano-sl-txp-1.2.0: copy/register
Installing library in /home/gromak/Work/cardano-sl/.stack-work/install/x86_64-linux-tinfo6/lts-11.2/8.2.2/lib/x86_64-linux-ghc-8.2.2/cardano-sl-txp-1.2.0-kSN1N3EEYBFSpz2b5PzUX
Registering library for cardano-sl-txp-1.2.0..
Completed 7 action(s).

Here is the output in --verbose mode: https://gist.github.com/gromakovsky/a23b77cb9bcf3cc0e415b9a65f5f422d

Stack version

$ stack --version
Version 1.7.1, Git revision 681c800873816c022739ca7ed14755e85a579565 (5807 commits) x86_64 hpack-0.28.2

Method of installation

I am using version 1.7.1 downloaded from https://github.com/commercialhaskell/stack/releases.

nh2 commented 6 years ago

We'll be looking into this.

snoyberg commented 6 years ago

Note that it's a better idea for repros to give a commit than a branch. Right now, the vanya branch is pointing at 6c109584a6a7f182334eedc79c063a591626cca9.

snoyberg commented 6 years ago

Also, as a side note: due to the extremely large number of extra-deps in this stack.yaml, I'd recommend moving over to a custom snapshot, which will allow for more precompiled cache usage.

snoyberg commented 6 years ago

I'm going to start a debug log in this comment. I'll be editing it as I move along.

My first step is to confirm that this bug reproduces with the Stack on master. Also, in order to avoid wasting lots of CPU cycles, I'm going to be using the following command as a replacement for step 3 above:

stack build --fast cardano-sl-txp --dry-run

This way I'll keep the same broken state between attempts to play with this bug. I'm also almost certainly going to throw in a --verbose, and then begin adding a bunch of logInfo calls to the Stack codebase to get more info on why it's decided to unregister the packages in question.

I'm still building the absolute latest master, but I've tested with:

Version 1.8.0, Git revision d20833a5c0d3b2b62495fb126ecfd8fdbf1f6864 (5895 commits) x86_64 hpack-0.28.2

I get the following output, indicating that the bug no longer exists:

$ stack build --fast cardano-sl-txp --dry-run

Warning: There were multiple candidates for the Cabal entry " hlint.hs 
         * /Users/michael/Documents/cardano-sl/.stack-work/downloaded/XTTklt4ROs4T/hlint.hs 
         picking:
         /Users/michael/Documents/cardano-sl/.stack-work/downloaded/XTTklt4ROs4T/tests/hlint.hs
Would unregister locally:
cardano-sl-txp-1.2.0 (local file changes: src/Pos/Txp/Base.hs)

Would build:
cardano-sl-txp-1.2.0: database=local, source=/Users/michael/Documents/cardano-sl/txp/

No executables to be installed.

However, building with Stack 1.7.1 produces the same bug as reported here:

``` $ stack-1.7.1 build --fast cardano-sl-txp --dry-run Warning: There were multiple candidates for the Cabal entry " hlint.hs * /Users/michael/Documents/cardano-sl/.stack-work/downloaded/XTTklt4ROs4T/hlint.hs picking: /Users/michael/Documents/cardano-sl/.stack-work/downloaded/XTTklt4ROs4T/tests/hlint.hs Would unregister locally: cardano-crypto-1.0.0 (missing dependencies: cryptonite, cryptonite-openssl) cardano-report-server-0.4.10 (missing dependencies: log-warper, wreq) cardano-sl-binary-1.2.0 (missing dependencies: serokell-util) cardano-sl-core-1.2.0 (missing dependencies: cardano-sl-binary, cardano-sl-crypto, cardano-sl-crypto-test, cardano-sl-networking, cardano-sl-u...) cardano-sl-crypto-1.2.0 (missing dependencies: cardano-crypto, cardano-sl-binary, cardano-sl-util, cryptonite, cryptonite-openssl, ed25519, pvss...) cardano-sl-crypto-test-1.1.0 (missing dependencies: cardano-sl-binary, cardano-sl-crypto, cardano-sl-util, cardano-sl-util-test, cryptonite, hedgehog) cardano-sl-db-1.2.0 (missing dependencies: cardano-sl-binary, cardano-sl-core, cardano-sl-crypto, cardano-sl-util, rocksdb-haskell-ng, serok...) cardano-sl-infra-1.2.0 (missing dependencies: cardano-report-server, cardano-sl-binary, cardano-sl-core, cardano-sl-crypto, cardano-sl-db, card...) cardano-sl-networking-1.2.0 (missing dependencies: cardano-sl-util, cryptonite, kademlia, log-warper, network-transport-tcp, serokell-util) cardano-sl-txp-1.2.0 (missing dependencies: cardano-sl-binary, cardano-sl-core, cardano-sl-crypto, cardano-sl-crypto-test, cardano-sl-db, car...) cardano-sl-util-1.2.0 (missing dependencies: cryptonite, log-warper, serokell-util) cardano-sl-util-test-1.1.0 (missing dependencies: cardano-sl-util, cryptonite) connection-0.2.8 (missing dependencies: tls, x509, x509-store, x509-system, x509-validation) cryptonite-0.24 (local file changes: benchs/Bench.hs benchs/Number/F2m.hs cbits/aes/block128.h cbits/aes/generic.h cbits/aes/gf.h cbit...) cryptonite-openssl-0.7 (missing dependencies: cryptonite) dns-3.0.2 (missing dependencies: cryptonite) ed25519-0.0.5.0 (local file changes: .travis.yml AUTHORS.txt CHANGELOG.md CONTRIBUTING.md README.md benchmarks/bench.hs ed25519.cabal ...) ekg-wai-0.1.0.2 (missing dependencies: wai-app-static) hedgehog-0.6 (local file changes: /Users/michael/Documents/cardano-sl/.stack-work/downloaded/27pmaoPEDNVl/CHANGELOG.md /Users/micha...) http-client-tls-0.3.5.3 (missing dependencies: connection, cryptonite, tls) kademlia-1.1.0.1 (missing dependencies: cryptonite) log-warper-1.8.10.1 (local file changes: CHANGES.md README.md examples/HowTo.md examples/PureLogging.md log-warper.cabal test/Spec.hs test...) network-transport-tcp-0.6.0 (local file changes: ChangeLog network-transport-tcp.cabal tests/TestQC.hs tests/TestTCP.hs) plutus-prototype-0.1.0.0 (missing dependencies: cardano-crypto, cryptonite, ed25519) pvss-0.2.0 (missing dependencies: cryptonite, cryptonite-openssl) rocksdb-haskell-ng-0.0.0 (local file changes: bench/Main.hs rocksdb-haskell-ng.cabal test/Spec.hs) serokell-util-0.8.0 (missing dependencies: log-warper) tls-1.4.1 (missing dependencies: cryptonite, x509, x509-store, x509-validation) wai-app-static-3.1.6.2 (missing dependencies: cryptonite) wreq-0.5.2.1 (missing dependencies: cryptonite, http-client-tls) x509-1.7.3 (missing dependencies: cryptonite) x509-store-1.6.6 (missing dependencies: cryptonite, x509) x509-system-1.6.6 (missing dependencies: x509, x509-store) x509-validation-1.6.10 (missing dependencies: cryptonite, x509, x509-store) Would build: cardano-crypto-1.0.0: database=local, source=/Users/michael/Documents/cardano-sl/.stack-work/downloaded/Du3of51NERkg/, after: cryptonite-0.24,cryptonite-openssl-0.7 cardano-report-server-0.4.10: database=local, source=/Users/michael/Documents/cardano-sl/.stack-work/downloaded/HxVyEV0NYAPr/, after: log-warper-1.8.10.1,wreq-0.5.2.1 cardano-sl-binary-1.2.0: database=local, source=/Users/michael/Documents/cardano-sl/binary/, after: serokell-util-0.8.0 cardano-sl-core-1.2.0: database=local, source=/Users/michael/Documents/cardano-sl/core/, after: cardano-sl-binary-1.2.0,cardano-sl-crypto-1.2.0,cardano-sl-crypto-test-1.1.0,cardano-sl-networking-1.2.0,cardano-sl-util-1.2.0,cardano-sl-util-test-1.1.0,cryptonite-0.24,log-warper-1.8.10.1,plutus-prototype-0.1.0.0,serokell-util-0.8.0 cardano-sl-crypto-1.2.0: database=local, source=/Users/michael/Documents/cardano-sl/crypto/, after: cardano-crypto-1.0.0,cardano-sl-binary-1.2.0,cardano-sl-util-1.2.0,cryptonite-0.24,cryptonite-openssl-0.7,ed25519-0.0.5.0,pvss-0.2.0,serokell-util-0.8.0 cardano-sl-crypto-test-1.1.0: database=local, source=/Users/michael/Documents/cardano-sl/crypto/test/, after: cardano-sl-binary-1.2.0,cardano-sl-crypto-1.2.0,cardano-sl-util-1.2.0,cardano-sl-util-test-1.1.0,cryptonite-0.24,hedgehog-0.6 cardano-sl-db-1.2.0: database=local, source=/Users/michael/Documents/cardano-sl/db/, after: cardano-sl-binary-1.2.0,cardano-sl-core-1.2.0,cardano-sl-crypto-1.2.0,cardano-sl-util-1.2.0,rocksdb-haskell-ng-0.0.0,serokell-util-0.8.0 cardano-sl-infra-1.2.0: database=local, source=/Users/michael/Documents/cardano-sl/infra/, after: cardano-report-server-0.4.10,cardano-sl-binary-1.2.0,cardano-sl-core-1.2.0,cardano-sl-crypto-1.2.0,cardano-sl-db-1.2.0,cardano-sl-networking-1.2.0,cardano-sl-util-1.2.0,dns-3.0.2,ekg-wai-0.1.0.2,http-client-tls-0.3.5.3,kademlia-1.1.0.1,log-warper-1.8.10.1,network-transport-tcp-0.6.0,serokell-util-0.8.0 cardano-sl-networking-1.2.0: database=local, source=/Users/michael/Documents/cardano-sl/networking/, after: cardano-sl-util-1.2.0,cryptonite-0.24,kademlia-1.1.0.1,log-warper-1.8.10.1,network-transport-tcp-0.6.0,serokell-util-0.8.0 cardano-sl-txp-1.2.0: database=local, source=/Users/michael/Documents/cardano-sl/txp/, after: cardano-sl-binary-1.2.0,cardano-sl-core-1.2.0,cardano-sl-crypto-1.2.0,cardano-sl-crypto-test-1.1.0,cardano-sl-db-1.2.0,cardano-sl-infra-1.2.0,cardano-sl-networking-1.2.0,cardano-sl-util-1.2.0,cardano-sl-util-test-1.1.0,log-warper-1.8.10.1,plutus-prototype-0.1.0.0,rocksdb-haskell-ng-0.0.0,serokell-util-0.8.0 cardano-sl-util-1.2.0: database=local, source=/Users/michael/Documents/cardano-sl/util/, after: cryptonite-0.24,log-warper-1.8.10.1,serokell-util-0.8.0 cardano-sl-util-test-1.1.0: database=local, source=/Users/michael/Documents/cardano-sl/util/test/, after: cardano-sl-util-1.2.0,cryptonite-0.24 connection-0.2.8: database=local, source=package index, after: tls-1.4.1,x509-1.7.3,x509-store-1.6.6,x509-system-1.6.6,x509-validation-1.6.10 cryptonite-0.24: database=local, source=/Users/michael/Documents/cardano-sl/.stack-work/downloaded/lnvg6q-bhfcw/ cryptonite-openssl-0.7: database=local, source=package index, after: cryptonite-0.24 dns-3.0.2: database=local, source=package index, after: cryptonite-0.24 ed25519-0.0.5.0: database=local, source=/Users/michael/Documents/cardano-sl/.stack-work/downloaded/XTTklt4ROs4T/ ekg-wai-0.1.0.2: database=local, source=package index, after: wai-app-static-3.1.6.2 hedgehog-0.6: database=local, source=/Users/michael/Documents/cardano-sl/.stack-work/downloaded/27pmaoPEDNVl/hedgehog/ http-client-tls-0.3.5.3: database=local, source=package index, after: connection-0.2.8,cryptonite-0.24,tls-1.4.1 kademlia-1.1.0.1: database=local, source=/Users/michael/Documents/cardano-sl/.stack-work/downloaded/uiwbEc8tbIi9/, after: cryptonite-0.24 log-warper-1.8.10.1: database=local, source=/Users/michael/Documents/cardano-sl/.stack-work/downloaded/DOh1FXO6uJXI/ network-transport-tcp-0.6.0: database=local, source=/Users/michael/Documents/cardano-sl/.stack-work/downloaded/17t1IvOFATIc/ plutus-prototype-0.1.0.0: database=local, source=/Users/michael/Documents/cardano-sl/.stack-work/downloaded/KQSGhWYbOHSL/, after: cardano-crypto-1.0.0,cryptonite-0.24,ed25519-0.0.5.0 pvss-0.2.0: database=local, source=package index, after: cryptonite-0.24,cryptonite-openssl-0.7 rocksdb-haskell-ng-0.0.0: database=local, source=/Users/michael/Documents/cardano-sl/.stack-work/downloaded/WoYdLUpErLYh/ serokell-util-0.8.0: database=local, source=package index, after: log-warper-1.8.10.1 tls-1.4.1: database=local, source=package index, after: cryptonite-0.24,x509-1.7.3,x509-store-1.6.6,x509-validation-1.6.10 wai-app-static-3.1.6.2: database=local, source=package index, after: cryptonite-0.24 wreq-0.5.2.1: database=local, source=package index, after: cryptonite-0.24,http-client-tls-0.3.5.3 x509-1.7.3: database=local, source=package index, after: cryptonite-0.24 x509-store-1.6.6: database=local, source=package index, after: cryptonite-0.24,x509-1.7.3 x509-system-1.6.6: database=local, source=package index, after: x509-1.7.3,x509-store-1.6.6 x509-validation-1.6.10: database=local, source=package index, after: cryptonite-0.24,x509-1.7.3,x509-store-1.6.6 No executables to be installed. ```

Meaning I'm fairly certain that this bug is already resolved on master. It will take more investigation to determine what change on master could have resolved this.

gromakovsky commented 6 years ago

I couldn't fetch d20833a5c0d3b2b62495fb126ecfd8fdbf1f6864, so I tried using its parent: dbc74ecce2e31b47626d87a1cf3ea6129fed1e99. d20833a5c0d3b2b62495fb126ecfd8fdbf1f6864 has only cosmetic changes, so it shouldn't matter.

For me it still unregisters packages that it shouldn't, here's the output of stack build --fast cardano-sl-txp --dry-run: https://gist.github.com/gromakovsky/7ca7bac5b4f5fedaf269665985eadb5a

dbaynard commented 6 years ago

It might be helpful to have a minimum example, though it seems that might be tricky. Would you try to produce one, @gromakovsky?

gromakovsky commented 6 years ago

I was successfully minimizing this example while keeping the bug reproducible, but then I did stack clean --full and since then I can't reproduce this bug anymore, even when I return to where I started (6c109584a6a7f182334eedc79c063a591626cca9) and use the same version of stack as before (681c800873816c022739ca7ed14755e85a579565) :man_shrugging:

Probably something bad appeared in .stack-work, I don't know how :(

gromakovsky commented 6 years ago

I suceeded to reproduce it using version 1.6.5 (revision 24ab0d6ff07f28276e082c3ce74dfdeb1a2ca9e9). I also minimized it as much as I could. Commit 571fd554b3e1613fc27e26563f4d9dee823b4de6 is the minimal example where the bug is present. It's a bit different though. Steps:

  1. stack clean --full.
  2. stack build --fast cardano-sl-core.
  3. stack build --fast cardano-sl-core again rebuilds hedgehog and others.

Commit adb0e36ab6be0d0f4c907d1b9fc866e67bcdf673 is the next one in the vanya branch and there doing stack build --fast cardano-sl-core again doesn't rebuild anything. I stopped at that point.

Once again, after stack clean --full the bug reproduces only with stack revision 24ab0d6ff07f28276e082c3ce74dfdeb1a2ca9e9. I can't reproduce it using latest release or master anymore (I could before probably because I didn't do stack clean --full).

dbaynard commented 6 years ago

OK, it's good that there's a resolution. Thank you for digging around (I editied your commit refs to links so I could follow them). I suspect you're right about something in the .stack-work folder being a culprit — and the extra-deps appear to have contributed, too.

I'd recommend @snoyberg's advice: it is worth using a custom snapshot.

I'm going to close — do get in touch if there's anything else you'd need, on this.