"stack repl <test suite>" fails if "stack test --no-run-tests" is not run in advance for test suite containing errors

[runeksvendsen]

General summary/comments

Running the REPL for only the test suite in my project fails with an error saying the library package cannot be satisfied (even though the library builds without errors).

Please see https://github.com/commercialhaskell/stack/issues/5380#issuecomment-686731779 for an accurate description of the problem (I didn't realize this was the exact problem when I created this issue).

Steps to reproduce

1. Run the following commands to clone and build the example repo (the last step results in a compilation error):

git clone https://github.com/runeksvendsen/bellman-ford.git
cd bellman-ford
git checkout 781835aa000ad40f6c54f2c7426e9378ff6905e4
stack build
stack test --no-run-tests

Now start a REPL for the test suite:

stack repl bellman-ford:test:bellman-ford-test

Expected

Not failing with a "cannot satisfy -package"-error since the library builds fine.

Actual

It fails with the following error:

<command line>: cannot satisfy -package bellman-ford-0.1.0.0
    (use -v for more information)

Verbose output: https://gist.github.com/runeksvendsen/4ef3125578dc9de82962912be29f2f89

Stack version

$ stack --version
Version 2.3.3, Git revision cb44d51bed48b723a5deb08c3348c0b3ccfc437e x86_64 hpack-0.33.0

Method of installation

• Download from GitHub/Releases, then upgraded via stack upgrade

[mattaudesse]

Hey @runeksvendsen - sorry for the delayed response.

When I follow your steps above stack test --no-run-tests fails on commit 781835aa000ad40f6c54f2c7426e9378ff6905e4:

...
[ 9 of 12] Compiling Digraph.Spec                                                                         

/apps/apps/stack-issues/i5380/test/Digraph/Spec.hs:47:22: error:
    • Couldn't match type ‘TestEdge’ with ‘Lib.IdxEdge String Double’                                     
      Expected type: TestEdge -> IO ()                                                                    
        Actual type: Lib.IdxEdge String Double -> IO ()                                                   
    • In the second argument of ‘forM_’, namely
        ‘(stToIO . Lib.removeEdge graph)’
      In a stmt of a 'do' block:                                                                          
        forM_ edges (stToIO . Lib.removeEdge graph)
      In the expression:                     
        do graph <- stToIO $ Lib.fromEdges edges
           forM_ edges (stToIO . Lib.removeEdge graph)                                                    
           vertices <- stToIO $ Lib.vertices graph                                                        
           return (graph, vertices)    
   |        
47 |         forM_ edges (stToIO . Lib.removeEdge graph)                                    
   |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                   

/apps/apps/stack-issues/i5380/test/Digraph/Spec.hs:62:44: error:                                          
    • Couldn't match type ‘edge’ with ‘TestEdge’
      ‘edge’ is a rigid type variable bound by
        the type signature for:      
          addEdgesCheckInOutgoing :: forall edge.
                                     (Lib.Digraph RealWorld String Double                                 
                                      -> Lib.VertexId -> IO [edge])                                       
                                     -> [TestEdge] -> Expectation      
        at test/Digraph/Spec.hs:(51,1)-(54,18)                                                            
      Expected type: [edge]                         
        Actual type: [TestEdge]        
    • In the second argument of ‘shouldBe’, namely
        ‘removeDuplicateSrcDst sortedEdges’
      In a stmt of a 'do' block:
        sort (concat outgoingEdges)
          `shouldBe` removeDuplicateSrcDst sortedEdges
      In the expression:
        do let sortedEdges = sort edges
           graph <- stToIO $ Lib.fromEdges (reverse sortedEdges)
           vertices <- stToIO $ Lib.vertices graph
           outgoingEdges <- foldM (collectInOutgoing graph) [] vertices
           ....
    • Relevant bindings include
        outgoingEdges :: [[edge]] (bound at test/Digraph/Spec.hs:61:5)
        collectInOutgoing :: Lib.Digraph RealWorld String Double
                             -> [[edge]] -> Lib.VertexId -> IO [[edge]]
          (bound at test/Digraph/Spec.hs:64:5)
        inOutEdges :: Lib.Digraph RealWorld String Double
                      -> Lib.VertexId -> IO [edge]
          (bound at test/Digraph/Spec.hs:55:25)
        addEdgesCheckInOutgoing :: (Lib.Digraph RealWorld String Double
                                    -> Lib.VertexId -> IO [edge])
                                   -> [TestEdge] -> Expectation
          (bound at test/Digraph/Spec.hs:55:1)
   |        
62 |     sort (concat outgoingEdges) `shouldBe` removeDuplicateSrcDst sortedEdges
   |                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

--  While building package bellman-ford-0.1.0.0 using:
      /usr/home/matt/.stack/setup-exe-cache/x86_64-freebsd-ino64/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.5 --builddir=.stack-work/dist/x86_64-freebsd-ino64/Cabal-2.4.0.1 build lib:bellman-ford test:bellman-ford-tes
t --ghc-options " -fdiagnostics-color=always"
    Process exited with code: ExitFailure 1
Progress 1/2

The same steps (including stack repl bellman-ford:test:bellman-ford-test) seem to run fine on master, though, and it looks like your updates there are more recent.

Is it possible this is a fluke or am I missing something? Thanks for reporting in any case.

[runeksvendsen]

Hi @mattaudesse

Thank you for getting back to me. No worries about the delay.

The crux of this bug report is as follows.

If I do the following:

git clone https://github.com/runeksvendsen/bellman-ford.git
cd bellman-ford
git checkout 8e72e83a1cc29ba99ab171676d1c50a77d507ca1
stack test --no-run-tests
echo "\nwrongDefinition = 'a' :: Int" >> test/Spec.hs
stack repl bellman-ford:test:bellman-ford-test

Then the stack repl command successfully launches GHCi ie. leaves me with a prompt -- while reporting a type mismatch error in test/Spec.hs (Couldn't match expected type ‘Int’ with actual type ‘Char’).

If, however, I do the following (ie. omit stack test --no-run-tests before running stack repl):

git clone https://github.com/runeksvendsen/bellman-ford.git
cd bellman-ford
git checkout 8e72e83a1cc29ba99ab171676d1c50a77d507ca1
echo "\nwrongDefinition = 'a' :: Int" >> test/Spec.hs
stack repl bellman-ford:test:bellman-ford-test

Then the stack repl command fails to launch GHCi (ie. does not leave me with a prompt, but exits) with the error cannot satisfy -package bellman-ford-0.1.0.0.

I would think that the success of running stack repl should not depend on whether any stack build command has been run in advance. That is, stack repl in the latter sequence of commands should behave exactly as stack repl in the former sequence of commands (ie. not requiring that e.g. stack test --no-run-tests is run before running stack repl ...).

The reason I report this as a bug is because it causes a problem when stack is used in combination with haskell-language-server (as described here: https://github.com/haskell/haskell-language-server/issues/366).

[mattaudesse]

Thanks for the extra info - that helps.

I've tagged this as a bug but also "for discussion" since I could see it posing some architectural questions about what should happen in this case + how.

[runeksvendsen]

Seems like this is related to #1406. The comment here seems to describe this bug perfectly: https://github.com/commercialhaskell/stack/issues/1406#issuecomment-199627419.

[qrilka]

@runeksvendsen from my quick glance it looks like this problem could be another case for #4745 but I didn't have time yet to look closer into it.

[runeksvendsen]

@runeksvendsen from my quick glance it looks like this problem could be another case for #4745 but I didn't have time yet to look closer into it.

Thank you for looking into this.

What makes you think the two are related? Would fixing #4745 get rid of the error described in https://github.com/commercialhaskell/stack/issues/1406#issuecomment-199627419?

[qrilka]

Yes, component-based builds seems to be exactly what's needed in that situation - currently Stack builds all the components at once with an option to enable/disable tests/benchmarks.

[jneira]

This looks quite similar to #5213 (first stack repl foo:test:bar needs stack build --test --no-run-tests), that it is supposed to be fixed (partially)

[runeksvendsen]

This looks quite similar to #5213 (first stack repl foo:test:bar needs stack build --test --no-run-tests), that it is supposed to be fixed (partially)

It's very similar, but there's a difference in that this fix seems to no longer work if there's a compilation error in the test suite. If there is a compilation error in the test suite then stack test --no-run-tests is still required in order for stack repl foo:test:bar to succeed (as outlined above: https://github.com/commercialhaskell/stack/issues/5380#issuecomment-686731779).

[jneira]

hi, is there some chance to have some progresss on this? hls+stack users are reporting this frequently, thanks!

[qrilka]

Unfortunately it looks like the work on component-based builds has stalled and there seem to be no budget for this or new volunteers :(

[jneira]

@qrilka thanks for the update. I want to note that i think the inclusion of show-build-info in stack leveraging @fendor and @chrisdone amazing work would help to properly fix the root issue (see https://github.com/haskell/hie-bios/issues/269)

[qrilka]

Thanks for sharing the link, I will at least check it out later today.

[fendor]

Apparently we are still having this issue. Independent of other solutions, as mentioned in https://github.com/commercialhaskell/stack/issues/5380#issuecomment-933251584, do we have progress on this issue, or how could we achieve some progress?

[mpilgrem]

Documenting my own understanding of the problem with the current version of Stack (the corrected release candidate for Stack 2.11.1) and a plain vanilla stack new foo project:

stack purge
stack build --test --no-run-tests
[edit test\Spec.hs to introduce an error]
stack ghci foo:test:foo-test

results in (ending with a GHCi prompt):

❯ stack build --test --no-run-tests
Building all executables for foo once. After a successful build of all of them, only specified executables will be rebuilt.
foo> configure (lib + exe + test)
Configuring foo-0.1.0.0...
foo> build (lib + exe + test)
Preprocessing library for foo-0.1.0.0..
Building library for foo-0.1.0.0..
[1 of 2] Compiling Lib
[2 of 2] Compiling Paths_foo
Preprocessing test suite 'foo-test' for foo-0.1.0.0..
Building test suite 'foo-test' for foo-0.1.0.0..
[1 of 2] Compiling Main
[2 of 2] Compiling Paths_foo
Linking .stack-work\dist\8a54c84f\build\foo-test\foo-test.exe ...
Preprocessing executable 'foo-exe' for foo-0.1.0.0..
Building executable 'foo-exe' for foo-0.1.0.0..
[1 of 2] Compiling Main
[2 of 2] Compiling Paths_foo
Linking .stack-work\dist\8a54c84f\build\foo-exe\foo-exe.exe ...
foo> copy/register
Installing library in C:\Users\mikep\Documents\Code\Haskell\foo\.stack-work\install\177ee476\lib\x86_64-windows-ghc-9.2.7\foo-0.1.0.0-LMq7g0IxiUP42GrTYx6WDl
Installing executable foo-exe in C:\Users\mikep\Documents\Code\Haskell\foo\.stack-work\install\177ee476\bin
Registering library for foo-0.1.0.0..
foo> Test running disabled by --no-run-tests flag.
Completed 2 action(s).

❯ stack ghci foo:test:foo-test
Using main module:
1.  Package foo, component foo:test:foo-test, with main-is file: C:\Users\mikep\Documents\Code\Haskell\foo\test\Spec.hs.

foo> initial-build-steps (lib + test)
foo> Test running disabled by --no-run-tests flag.
Completed 2 action(s).

Warning: The following GHC options are incompatible with GHCi and have not been passed to it: -threaded.

Configuring GHCi with the following packages: foo.
GHCi, version 9.2.7: https://www.haskell.org/ghc/  :? for help
[1 of 2] Compiling Main             ( C:\Users\mikep\Documents\Code\Haskell\foo\test\Spec.hs, interpreted )

C:\Users\mikep\Documents\Code\Haskell\foo\test\Spec.hs:4:1: error:
    The type signature for ‘duff’ lacks an accompanying binding
  |
4 | duff :: IO ()
  | ^^^^
Failed, no modules loaded.

<no location info>: error:
    Could not load module ‘Paths_foo’
    it is a hidden module in the package ‘foo-0.1.0.0’
Loaded GHCi configuration from C:\Users\mikep\AppData\Local\stack\ghci-script\28f6f0ef\ghci-script
ghci>

While

stack purge
[edit test\Spec.hs to introduce an error]
stack ghci foo:test:foo-test

results in (ending with GHCi's exit):

❯ stack ghci foo:test:foo-test
Using main module:
1.  Package foo, component foo:test:foo-test, with main-is file: C:\Users\mikep\Documents\Code\Haskell\foo\test\Spec.hs.

Building all executables for foo once. After a successful build of all of them, only specified executables will be rebuilt.
foo> configure (lib + exe + test)
Configuring foo-0.1.0.0...
foo> build (lib + exe + test)
Preprocessing library for foo-0.1.0.0..
Building library for foo-0.1.0.0..
[1 of 2] Compiling Lib
[2 of 2] Compiling Paths_foo
Preprocessing test suite 'foo-test' for foo-0.1.0.0..
Building test suite 'foo-test' for foo-0.1.0.0..
[1 of 2] Compiling Main

test\Spec.hs:4:1: error:
    The type signature for ‘duff’ lacks an accompanying binding
  |
4 | duff :: IO ()
  | ^^^^

Error: [S-7282]
       Stack failed to execute the build plan.

       While executing the build plan, Stack encountered the error:

       [S-7011]
       While building package foo-0.1.0.0 (scroll up to its section to see the error) using:
       C:\sr\setup-exe-cache\x86_64-windows\Cabal-simple_9p6GVs8J_3.6.3.0_ghc-9.2.7.exe --verbose=1 --builddir=.stack-work\dist\8a54c84f build lib:foo exe:foo-exe test:foo-test --ghc-options " -fdiagnostics-color=always"
       Process exited with code: ExitFailure 1

Warning: Build failed, but trying to launch GHCi anyway

Warning: The following GHC options are incompatible with GHCi and have not been passed to it: -threaded.

Configuring GHCi with the following packages: foo.
Progress 1/2GHCi, version 9.2.7: https://www.haskell.org/ghc/  :? for help
<command line>: cannot satisfy -package foo-0.1.0.0
    (use -v for more information)

[mpilgrem]

So, in the first example, the library foo is built, installed and registered outside of GHCi before any error is introduced into test/Spec.hs. GHCi can be launched with package foo, but it can't compile Spec.hs after launch.

In the second example, the library foo is built (outside of GHCi) but the error already introduced into test/Spec.hs brings the build plan to an end before the library is installed and registered - and so GHCi then complains on launch that it cannot satisfy -package foo-0.1.0.0 and exits before completing launch.

At first blush, the different behaviour seems to me to explicable by reference to how GHCi behaves. Is the suggestion that Stack should always go on to install/register a package's library (if built) - even if another part of the build plan for the package fails?

[mpilgrem]

I think the key part of Stack's code is the realbuild where of Stack.Build.Execute.singleBuild (extracts only below):

  realBuild ::
       ConfigCache
    -> Package
    -> Path Abs Dir
    -> (KeepOutputOpen -> ExcludeTHLoading -> [String] -> RIO env ())
    -> (Utf8Builder -> RIO env ())
       -- ^ A plain 'announce' function
    -> Map Text ExecutableBuildStatus
    -> RIO env Installed
  realBuild cache package pkgDir cabal0 announce executableBuildStatuses = do
    let cabal = cabal0 CloseOnException
...
    () <- announce
      (  "build"
      <> display (annSuffix executableBuildStatuses)
      )
...
    cabal stripTHLoading (("build" :) $ (++ extraOpts) $ ...
      `catch` \ex -> case ex of
        CabalExitedUnsuccessfully{} ->
          postBuildCheck False >> prettyThrowM ex
        _ -> throwM ex
...
    when shouldCopy $ withMVar eeInstallLock $ \() -> do
      announce "copy/register"
      eres <- try $ cabal KeepTHLoading ["copy"]
...
      when hasLibrary $ cabal KeepTHLoading ["register"]

Stack uses Cabal (the library) to build. If Cabal fails for some reason, an exception is thown and Stack never gets to the actions where Stack uses Cabal to register. I can't see an easy way for Stack to know that Cabal's exception related to a test suite and did not relate to relate to a library.

[googleson78]

For whoever is deciding how much work/budget gets put into Stack/how issues are prioritised:

I want to mention that HLS not working smoothly with stack (which this particular components issue is seemingly 90% of the reason why) is by far the #\1 reason why I've currently switched to using cabal.