Running tests on local through `stack` fails

[german1608]

I have run tests in my machine through stack test and the test for interactive check fails:

Iris
  Tool
    should find 'curl' [✔]
    shouldn't find 'xxx_unknown_executable' [✔]
    shouldn't find 'ghc' version 100 [✔]
  Cli Options
    help without version environment [✔]
    help with version environment [✔]
    --numeric-version returns correct version [✔]
    CI interactivity check [✘]
    --version returns correct version text [✔]
  Cli Parser Conflicts
    --no-input=someValue defined by user - arg provided [✔]
    --no-input=someValue defined by user - no arg provided [✔]
    --no-input=someValue defined by user - not provided at all [✔]
    --no-input switch defined by user - provided [✔]
    --no-input switch defined by user - not provided [✔]
    --no-input switch with command defined by user - user provided [✔]
    --no-input switch with command defined by user - internal provided [✔]

Failures:

  test/Test/Iris/Cli.hs:70:48: 
  1) Iris, Cli Options, CI interactivity check
       expected: Interactive
        but got: NonInteractive

  To rerun use: --match "/Iris/Cli Options/CI interactivity check/"

[german1608]

Worth mentioning that cabal test --enable-tests --test-show-details=direct works fine.

[chshersh]

This is really weird 🤔 Looks like stack provides a different interactivity check for the test suite.

Not sure what happens here though. More investigation is required 🕵🏻

[marcellourbani]

Not that weird.I knew those tests were brittle as they make assumptions about the terminal environment of the test runner

The alternative is replicating the terminal property check in the test, or checking the environment

I'd like to have a look myself, don't think I'll find the time this week

[martinhelmer]

seems stack "turns off" stdin during test:

added a q <- getLine in the test and got

  1) Iris, Cli Options, CI interactivity check
       uncaught exception: IOException of type EOF
       <stdin>: hGetLine: end of file

when running stack test

(also here )

cabal just nicely asks for input and continues.

I would say the correct solution is to treat stack test same as CI. easiest would be if we could set the "CI" env when running stack test. Then checkCI == True and we and we get the right expectations.

[martinhelmer]

nope. doesn't work . setting CI breaks a bunch of other tests instead

 CI="a" stack test
iris> test (suite: iris-test)

Iris
  Cli Options
    help without version environment [✔]
    help with version environment [✔]
    --numeric-version returns correct version [✔]
    CI interactivity check [✔]
    Handles colour mode [✔]
    --version returns correct version text [✔]
  Cli Parser Conflicts
    --no-input=someValue defined by user - arg provided [✔]
    --no-input=someValue defined by user - no arg provided [✔]
    --no-input=someValue defined by user - not provided at all [✔]
    --no-input switch defined by user - provided [✔]
    --no-input switch defined by user - not provided [✔]
    --no-input switch with command defined by user - user provided [✔]
    --no-input switch with command defined by user - internal provided [✔]
  Colour
    Mode
      DisableColour when --no-colour [✔]
      EnableColour when --colour [✔]
      EnableColour in clear environment [✘]
      DisableColour when NO_COLOR is set [✔]
      DisableColour when NO_COLOUR is set [✔]
      DisableColour when MYAPP_NO_COLOR is set [✔]
      DisableColour when MYAPP_NO_COLOUR is set [✔]
      DisableColour when TERM=dumb [✔]
      EnableColour when TERM=xterm-256color [✘]
      DisableColour when CI is set [✘]
  Tool
    should find 'curl' [✔]
    shouldn't find 'xxx_unknown_executable' [✔]
    shouldn't find 'ghc' version 100 [✔]
    should not fail when 'need'ing 'curl' [✔]
    should fail when 'need'ing not found tools [✔]
    should fail when 'need'ing tools with wrong version [✔]

Failures:

  test/Test/Iris/Colour/Mode.hs:28:33:
  1) Iris.Colour.Mode EnableColour in clear environment
       expected: DisableColour
        but got: EnableColour

  To rerun use: --match "/Iris/Colour/Mode/EnableColour in clear environment/"

  test/Test/Iris/Colour/Mode.hs:59:33:
  2) Iris.Colour.Mode EnableColour when TERM=xterm-256color
       expected: DisableColour
        but got: EnableColour

  To rerun use: --match "/Iris/Colour/Mode/EnableColour when TERM=xterm-256color/"

  test/Test/Iris/Colour/Mode.hs:64:33:
  3) Iris.Colour.Mode DisableColour when CI is set
       expected: DisableColour
        but got: EnableColour

  To rerun use: --match "/Iris/Colour/Mode/DisableColour when CI is set/"

Randomized with seed 1290753107

Finished in 0.0872 seconds
29 examples, 3 failures

iris> Test suite iris-test failed
Test suite failure for package iris-0.1.0.0
    iris-test:  exited with: ExitFailure 1
Logs printed to console

[martinhelmer]

ok. fixed in stack 2.9.3

  Cli Options
    help without version environment [✔]
    help with version environment [✔]
    --numeric-version returns correct version [✔]
    CI interactivity check [✔]
    Handles colour mode [✔]
    --version returns correct version text [✔]

i think we can close it.

stack issue

[chshersh]

@martinhelmer Thanks for finding the relevant stack issue and confirming that it works now! 🎉

[marcellourbani]

For once my lazyness paid off :+1: