test-sandbox-compose-0.1.1 test suite failure

[peti]

Citing from http://hydra.cryp.to/build/599429/nixlog/1/raw:

Test suite test: RUNNING...

Server Test
  help
  up FAILED [1]
  kill FAILED [2]
  up FAILED [3]
  destroy FAILED [4]

Failures:

  1) Server Test up
       expected: ""
        but got: "test-sandbox-compose: \nRan commands: \n\nException: ExitFailure 1\n"

  2) Server Test kill
       expected: ""
        but got: "test-sandbox-compose: FailedConnectionException2 \"localhost\" 23030 False connect: does not exist (Connection refused)\n"

  3) Server Test up
       expected: ""
        but got: "test-sandbox-compose: FailedConnectionException2 \"localhost\" 23030 False connect: does not exist (Connection refused)\n"

  4) Server Test destroy
       expected: ""
        but got: "test-sandbox-compose: FailedConnectionException2 \"localhost\" 23030 False connect: does not exist (Connection refused)\n"

Randomized with seed 2078520677

Finished in 10.1062 seconds
5 examples, 4 failures
Test suite test: FAIL

[junjihashimoto]

Thank you for testing. https://github.com/junjihashimoto/test-sandbox-compose/blob/master/.travis.yml#L27 Currently, test-sandbox-compose-daemon should run before cabal-test.

[peti]

In other words, cabal test is broken?

[junjihashimoto]

When setup is nothing before test, cabal test is broken. I improve test so that the setup is not needed.

[junjihashimoto]

I have improved test( add setup code to haskell-code). Version-bump this package.

[peti]

I re-enabled the test suite for the current version, and the errors I used to get are now gone. Unfortunately, there's plenty of new errors, because the test suite has undeclared dependencies on nc, etc.:

Failures:

  1) Basic Test Run nc : port 12181
       uncaught exception: IOException of type NoSuchThing (/tmp/nix-build-haskell-test-sandbox-0.1.3.drv-0/test_6970/ncfile16970: createProcess: runInteractiveProcess: exec: does not exist (No such file or directory))

  2) Basic Test isBinable' 12181
       expected: False
        but got: True

  3) Basic Test Test isBinable 12181
       expected: False
        but got: True

  4) Basic Test Test sendTo
       uncaught exception: IOException of type NoSuchThing (/tmp/nix-build-haskell-test-sandbox-0.1.3.drv-0/test_6970/ncfile6970: createProcess: runInteractiveProcess: exec: does not exist (No such file or directory))

  5) Basic Test : run
       uncaught exception: IOException of type NoSuchThing (/tmp/nix-build-haskell-test-sandbox-0.1.3.drv-0/test_6971/echoecho6970: createProcess: runInteractiveProcess: exec: does not exist (No such file or directory))

  6) signal test send kill signal for process groups
       uncaught exception: IOException of type NoSuchThing (/tmp/nix-build-haskell-test-sandbox-0.1.3.drv-0/test_6971/str26970: createProcess: runInteractiveProcess: exec: does not exist (No such file or directory))

  7) signal test send kill signal for process groups
       uncaught exception: IOException of type NoSuchThing (/tmp/nix-build-haskell-test-sandbox-0.1.3.drv-0/test_6971/str26970: createProcess: runInteractiveProcess: exec: does not exist (No such file or directory))

  8) signal test not send kill signal for process groups
       uncaught exception: IOException of type NoSuchThing (/tmp/nix-build-haskell-test-sandbox-0.1.3.drv-0/test_6971/str26970: createProcess: runInteractiveProcess: exec: does not exist (No such file or directory))

I'm not sure whether it's worth trying to fix these issues. It appears that the test suite is based on assumptions that really don't mix well with a batch-oriented CI system, like NixOS uses. It might be easiest to just keep the test suite disabled and to trust that users of the packaged will run it manually.

[peti]

Duh, I just realized that I ran the tests for the wrong package! Those results are for test-sandbox, of course. For test-sandbox-composable, the outcome looks like this:

Starting process daemon... InvalidYaml (Just (YamlException "Yaml file not found: sample/test-sandbox-compose.yml"))
Process daemon not running.
 - command-line: /tmp/nix-build-haskell-test-sandbox-compose-0.1.3.drv-0/test-sandbox-compose-0.1.3/dist/build/test-sandbox-compose/test-sandbox-compose daemon --conf sample/test-sandbox-compose.yml
 - exit code: ExitFailure 1
  run daemon FAILED [1]
Server Test
  help
  up FAILED [2]
  kill FAILED [3]
  up FAILED [4]
  destroy FAILED [5]

Failures:

  1) run daemon run daemon
       uncaught exception: IOException of type UserError (user error (Process daemon not running.
        - command-line: /tmp/nix-build-haskell-test-sandbox-compose-0.1.3.drv-0/test-sandbox-compose-0.1.3/dist/build/test-sandbox-compose/test-sandbox-compose daemon --conf sample/test-sandbox-compose.yml
        - exit code: ExitFailure 1))

  2) Server Test up
       expected: ""
        but got: "test-sandbox-compose: FailedConnectionException2 \"localhost\" 16968 False connect: does not exist (Connection refused)\n"

  3) Server Test kill
       expected: ""
        but got: "test-sandbox-compose: FailedConnectionException2 \"localhost\" 16968 False connect: does not exist (Connection refused)\n"

  4) Server Test up
       expected: ""
        but got: "test-sandbox-compose: FailedConnectionException2 \"localhost\" 16968 False connect: does not exist (Connection refused)\n"

  5) Server Test destroy
       expected: ""
        but got: "test-sandbox-compose: FailedConnectionException2 \"localhost\" 16968 False connect: does not exist (Connection refused)\n"

Randomized with seed 1544686696

Anyway, the conclusions I drew earlier probably still apply.

[junjihashimoto]

Thank you for comments. I agree with your opinion that assumptions are not well for tests of the CI system. I try to create another nc-server to prevent manual test.

[junjihashimoto]

I replace nc with runhaskell and https://gist.github.com/junjihashimoto/42f6519e9e63f7d66347.

[junjihashimoto]

I tried to update the test like this https://github.com/junjihashimoto/test-sandbox-compose/pull/4. But this is failed. I realize this is bad idea. You opinion is right. I should change so that users of the packaged can run it manually.