search

containers / toolbox

Tool for interactive command line environments on Linux
https://containertoolbx.org/
Apache License 2.0
2.39k stars 208 forks source link

test/system: Make it easier to debug 'podman logs' failures #1372

Closed debarshiray closed 9 months ago

debarshiray commented 9 months ago

Bats' run helper returns with an exit code of 0 even when the command that it was given to run failed with a non-zero exit code [1]. This is to enable making further assertions about the command after run has finished. If there's nothing that checks for failures, then it will continue as if everything is alright.

Therefore, currently, if podman logs fails, there's no indication of it and the test only fails later because it thinks that the container failed to initialize:

  not ok 106 container: Check container starts without issues
  # (from function `assert_success' in file
       test/system/libs/bats-assert/src/assert.bash, line 114,
  #  in test file test/system/103-container.bats, line 39)
  #   `assert_success' failed
  #
  # -- command failed --
  # status : 1
  # output :
  # --
  #

Instead, from now on, it will be more obvious:

  not ok 106 container: Check container starts without issues
  # (from function `assert_success' in file
       test/system/libs/bats-assert/src/assert.bash, line 114,
  #  in test file test/system/103-container.bats, line 39)
  #   `assert_success' failed
  #
  # -- command failed --
  # status : 125
  # output (2 lines):
  #   Failed to invoke '/usr/bin/podman logs'
  #   Error: no container with name or ID "foo" found: no such container
  # --
  #

One alternative was to use assert_success [2] to assert that the command given to run succeeded. That would show the podman logs failure as:

  not ok 106 container: Check container starts without issues
  # (from function `assert_success' in file
       test/system/libs/bats-assert/src/assert.bash, line 114,
  #  in test file test/system/103-container.bats, line 39)
  #   `assert_success' failed
  #
  # -- command failed --
  # status : 1
  # output (29 lines):
  #
  #   -- command failed --
  #   status : 125
  #   output : Error: no container with name or ID "foo" found: no such
        container
  #   --
  #
  # ...
  #
  #   -- command failed --
  #   status : 125
  #   output : Error: no container with name or ID "foo" found: no such
        container
  #   --
  # --
  #

However, it's a bit too noisy because of the assert_success not terminating container_started() and continuing to loop for the remaining attempts.

[1] https://bats-core.readthedocs.io/en/stable/writing-tests.html

[2] https://github.com/bats-core/bats-assert

softwarefactory-project-zuul[bot] commented 9 months ago

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/748dc14676db459dbd1a41ba695d9c6e

:heavy_check_mark: unit-test SUCCESS in 9m 21s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 09s :heavy_check_mark: unit-test-restricted SUCCESS in 7m 24s :x: system-test-fedora-rawhide FAILURE in 31m 48s :x: system-test-fedora-38 TIMED_OUT in 40m 32s :x: system-test-fedora-37 TIMED_OUT in 40m 32s

softwarefactory-project-zuul[bot] commented 9 months ago

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/c545b217b31f4295be67b457c81c3c0a

:heavy_check_mark: unit-test SUCCESS in 8m 14s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 08s :heavy_check_mark: unit-test-restricted SUCCESS in 7m 27s :x: system-test-fedora-rawhide FAILURE in 32m 21s :heavy_check_mark: system-test-fedora-38 SUCCESS in 24m 05s :heavy_check_mark: system-test-fedora-37 SUCCESS in 23m 14s

softwarefactory-project-zuul[bot] commented 9 months ago

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/938329cff6244477a124685f74b245c0

:heavy_check_mark: unit-test SUCCESS in 8m 51s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 08s :heavy_check_mark: unit-test-restricted SUCCESS in 7m 43s :x: system-test-fedora-rawhide FAILURE in 39m 33s :heavy_check_mark: system-test-fedora-38 SUCCESS in 33m 53s :heavy_check_mark: system-test-fedora-37 SUCCESS in 32m 56s

softwarefactory-project-zuul[bot] commented 9 months ago

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/583fe98ae2064f2cb3cdde8b5c7629e0

:heavy_check_mark: unit-test SUCCESS in 8m 24s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 43s :heavy_check_mark: unit-test-restricted SUCCESS in 7m 47s :x: system-test-fedora-rawhide FAILURE in 36m 18s :heavy_check_mark: system-test-fedora-38 SUCCESS in 29m 32s :heavy_check_mark: system-test-fedora-37 SUCCESS in 28m 44s

softwarefactory-project-zuul[bot] commented 9 months ago

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/f62ecea8c6d34bf8b8f3b891ac893d9c

:heavy_check_mark: unit-test SUCCESS in 9m 01s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 4m 46s :heavy_check_mark: unit-test-restricted SUCCESS in 7m 50s :x: system-test-fedora-rawhide FAILURE in 33m 01s :heavy_check_mark: system-test-fedora-38 SUCCESS in 26m 00s :heavy_check_mark: system-test-fedora-37 SUCCESS in 25m 47s

softwarefactory-project-zuul[bot] commented 9 months ago

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/eecf9f96f48a47329bcda295e3db01e4

:heavy_check_mark: unit-test SUCCESS in 8m 06s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 45s :heavy_check_mark: unit-test-restricted SUCCESS in 7m 32s :x: system-test-fedora-rawhide FAILURE in 34m 23s :heavy_check_mark: system-test-fedora-38 SUCCESS in 27m 16s :heavy_check_mark: system-test-fedora-37 SUCCESS in 23m 53s

softwarefactory-project-zuul[bot] commented 9 months ago

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/df2f77409a7848bc809a20ebe87b5c45

:heavy_check_mark: unit-test SUCCESS in 8m 12s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 21s :heavy_check_mark: unit-test-restricted SUCCESS in 7m 11s :x: system-test-fedora-rawhide FAILURE in 35m 09s :heavy_check_mark: system-test-fedora-38 SUCCESS in 27m 37s :heavy_check_mark: system-test-fedora-37 SUCCESS in 27m 22s

debarshiray commented 9 months ago

There are still some test failures on Fedora Rawhide. For example:

fedora-rawhide | not ok 3 help: Run command 'help' in 145ms
fedora-rawhide | # (from function `assert_line' in file test/system/libs/bats-assert/src/assert.bash, line 479,
fedora-rawhide | #  in test file test/system/002-help.bats, line 45)
fedora-rawhide | #   `assert_line --index 0 --partial "toolbox(1)"' failed
fedora-rawhide | # /usr/bin/man
fedora-rawhide | #
fedora-rawhide | # -- line does not contain substring --
fedora-rawhide | # index     : 0
fedora-rawhide | # substring : toolbox(1)
fedora-rawhide | # line      : troff:<standard input>:33: warning: cannot select font 'C'
fedora-rawhide | # --

I believe these are because of changes in various other components in Fedora 39, which we need to track down one by one and work out a fix.

In the mean time, I am going to temporarily override these failures.