test/system: Ensure failure if an invalid distribution is specified

[debarshiray]

Contrary to what the documentation might seem to imply [1], Bats' fai' helper only aborts a test case under certain circumstances. eg., when called from setup_suite(), but not from within a child function, and a @test case, but not from within the run helper.

If fail is called from within run, then the code after it will continue to execute. The test case will only fail if run eventually catches a non-zero exit code that's caught by assert_success [2]. Similarly, it doesn't abort if called from within a child function in setup_suite().

Currently, _pull_and_cache_distro_image() is a child function called from setup_suite(). So fail won't abort if an invalid distribution is specified.

Fortunately, pull_distro_image() is being called from within @test cases, but outside run. So, there's no problem with it now. However, some future code changes can unknowingly alter this reality and it too can run into unexpected behaviour.

Therefore, it's better to be safe, and explicitly specify a non-zero exit code after fail. It will ensure that it works as expected under all circumstances.

[1] https://github.com/bats-core/bats-support

[2] https://github.com/bats-core/bats-assert

[softwarefactory-project-zuul[bot]]

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/b92aceb37bf64133b7a57eb29326278c

:heavy_check_mark: unit-test SUCCESS in 8m 35s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 46s :heavy_check_mark: unit-test-restricted SUCCESS in 7m 20s :x: system-test-fedora-rawhide FAILURE in 33m 43s :heavy_check_mark: system-test-fedora-38 SUCCESS in 27m 13s :heavy_check_mark: system-test-fedora-37 SUCCESS in 25m 54s

[softwarefactory-project-zuul[bot]]

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/39cdcc92ca5a4fb1a3d793797ce0edaf

:heavy_check_mark: unit-test SUCCESS in 8m 24s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 15s :heavy_check_mark: unit-test-restricted SUCCESS in 7m 29s :x: system-test-fedora-rawhide FAILURE in 33m 48s :heavy_check_mark: system-test-fedora-38 SUCCESS in 25m 58s :heavy_check_mark: system-test-fedora-37 SUCCESS in 25m 53s

[softwarefactory-project-zuul[bot]]

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/3301603f6e3c4fcd9ea5fc991ca31715

:heavy_check_mark: unit-test SUCCESS in 8m 02s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 15s :heavy_check_mark: unit-test-restricted SUCCESS in 7m 15s :x: system-test-fedora-rawhide FAILURE in 32m 21s :heavy_check_mark: system-test-fedora-38 SUCCESS in 24m 36s :heavy_check_mark: system-test-fedora-37 SUCCESS in 23m 46s

[softwarefactory-project-zuul[bot]]

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/83f3fb5fb93b42d6b501b7824c103b83

:heavy_check_mark: unit-test SUCCESS in 8m 35s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 27s :heavy_check_mark: unit-test-restricted SUCCESS in 7m 30s :x: system-test-fedora-rawhide FAILURE in 33m 51s :heavy_check_mark: system-test-fedora-38 SUCCESS in 26m 16s :heavy_check_mark: system-test-fedora-37 SUCCESS in 24m 25s

[debarshiray]

There are still some test failures on Fedora Rawhide. For example:

fedora-rawhide | not ok 3 help: Run command 'help' in 145ms
fedora-rawhide | # (from function `assert_line' in file test/system/libs/bats-assert/src/assert.bash, line 479,
fedora-rawhide | #  in test file test/system/002-help.bats, line 45)
fedora-rawhide | #   `assert_line --index 0 --partial "toolbox(1)"' failed
fedora-rawhide | # /usr/bin/man
fedora-rawhide | #
fedora-rawhide | # -- line does not contain substring --
fedora-rawhide | # index     : 0
fedora-rawhide | # substring : toolbox(1)
fedora-rawhide | # line      : troff:<standard input>:33: warning: cannot select font 'C'
fedora-rawhide | # --

I believe these are because of changes in various other components in Fedora 39, which we need to track down one by one and work out a fix.

In the mean time, I am going to temporarily override these failures.