search

containers / toolbox

Tool for interactive command line environments on Linux
https://containertoolbx.org/
Apache License 2.0
2.39k stars 208 forks source link

test/system: Make it easier to spot failures to download & cache images #1377

Closed debarshiray closed 9 months ago

debarshiray commented 9 months ago

Currently, if skopeo copy ... fails to download and cache an OCI image during setup_suite(), the test suite doesn't immediately fail, but continues. It only fails later when trying to set up the Docker registry and contains a lot of noise:

  not ok 1 setup_suite
  # (from function `assert_success' in file
       test/system/libs/bats-assert/src/assert.bash, line 114,
  #  from function `_setup_docker_registry' in file
       test/system/libs/helpers.bash, line 211,
  #  from function `setup_suite' in test file
       test/system/setup_suite.bash, line 59)
  #   `_setup_docker_registry' failed
  # Failed to cache image registry.fedoraproject.org/fedora-toolbox:38
      to /tmp/bats-run-GyTP7A/image-cache/fedora-toolbox-38
  #
  # -- command failed --
  # status : 1
  # output : time="2023-09-25T12:19:52+02:00" level=fatal
      msg="initializing source
        docker://registry.fedoraproject.org/fedora-toolbox:38-foo:
        reading manifest 38-foo in
        registry.fedoraproject.org/fedora-toolbox: manifest unknown"
  # --
  #
  # Failed to cache image quay.io/toolbx/arch-toolbox:latest to
      /tmp/bats-run-GyTP7A/image-cache/arch-toolbox-latest
  #
  # -- command failed --
  # status : 1
  # output : time="2023-09-25T12:20:48+02:00" level=fatal
      msg="initializing source
        docker://quay.io/toolbx/arch-toolbox:latest-foo: reading
        manifest latest-foo in quay.io/toolbx/arch-toolbox: manifest
        unknown"
  # --
  #
  # Failed to cache image registry.fedoraproject.org/fedora-toolbox:34
      to /tmp/bats-run-GyTP7A/image-cache/fedora-toolbox-34
  #
  # -- command failed --
  # status : 1
  # output : time="2023-09-25T12:21:42+02:00" level=fatal
      msg="initializing source
        docker://registry.fedoraproject.org/fedora-toolbox:34-foo:
        reading manifest 34-foo in
        registry.fedoraproject.org/fedora-toolbox: manifest unknown"
  # --
  #
  # ...
  #
  # -- command failed --
  # status : 1
  # output : time="2023-09-25T12:26:33+02:00" level=fatal
      msg="determining manifest MIME type for
        dir:/tmp/bats-run-GyTP7A/image-cache/fedora-toolbox-34: open
        /tmp/bats-run-GyTP7A/image-cache/fedora-toolbox-34/manifest.json:
        no such file or directory"
  # --
  #
  # docker-registry
  # 27fa141e291e64e4c7a148c88ddab219ff2bfb5802a2982dc4188dc11f41692d
  # Untagged: quay.io/toolbox_tests/registry:latest
  # Deleted: fea5a12cde107bb407bc44ede6dd9edea1d2b4171cd8e52b0cb330bf45e517e1

It makes it look as if the root cause of the failure is related to setting up the Docker registry, which it isn't, and all that noise makes it difficult to spot the actual problem.

Instead, from now on, it will be more obvious:

  not ok 1 setup_suite
  # (from function `setup_suite' in test file
       test/system/setup_suite.bash, line 44)
  #   `_pull_and_cache_distro_image "$system_id" "$system_version" ||
         false' failed
  # Failed to cache image registry.fedoraproject.org/fedora-toolbox:38
      to /tmp/bats-run-62b8CU/image-cache/fedora-toolbox-38
  # time="2023-09-25T13:55:42+02:00" level=fatal msg="initializing
      source docker://registry.fedoraproject.org/fedora-toolbox:38-foo:
      reading manifest 38-foo in
      registry.fedoraproject.org/fedora-toolbox: manifest unknown"

Note that Bats' run helper [1] isn't designed to work inside setup_suite(). eg., run --separate-stderr doesn't work because BATS_TEST_TMPDIR isn't defined.

[1] https://bats-core.readthedocs.io/en/stable/writing-tests.html

softwarefactory-project-zuul[bot] commented 9 months ago

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/0cf8e12bf56e425c9d0dd195b0971f82

:heavy_check_mark: unit-test SUCCESS in 8m 15s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 14s :heavy_check_mark: unit-test-restricted SUCCESS in 7m 13s :x: system-test-fedora-rawhide FAILURE in 31m 41s :heavy_check_mark: system-test-fedora-38 SUCCESS in 25m 04s :heavy_check_mark: system-test-fedora-37 SUCCESS in 23m 36s

softwarefactory-project-zuul[bot] commented 9 months ago

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/2f758ac1f14b4bacbc0d8b354a0edb09

:heavy_check_mark: unit-test SUCCESS in 8m 48s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 19s :heavy_check_mark: unit-test-restricted SUCCESS in 7m 39s :x: system-test-fedora-rawhide FAILURE in 33m 17s :heavy_check_mark: system-test-fedora-38 SUCCESS in 24m 52s :heavy_check_mark: system-test-fedora-37 SUCCESS in 23m 33s

softwarefactory-project-zuul[bot] commented 9 months ago

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/e707c145d8fe459eab3d3b22cf42fbdb

:heavy_check_mark: unit-test SUCCESS in 8m 09s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 33s :heavy_check_mark: unit-test-restricted SUCCESS in 7m 15s :x: system-test-fedora-rawhide FAILURE in 32m 09s :heavy_check_mark: system-test-fedora-38 SUCCESS in 23m 25s :heavy_check_mark: system-test-fedora-37 SUCCESS in 22m 33s

softwarefactory-project-zuul[bot] commented 9 months ago

Build failed. https://softwarefactory-project.io/zuul/t/local/buildset/d33a767dbd5e40718631861ad9fd0c14

:heavy_check_mark: unit-test SUCCESS in 8m 12s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 4m 19s :heavy_check_mark: unit-test-restricted SUCCESS in 7m 08s :x: system-test-fedora-rawhide FAILURE in 32m 35s :heavy_check_mark: system-test-fedora-38 SUCCESS in 24m 12s :heavy_check_mark: system-test-fedora-37 SUCCESS in 23m 10s

debarshiray commented 9 months ago

There are still some test failures on Fedora Rawhide. For example:

fedora-rawhide | not ok 3 help: Run command 'help' in 145ms
fedora-rawhide | # (from function `assert_line' in file test/system/libs/bats-assert/src/assert.bash, line 479,
fedora-rawhide | #  in test file test/system/002-help.bats, line 45)
fedora-rawhide | #   `assert_line --index 0 --partial "toolbox(1)"' failed
fedora-rawhide | # /usr/bin/man
fedora-rawhide | #
fedora-rawhide | # -- line does not contain substring --
fedora-rawhide | # index     : 0
fedora-rawhide | # substring : toolbox(1)
fedora-rawhide | # line      : troff:<standard input>:33: warning: cannot select font 'C'
fedora-rawhide | # --

I believe these are because of changes in various other components in Fedora 39, which we need to track down one by one and work out a fix.

In the mean time, I am going to temporarily override these failures.