playbooks/system-test, test/system: Avoid running out of storage space

[debarshiray]

The system tests download several images when setting up the test suite, and cache them for later use by the tests [1]. This saves time and avoids hitting rate limits imposed by OCI registries by not downloading the same images repeatedly for several tests, but at the cost of increased use of storage space to cache the images.

The images are cached under BATS_TMPDIR. It defaults to the TMPDIR environment variable, and if that's not set then to /tmp [2]. Normally, TMPDIR isn't set, and the images end up getting cached under /tmp. Now, /tmp is typically on tmpfs backed by RAM or swap, which means that it should be used for smaller size-bounded files only, and /var/tmp should be used for everything else [3].

The images are big enough that a collection of them can't be described as smaller and size-bounded, and it led to:

  1..306
  # test suite: Set up
  # test suite: Tear down
  not ok 1 setup_suite
  # (from function `setup_suite' in test file ./setup_suite.bash, line
      55)
  #   `_pull_and_cache_distro_image fedora "$((system_version-1))" ||
      false' failed
  # Failed to cache image registry.fedoraproject.org/fedora-toolbox:40
      to /tmp/bats-run-IPz4Cn/image-cache/fedora-toolbox-40
  # time="2024-02-19T11:41:43Z" level=fatal msg="copying system image
      from manifest list: writing blob: write
      /tmp/bats-run-IPz4Cn/image-cache/fedora-toolbox-40/dir-put-blob607392514:
      no space left on device"
  # bats warning: Executed 1 instead of expected 306 tests

So, change the default location of the BATS_TMPDIR environment variable to /var/tmp by setting TMPDIR.

[1] Commit 50683c9d9a78adc9 https://github.com/containers/toolbox/commit/50683c9d9a78adc9 https://github.com/containers/toolbox/pull/375

[2] https://bats-core.readthedocs.io/en/stable/writing-tests.html

[3] https://systemd.io/TEMPORARY_DIRECTORIES/

[softwarefactory-project-zuul[bot]]

Build succeeded. https://softwarefactory-project.io/zuul/t/local/buildset/8be2e3e991de45ba95a149688b35b9ea

:heavy_check_mark: unit-test SUCCESS in 5m 09s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 4m 10s :heavy_check_mark: unit-test-restricted SUCCESS in 3m 59s :heavy_check_mark: system-test-fedora-rawhide SUCCESS in 41m 15s :heavy_check_mark: system-test-fedora-39 SUCCESS in 41m 33s :heavy_check_mark: system-test-fedora-38 SUCCESS in 42m 27s

[HarryMichal]

Nice catch. I just wonder whether we should set the variable only in the CI or make it a requirement when invoking the tests (therefore also update the README).

[debarshiray]

I just wonder whether we should set the variable only in the CI or make it a requirement when invoking the tests (therefore also update the README).

Good point. Yes, let's mention it in test/system/README.md.

[softwarefactory-project-zuul[bot]]

Build succeeded. https://softwarefactory-project.io/zuul/t/local/buildset/ca484b0fc5ae470f805e092761915e01

:heavy_check_mark: unit-test SUCCESS in 6m 46s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 48s :heavy_check_mark: unit-test-restricted SUCCESS in 6m 02s :heavy_check_mark: system-test-fedora-rawhide SUCCESS in 33m 03s :heavy_check_mark: system-test-fedora-39 SUCCESS in 36m 05s :heavy_check_mark: system-test-fedora-38 SUCCESS in 35m 57s

[softwarefactory-project-zuul[bot]]

Build succeeded. https://softwarefactory-project.io/zuul/t/local/buildset/100d3ff1a3f148578feba2ff4dda7b1b

:heavy_check_mark: unit-test SUCCESS in 4m 41s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 22s :heavy_check_mark: unit-test-restricted SUCCESS in 3m 50s :heavy_check_mark: system-test-fedora-rawhide SUCCESS in 32m 28s :heavy_check_mark: system-test-fedora-39 SUCCESS in 36m 41s :heavy_check_mark: system-test-fedora-38 SUCCESS in 33m 58s

[softwarefactory-project-zuul[bot]]

Build succeeded. https://softwarefactory-project.io/zuul/t/local/buildset/d82a88370d374634aa21f384202a2f0f

:heavy_check_mark: unit-test SUCCESS in 4m 43s :heavy_check_mark: unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 19s :heavy_check_mark: unit-test-restricted SUCCESS in 3m 45s :heavy_check_mark: system-test-fedora-rawhide SUCCESS in 34m 34s :heavy_check_mark: system-test-fedora-39 SUCCESS in 33m 53s :heavy_check_mark: system-test-fedora-38 SUCCESS in 34m 39s