tar driver does not work on MacOS

[sluongng]

I am trying to run some basic fileContentTests and fileExistenceTests on MacOS with tar driver

schemaVersion: 2.0.0

fileContentTests:
  - name: "os-version"
    path: "/etc/os-release"
    expectedContents: [
      'NAME="CentOS Linux"',
      'ID="centos"',
      'ID_LIKE="rhel fedora"',
      'VERSION_ID="7"',
      'CPE_NAME="cpe:/o:centos:centos:7"',
      'HOME_URL="https://www.centos.org/"',
      'BUG_REPORT_URL="https://bugs.centos.org/"',
      'CENTOS_MANTISBT_PROJECT="CentOS-7"',
      'CENTOS_MANTISBT_PROJECT_VERSION="7"',
      'REDHAT_SUPPORT_PRODUCT="centos"',
      'REDHAT_SUPPORT_PRODUCT_VERSION="7"',
    ]

With this bazel test setup

container_test(
    name = "base-container-test-tar",
    size = "small",
    configs = ["testdata/base-container-test.yml"],
    driver = "tar",
    image = "@centos-7//image:image",
)

Which resulted in the following error

================================================
====== Test file: base-container-test.yml ======
================================================
time="2021-03-02T22:14:35Z" level=error msg="Error updating file permissions on /var/folders/l2/_dzk7hhs4gd9rpfb7c6hj63cc99qy8/T/..centos-7imageimage.tar@sha256:af9789f8f68cd80ff836d076542f4f22a519c43554c88d2a2710c174f29803a4312023790/usr/bin/chfn"
=== RUN: os-version
--- FAIL
duration: 0s
Error: error creating driver: processing tar image reference: getting filesystem for image: chmod /var/folders/l2/_dzk7hhs4gd9rpfb7c6hj63cc99qy8/T/..centos-7imageimage.tar@sha256:af9789f8f68cd80ff836d076542f4f22a519c43554c88d2a2710c174f29803a4312023790/usr/bin/chfn: operation not permitted

I am using bazel rules_docker on latest HEAD https://github.com/bazelbuild/rules_docker/commit/e15c9ebf203b7fa708e69ff5f1cdcf427d7edf6f

This test passed just fine when run on Linux + tar-driver or MacOS + docker-driver. But not MacOS + tar-driver.

[sluongng]

@nkubala I think you maintain both this project and https://github.com/GoogleContainerTools/container-diff, could you please take a look?

[nkubala]

hey @sluongng, thanks for the issue. our team isn't currently working on this project anymore, and we're not really submitting bug fixes ourselves. if you'd like to take a stab at fixing this issue, I'll try and take a look at any opened PRs!

[skirino]

I also encountered the same issue. IIUC this is not caused by container-structure-test itself; rather it comes from the bazel's sandbox implementation in macOS. Using container_test() defined in rules_docker I get the same error as in the PR description above, while directly invoking container-structure-test succeeds as expected.

My understanding:

• There are files/directories with setuid/setgid bit enabled in any of the target image layers.
• In macOS bazel tests are executed within sandbox-exec and the sandbox profile used does not contain file-write-setugid.
• During test execution, when container-structure-test tries to construct the image filesystem (under $TMPDIR), it fails to add setuid/setgid bit for the files/directories (operation not permitted).

Note that one can confirm that changing setuid/setgid of a file fails in a simple sh_test. I don't have any idea on how to fix/workaround this issue, but I hope this comment helps someone.

[sluongng]

Just FYI my current approach with this is to create 2 targets: 1 with compatible_with Linux and the other with MacOS. And use Docker driver on MacOS. It's a hack but it allow you to do some basic tests such as verifying the distro of the container OS.

@skirino interesting observation... I think bazel test tmpdir might not allow the operation for a good reason (but it does not make much sense why Linux is ok with it). Perhaps a workaround would be to write to the host's tmpdir directly, bypassing bazel tmpdir but then you will run into problems with tests running concurrently and having leftover artifacts on the host 🤔