Lockfile generation fails in Docker containers

[maxrake]

Overview

Lockfile generation with the v5.9.0-rc1 release fails when run from a Debian based linux/amd64 Docker container.

How To Reproduce

# Confirm the current version
❯ phylum --version
phylum v5.9.0-rc1

# Create or use a manifest
❯ cat requirements.txt
pyyaml

# See that it parses outside of a Docker container
❯ phylum parse --lockfile-type pip ./requirements.txt
Generating lockfile for manifest "requirements.txt" using Pip…
[
  {
    "name": "PyYAML",
    "version": "6.0.1",
    "type": "pypi",
    "lockfile": "requirements.txt"
  }
]

# Prove that the same version of Phylum is installed in a Docker container
❯ docker run -it --rm -e PHYLUM_API_KEY=$(phylum auth token) --mount type=bind,src=$(pwd),dst=/phylum -w /phylum maxrake/phylum-ci:gh_pr_target phylum --version
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
phylum v5.9.0-rc1

# See that it fails to parse inside of a Docker container
❯ docker run -it --rm -e PHYLUM_API_KEY=$(phylum auth token) --mount type=bind,src=$(pwd),dst=/phylum -w /phylum maxrake/phylum-ci:gh_pr_target phylum parse --lockfile-type pip
./requirements.txt
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Generating lockfile for manifest "requirements.txt" using Pip…
❗ Error: could not parse lockfile: ./requirements.txt

Caused by:
    input/output error: Invalid argument (os error 22)

Expected Behavior

Lockfiles are able to be generated by Phylum when operating within a Debian based linux/amd64 Docker container (which is what phylumio/phylum-ci on Docker Hub is).

Additional Context

This appears to affect the CLI v5.9.0-rc1 release only at this time. The previous v5.8.1 release does not have this issue. The difference between those two releases is generating lockfiles within a sandbox. It is not clear from the error what went wrong, but the suspicion is a missing sandbox permission for the linux/amd64 Docker environment.

The maxrake/phylum-ci:gh_pr_target tag on Docker Hub exists to test this behavior.

[kylewillmon]

Docker heavily restricts the unshare syscall which is required for Birdcage to function.

If you have control over the execution of the container, you can try modifying the docker run command to make it work. It might just need --cap-add=CAP_SYS_ADMIN, but the docs are a bit unclear on user namespaces and I have not tested that... Alternately, you could use a custom seccomp filter with --security-opt="seccomp=profile.json" or even disable the seccomp filter entirely with --security-opt="seccomp=unconfined"

Ultimately, this is the reason that we included a --skip-sanbox option in 732b0d3. When docker is in use, further sandboxing is impractical and potentially unnecessary.

[maxrake]

Knowing when to skip using the sandbox (at runtime) can be done in several ways:

• Check for when Docker is in use
  • PRO: easy, by checking for the existence of a /.dockerenv file
  • CON: disabling use of the sandbox is possible by creating that file
  • CON: Docker may not be the only environment where the sandbox does not work
• Attempt to use the sandbox for something that should always succeed
  • PRO: can be used for any environment and will survive implementation changes to both Phylum and Docker
  • CON: it may still be possible to manipulate the environment to cause a failure and thus control the use of the sandbox

Discussion with the team on 28 NOV 2023 resulted in the decision to go with the second option. In particular, the command to use was determined to be:

phylum sandbox --allow-run / true

If that command fails, then it must be due to the environment not playing well with the Birdcage sandbox. Only then should the use of the sandbox be skipped.