openbmc / openbmc-build-scripts

Apache License 2.0
19 stars 50 forks source link

build-unit-test-docker: Permission denied: '/usr/lib/python3.9/__pycache__/__future__.cpython-39.pyc #32

Closed geissonator closed 2 years ago

geissonator commented 2 years ago

I recently got a new Ubuntu 20 system and I can't seem to get build-unit-test-docker to build on this system.

System Info:

NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
andrewg@ubuntu20:~/Code/openbmc-build-scripts$ uname -a
Linux ubunto20 5.4.0-84-generic #94-Ubuntu SMP Thu Aug 26 20:27:37 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
base: Preparing to unpack .../libpython3.9-minimal_3.9.5-3~21.04_amd64.deb ...
base: Unpacking libpython3.9-minimal:amd64 (3.9.5-3~21.04) ...
base: Selecting previously unselected package libexpat1:amd64.
base: Preparing to unpack .../libexpat1_2.2.10-2_amd64.deb ...
base: Unpacking libexpat1:amd64 (2.2.10-2) ...
base: Selecting previously unselected package python3.9-minimal.
base: Preparing to unpack .../python3.9-minimal_3.9.5-3~21.04_amd64.deb ...
base: Unpacking python3.9-minimal (3.9.5-3~21.04) ...
base: Setting up libpython3.9-minimal:amd64 (3.9.5-3~21.04) ...
base: Setting up libexpat1:amd64 (2.2.10-2) ...
base: Setting up python3.9-minimal (3.9.5-3~21.04) ...
base: Traceback (most recent call last):
base:   File "/usr/lib/python3.9/py_compile.py", line 215, in <module>
base:     sys.exit(main())
base:   File "/usr/lib/python3.9/py_compile.py", line 207, in main
base:     compile(filename, doraise=True)
base:   File "/usr/lib/python3.9/py_compile.py", line 172, in compile
base:     importlib._bootstrap_external._write_atomic(cfile, bytecode, mode)
base:   File "<frozen importlib._bootstrap_external>", line 186, in _write_atomic
base: PermissionError: [Errno 13] Permission denied: '/usr/lib/python3.9/__pycache__/__future__.cpython-39.pyc.140420583859120'
base: dpkg: error processing package python3.9-minimal (--configure):
base:  installed python3.9-minimal package post-installation script subprocess returned error exit status 1
base: Errors were encountered while processing:
base:  python3.9-minimal
base: E: Sub-process /usr/bin/dpkg returned an error code (1)
base: Removing intermediate container 92a5fbc075ed
Traceback (most recent call last):
  File "./scripts/build-unit-test-docker", line 846, in <module>
    Docker.build("base", docker_base_img_name, dockerfile_base)
  File "./scripts/build-unit-test-docker", line 629, in build
    docker.build(
  File "/usr/lib/python3/dist-packages/sh.py", line 1427, in __call__
    return RunningCommand(cmd, call_args, stdin, stdout, stderr)
  File "/usr/lib/python3/dist-packages/sh.py", line 774, in __init__
    self.wait()
  File "/usr/lib/python3/dist-packages/sh.py", line 792, in wait
    self.handle_command_exit_code(exit_code)
  File "/usr/lib/python3/dist-packages/sh.py", line 815, in handle_command_exit_code
    raise exc
sh.ErrorReturnCode_100: 

I've tried a fresh run of the script on my Ubuntu 18 system without issue so I'm not sure what's going on here. The issue looks like https://forums.docker.com/t/bug-on-apt-install-permission-denied/100196 which only seemed to hit certain machines as well.

I've tried a few things like ensuring all packages are up to date, rebooting the machine, and hacking at the script a bit but so far, no dice.

The non-working system is running Docker version 20.10.8. The working system has 20.10.7 but I don't see any obvious changes in the change log. Downleveling may be the next move though.

williamspatrick commented 2 years ago

This sounds vaguely familiar, like we ran into this before on the fb Jenkins instances...

I think there was an update to the runc package needed in order to get newer Ubuntu images running on certain versions of docker. Can you get the docker-runc --version?

geissonator commented 2 years ago

Ahh, good point, that's bringing back some memories. It seems on more recent releases they dropped the docker- part and it's just "runc". My working system has this:

$ runc --version
runc version 1.0.1
commit: v1.0.1-0-g4144b63
spec: 1.0.2-dev
go: go1.15.14
libseccomp: 2.5.1

My non-working system doesn't even have it installed. When I do install it I get an older version that doesn't seem to help. I'm working on trying to get the newer one installed but have apparently reached my docker pull limits (even logged in).

mdmillerii commented 2 years ago

https://www.hashbangcode.com/article/permission-denied-error-when-first-running-docker-ubuntu-2004

runc ran into a container / libc fallback vs host kernel issue

geissonator commented 2 years ago

This fixed it up for me. I basically had to install the docker-ce version vs. the one that comes with Ubuntu.

# https://stackoverflow.com/questions/45023363/what-is-docker-io-in-relation-to-docker-ce-and-docker-ee
# https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-20-04 
sudo apt-get remove docker docker-engine docker.io containerd runc
sudo apt install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"
sudo apt update
sudo apt install docker-ce