search

docker-library / python

Docker Official Image packaging for Python
https://www.python.org/
MIT License
2.56k stars 1.07k forks source link

`python:3.9` - Failed run `apt update` from the last version of the image #837

Closed aviadpriel closed 11 months ago

aviadpriel commented 1 year ago

as titled

today morning we found that our builds failed
we are using python:3.9

when we run apt update the build failed

it looks that it fails because of the upgrade to bookworm

we get this errors

#5 [build 2/4] RUN apt-get update &&     apt-get install -y --no-install-recommends     poppler-utils
#5 sha256:fdb97fe2ebc3fcf79626c6628756f871cc0d6b0fc02ba21a2949fe33d31a2373
#5 0.240 Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
#5 0.247 Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
#5 0.248 Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
#5 0.277 Err:1 http://deb.debian.org/debian bookworm InRelease
#5 0.277   The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 648ACFD622F3D138 NO_PUBKEY 0E98404D386FA1D9 NO_PUBKEY F8D2585B8783D481
#5 0.303 Err:2 http://deb.debian.org/debian bookworm-updates InRelease
#5 0.303   The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 0E98404D386FA1D9 NO_PUBKEY 6ED0E7B82643E131
#5 0.329 Err:3 http://deb.debian.org/debian-security bookworm-security InRelease
#5 0.329   The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 54404762BBB6E853 NO_PUBKEY BDE6D2B9216EC7A8
#5 0.330 Reading package lists...
#5 0.337 W: GPG error: http://deb.debian.org/debian bookworm InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 648ACFD622F3D138 NO_PUBKEY 0E98404D386FA1D9 NO_PUBKEY F8D2585B8783D481
#5 0.337 E: The repository 'http://deb.debian.org/debian bookworm InRelease' is not signed.
#5 0.337 W: GPG error: http://deb.debian.org/debian bookworm-updates InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 0E98404D386FA1D9 NO_PUBKEY 6ED0E7B82643E131
#5 0.337 E: The repository 'http://deb.debian.org/debian bookworm-updates InRelease' is not signed.
#5 0.337 W: GPG error: http://deb.debian.org/debian-security bookworm-security InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 54404762BBB6E853 NO_PUBKEY BDE6D2B9216EC7A8
#5 0.337 E: The repository 'http://deb.debian.org/debian-security bookworm-security InRelease' is not signed.
#5 0.337 E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
#5 0.337 E: Sub-process returned an error cod
Cookiehook commented 1 year ago

We're seeing the same issues on python:3.10. Worked around by using python:3.10-bullseye for now.

aviadpriel commented 1 year ago

yes, same for python:3.9-bullseye

but we have couple of security fixes on Debian 12 that we want to get

a1723 commented 1 year ago

The main question is: Will it be fixed in python:xx images or now we must always used images with "bullseye" postfix ?

yosifkit commented 1 year ago

I am unable to reproduce. I'd suggest updating docker and libseccomp on the host. Newer base OS's use newer system calls and an older libseccomp can block them since they are unknown to it. You can verify that it is libseccomp by running the bookworm image with --security-opt seccomp=unconfined.

This is similar to the update to Ubuntu focal: https://github.com/docker-library/mongo/issues/606#issuecomment-1424965831

$ docker run -it python:3.9-bookworm bash
Unable to find image 'python:3.9-bookworm' locally
3.9-bookworm: Pulling from library/python
bba7bb10d5ba: Already exists
ec2b820b8e87: Already exists
284f2345db05: Already exists
fea23129f080: Already exists
7c62c924b8a6: Already exists
b2210932934e: Pull complete
ee9c01829d92: Pull complete
d6285f41f1b6: Pull complete
Digest: sha256:98f018a1afd67f2e17a4abd5bfe09b998734ba7c1ee54780e7ed216f8b8095c3
Status: Downloaded newer image for python:3.9-bookworm
root@0b679f272394:/# apt-get update
Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8904 kB]
Get:5 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [24.2 kB]
Fetched 9176 kB in 1s (7771 kB/s)
Reading package lists... Done
root@0b679f272394:/#
InterferencePattern commented 1 year ago

I see the same for python:3.8-slim and others after the update from buster -> bookworm. I think the image needs an update of the GPG public keys for the bookworm apt repos.

EDIT: Until this is fixed, you can use the -buster versions of images, such as python:3.8-slim-buster

jmmclean commented 1 year ago

hitting us as well, using python:3.7

keever commented 1 year ago

This broke our ci docker build jobs that used python:3.9-slim We used docker v24.0.2 to do the build.

muasif80 commented 1 year ago

It seems like the pointers for the latest image tags are moved to bookworm. The python:3.9-slim-bullseye might work instead of python:3.9-slim or maybe python:3.9-slim-buster will work too. The official 3.9 image github update shows that the version is updated from buster to bookworm.

jacobeturpin commented 1 year ago

Can confirm that our builds were also impacted using python:3.8-slim. Modifying to python:3.8-slim-bullseye resolved the issue, so we're temporarily using that until the issue is resolved.

mitsuo0114 commented 1 year ago

Same here in out CI tool. I could solved with downgrading.

I'm assuming following change in Debian may have some affect to CI. (I haven't checked deeply)

https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#python3-pep-668

5.2.2. Python Interpreters marked externally-managed The Debian provided python3 interpreter packages (python3.11 and pypy3) are now marked as being externally-managed, following PEP-668. The version of python3-pip provided in Debian follows this, and ?will refuse to manually install packages on Debian's python interpreters, unless the --break-system-packages option is specified.

If you need to install a Python application (or version) that isn't packaged in Debian, we recommend that you install it with pipx (in the pipx Debian package). pipx will set up an environment isolated from other applications and system Python modules, and install the application and its dependencies into that.

aebm commented 1 year ago

Hi, that is a "problem" with the change of the debian base image used. It seems related to seccomp and it is mentioned here https://github.com/debuerreotype/docker-debian-artifacts/issues/197#issuecomment-1590169930

Example (PS: Don't run you containers with more privileges that necessary =) )

docker run --name=dummy --rm -ti python:3.11-slim bash -c 'apt-get update && apt-get install -y less && echo OK'
Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8904 kB]
Get:5 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [28.3 kB]
Fetched 9180 kB in 2s (4697 kB/s)                         
Reading package lists... Done
E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
E: Sub-process returned an error code

vs

docker run --name=dummy --rm -ti --privileged python:3.11-slim bash -c 'apt-get update && apt-get install -y less && echo OK'
Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8904 kB]
Get:5 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [28.3 kB]
Fetched 9180 kB in 2s (4844 kB/s)                          
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  less
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 131 kB of archives.
After this operation, 321 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bookworm/main amd64 less amd64 590-2 [131 kB]
Fetched 131 kB in 0s (3313 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package less.
(Reading database ... 8386 files and directories currently installed.)
Preparing to unpack .../archives/less_590-2_amd64.deb ...
Unpacking less (590-2) ...
Setting up less (590-2) ...
OK
aebm commented 1 year ago

I can confirm that it "works" when unconfined. Don't try this at home


docker run --name=dummy --rm -ti --security-opt seccomp=unconfined python:3.11-slim bash -c 'apt-get update && apt-get install -y less && echo OK'
Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8904 kB]
Get:5 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [28.3 kB]
Fetched 9180 kB in 2s (4821 kB/s)                          
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  less
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 131 kB of archives.
After this operation, 321 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bookworm/main amd64 less amd64 590-2 [131 kB]
Fetched 131 kB in 0s (2989 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package less.
(Reading database ... 8386 files and directories currently installed.)
Preparing to unpack .../archives/less_590-2_amd64.deb ...
Unpacking less (590-2) ...
Setting up less (590-2) ...
OK
tianon commented 1 year ago

The proper fix is updating libseccomp2 (and Docker/runc) on your host. :+1:

yosifkit commented 1 year ago

(basically the same comment as https://github.com/docker-library/redis/issues/365#issuecomment-1599633849)

Root cause: it is Docker with libseccomp so a newer syscall used in Debian Bookworm packages/libs is being blocked. (Thanks for confirming @aebm)

libseccomp lets you configure allowed syscalls for a process. Docker sets a default seccomp profile for all containers such that only certain syscalls are allowed and everything else is blocked (so, newer syscalls that are not yet known to libseccomp or docker are blocked).

chazapp commented 1 year ago

We've encountered this issue during docker builds on CircleCI runners using an old machine image (ubuntu-2004:202101-01). Updating the machine config to ubuntu-2004:current solved the issue for us.

aporwal18 commented 1 year ago

Facing this same issue, any proper fix to this?

yosifkit commented 1 year ago

Facing this same issue, any proper fix to this?

https://github.com/docker-library/python/issues/837#issuecomment-1599640563:

  • update libseccomp and docker on the host running the containers
davidt99 commented 1 year ago

I'm having this issue on Mac, on python 3.11. I have the latest docker installed, but licseccomp is not used in Mac as far as I know, any suggestions?

k0d3r1s commented 1 year ago

@davidt99 on mac I found a workaround: exit 0, so apt-get update becomes bash -c "apt-get update; exit 0". probably not proper solution but it works for now

ad-m-ss commented 1 year ago

I'm having this issue on Mac, on python 3.11. I have the latest docker installed, but licseccomp is not used in Mac as far as I know, any suggestions?

Docker on Mac usually uses Linux VM. Linux VM uses libseccomp.

k0d3r1s commented 1 year ago

which libseccomp version is required for this to be working?

openbrian commented 1 year ago

Just a comment here. If you're running your builds in a dockerized Jenkins, for example, you'll also be using docker:dind (docker in docker). You may need to upgrade docker here. docker pull docker:dind

skabbit commented 1 year ago

I verify, that switching to python:3.7-slim-buster (or python:3.7-buster in my case) resolved this issue with NO_PUBKEY.

Also, faced this issue only while using docker:18.09.7-dind runner in CI/CD, but cannot reproduce it on docker v20.10.13 manually running docker cmd.

mke21 commented 1 year ago

I've got this problem running in gitlab-ci with the worker running on a private Kubernetes cluster. I'm not sure how to fix it apart from using the older bullseye image which obviously is a stopgap solution. This is the setup for the build:

variables:
  DOCKER_HOST: tcp://localhost:2375
  DOCKER_TLS_CERTDIR: ""

services:
  - name: docker:stable-dind

image: docker:stable
sergei-lobanov commented 1 year ago

Any plans for a normal fix?

tianon commented 11 months ago

Any plans for a normal fix?

https://github.com/docker-library/python/issues/837#issuecomment-1599631543

The proper fix is updating libseccomp2 (and Docker/runc) on your host. 👍

fgoura commented 8 months ago

I am using python:3.11-slim for building my docker image and I am getting same error :

Reading package lists... W: GPG error: http://deb.debian.org/debian bookworm InRelease: The following signatures couldn't be verified because the public key is not available: Is this issue fixed in any slim version ?

tianon commented 8 months ago

No, the issue isn't caused by our images directly, so cannot be "fixed" in the images. You need to update your host:

Any plans for a normal fix?

#837 (comment)

The proper fix is updating libseccomp2 (and Docker/runc) on your host. 👍

fgoura commented 8 months ago

Regarding the fix : I tried installing the libseccomp2 but it says it is already on the latest version

libseccomp2 is already the newest version (2.5.4-1+b3).
libseccomp2 set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.