search

containers / buildah

A tool that facilitates building OCI images.
https://buildah.io
Apache License 2.0
7.46k stars 785 forks source link

Missing gpg key when using ubuntu / debian images #4216

Closed g-braeunlich closed 2 years ago

g-braeunlich commented 2 years ago

Description

When using FROM ubuntu:jammy or FROM debian:bookworm-slim, and use RUN apt-get update, I receive:

GPG error: http://deb.debian.org/debian bookworm-updates InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 648ACFD622F3D138 NO_PUBKEY 0E98404D386FA1D9

The same steps work perfectly using docker build or running apt-get update interactively in a podman / docker session using the images mentioned above.

Also debian:bullseye(-slim) and ubuntu:focal work like expected.

Steps to reproduce the issue:

  1. Create a Dockerfile containing: 
    FROM debian:bookworm-slim
RUN apt-get update
  2. Run podman build -t test .

Describe the results you received:

podman / buildah fails to build with:

STEP 1/2: FROM debian:bookworm-slim
STEP 2/2: RUN apt-get update
Get:1 http://deb.debian.org/debian bookworm InRelease [157 kB]
Get:2 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:3 http://deb.debian.org/debian bookworm-updates InRelease [49.6 kB]
Err:1 http://deb.debian.org/debian bookworm InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 648ACFD622F3D138 NO_PUBKEY 0E98404D386FA1D9
Err:2 http://deb.debian.org/debian-security bookworm-security InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 112695A0E562B32A NO_PUBKEY 54404762BBB6E853
Err:3 http://deb.debian.org/debian bookworm-updates InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 648ACFD622F3D138 NO_PUBKEY 0E98404D386FA1D9
Reading package lists...
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://deb.debian.org/debian bookworm InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 648ACFD622F3D138 NO_PUBKEY 0E98404D386FA1D9
E: The repository 'http://deb.debian.org/debian bookworm InRelease' is not signed.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://deb.debian.org/debian-security bookworm-security InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 112695A0E562B32A NO_PUBKEY 54404762BBB6E853
E: The repository 'http://deb.debian.org/debian-security bookworm-security InRelease' is not signed.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://deb.debian.org/debian bookworm-updates InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 648ACFD622F3D138 NO_PUBKEY 0E98404D386FA1D9
E: The repository 'http://deb.debian.org/debian bookworm-updates InRelease' is not signed.
E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
E: Sub-process returned an error code
Error: error building at STEP "RUN apt-get update": error while running runtime: exit status 100

Describe the results you expected:

podman buildah can build the image

Output of rpm -q buildah or apt list buildah:

No rpm or apt. Using podman 4.2.0 on calculate linux.

podman info | grep build
  buildahVersion: 1.27.0

Output of buildah version:

bash: buildah: command not found

Output of podman version if reporting a podman build issue:

Client:       Podman Engine
Version:      4.2.0
API Version:  4.2.0
Go Version:   go1.18.4
Git Commit:   7fe5a419cfd2880df2028ad3d7fd9378a88a04f4
Built:        Fri Sep  2 14:33:55 2022
OS/Arch:      linux/amd64

*Output of `cat /etc/release`:**

Calculate Linux Desktop 22.0.1 Cinnamon
DISTRIB_ID="Gentoo"
#------------------------------------------------------------------------------
# Modified Calculate Utilities 3.7.2.15
# Processed template files:
# /var/db/repos/calculate/profiles/templates/3.6/2_ac_install_merge/sys-apps/baselayout/os-release
# To modify this file, create a /etc/os-release.clt template.
#------------------------------------------------------------------------------

NAME="Calculate"
ID="calculate"
ID_LIKE="gentoo"
PRETTY_NAME="Calculate Linux Desktop 22.0.1 Cinnamon"
VERSION="22.0.1 (Cinnamon)"
VERSION_ID=22.0.1
BUILD_ID=20181228
ANSI_COLOR="1;32"
HOME_URL="https://www.calculate-linux.org"
DOCUMENTATION_URL="https://wiki.calculate-linux.org"
SUPPORT_URL="https://chat.calculate-linux.org"
BUG_REPORT_URL="https://forum.calculate-linux.org"
VARIANT_ID=desktop

Output of uname -a:

Linux calculate 5.15.29-calculate #1 SMP PREEMPT Thu Mar 17 13:22:02 UTC 2022 x86_64 Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz GenuineIntel GNU/Linux

Output of cat /etc/containers/storage.conf:

cat: /etc/containers/storage.conf: No such file or directory
rhatdan commented 2 years ago

Works fine for me on Fedora 36

buildah build /tmp
STEP 1/2: FROM debian:bookworm-slim
Resolved "debian" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/debian:bookworm-slim...
Getting image source signatures
Copying blob c530f8c8af9e done  
Copying config 7434046890 done  
Writing manifest to image destination
Storing signatures
STEP 2/2: RUN apt-get update
Get:1 http://deb.debian.org/debian bookworm InRelease [157 kB]
Get:2 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:3 http://deb.debian.org/debian bookworm-updates InRelease [49.6 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8471 kB]
Fetched 8725 kB in 2s (4227 kB/s)
Reading package lists...
COMMIT
Getting image source signatures
Copying blob 628d2b2f8e7b skipped: already exists  
Copying blob 212c0c32b14f done  
Copying config 7a571dd262 done  
Writing manifest to image destination
Storing signatures
--> 7a571dd262c
7a571dd262c1e8a00e82534148bb903ca452d594fefbeed4795f89bc2318405d
flouthoc commented 2 years ago

Works fine for me as well

./buildah build -t test .
STEP 1/2: FROM debian:bookworm-slim
Resolved "debian" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/debian:bookworm-slim...
Getting image source signatures
Copying blob c530f8c8af9e done  
Copying config 7434046890 done  
Writing manifest to image destination
Storing signatures
STEP 2/2: RUN apt-get update
Get:1 http://deb.debian.org/debian bookworm InRelease [157 kB]
Get:2 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:3 http://deb.debian.org/debian bookworm-updates InRelease [49.6 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8457 kB]
Fetched 8712 kB in 6s (1430 kB/s)
Reading package lists...
COMMIT test
Getting image source signatures
Copying blob 628d2b2f8e7b skipped: already exists  
Copying blob 573f295f0ca6 done  
Copying config dd3c11d4b0 done  
Writing manifest to image destination
Storing signatures
--> dd3c11d4b0a
Successfully tagged localhost/test:latest
dd3c11d4b0a7fd5fe5445afc8aa9bf6a45d6fa0d9d0f22cab53fcdfd8bf2803a
flouthoc commented 2 years ago

ubuntu:jammy works as well.

STEP 1/2: FROM ubuntu:jammy
Resolved "ubuntu" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/ubuntu:jammy...
Getting image source signatures
Copying blob 2b55860d4c66 done  
Copying config 2dc39ba059 done  
Writing manifest to image destination
Storing signatures
STEP 2/2: RUN apt-get update
Get:1 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]
Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Get:3 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [148 kB]
Get:4 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [114 kB]
Get:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [99.8 kB]
Get:6 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [4644 B]
Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [366 kB]
Get:8 http://archive.ubuntu.com/ubuntu jammy/restricted amd64 Packages [164 kB]
Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages [1792 kB]
Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [349 kB]
Get:11 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [266 kB]
Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [17.5 MB]
Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [390 kB]
Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [7791 B]
Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [283 kB]
Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [679 kB]
Get:17 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [3175 B]
Get:18 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [7275 B]
Fetched 22.5 MB in 5s (4864 kB/s)
Reading package lists...
COMMIT test
Getting image source signatures
Copying blob 7f5cbd8cc787 skipped: already exists  
Copying blob ff0df2770df7 done  
Copying config 635d56c349 done  
Writing manifest to image destination
Storing signatures
--> 635d56c3496
Successfully tagged localhost/test:latest
635d56c3496f1ed39fcfabf861b8829ef46ef7bda1261b20ef6ccf4f30291fd1
flouthoc commented 2 years ago

I think i'm going to close this as temporary glitch in base image's registry, since none of us are able to reproduce this from the provided base image. Please comment below if you think this is unresolved and its still a buildah issue so we could reopen or just continue the discussion below.

Thanks

g-braeunlich commented 2 years ago

I now also installed buildah and tried with buildah directly.

$ buildah build -t test .
STEP 1/2: FROM debian:bookworm-slim
Resolved "debian" as an alias (/home/g/.cache/containers/short-name-aliases.conf)
Trying to pull docker.io/library/debian:bookworm-slim...
Getting image source signatures
Copying blob c530f8c8af9e done  
Copying config 7434046890 done  
Writing manifest to image destination
Storing signatures
STEP 2/2: RUN apt-get update
Get:1 http://deb.debian.org/debian bookworm InRelease [157 kB]
Get:2 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:3 http://deb.debian.org/debian bookworm-updates InRelease [49.6 kB]
Err:1 http://deb.debian.org/debian bookworm InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 648ACFD622F3D138 NO_PUBKEY 0E98404D386FA1D9
Err:2 http://deb.debian.org/debian-security bookworm-security InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 112695A0E562B32A NO_PUBKEY 54404762BBB6E853
Err:3 http://deb.debian.org/debian bookworm-updates InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 648ACFD622F3D138 NO_PUBKEY 0E98404D386FA1D9
Reading package lists...
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://deb.debian.org/debian bookworm InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 648ACFD622F3D138 NO_PUBKEY 0E98404D386FA1D9
E: The repository 'http://deb.debian.org/debian bookworm InRelease' is not signed.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://deb.debian.org/debian-security bookworm-security InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 112695A0E562B32A NO_PUBKEY 54404762BBB6E853
E: The repository 'http://deb.debian.org/debian-security bookworm-security InRelease' is not signed.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://deb.debian.org/debian bookworm-updates InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 648ACFD622F3D138 NO_PUBKEY 0E98404D386FA1D9
E: The repository 'http://deb.debian.org/debian bookworm-updates InRelease' is not signed.
E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
E: Sub-process returned an error code
error building at STEP "RUN apt-get update": error while running runtime: exit status 100

Even if you cannot reproduce this, do you have an idea what could cause this? As you can see, I am retrieving the exact same hashes c530f8c8af9e and 7434046890 for bookworm. Therefore I exclude a wrong config in /etc/containers/registries.conf.d/.

g-braeunlich commented 2 years ago

Btw.: Having this issue since a month now. So yes: I think it is unresolved and a buildah issue. And in my case it unfortunately is not temporary and is very annoying.

g-braeunlich commented 2 years ago

I tested buildah version 1.25.1 and 1.27.0. I am getting the same error with both versions.

vrothberg commented 2 years ago

Can you run the following command? The digest/sha should match the one below. If it differs, I suggest to repull the image.

~ $ buildah inspect --format "{{.FromImageDigest}}" debian:bookworm-slim
sha256:35b43a17becf838cfabc13f278e9401c75f52e2837a0efd93fa83b7576e33c36

It screams like a bug inside the image and independent of Buildah.

g-braeunlich commented 2 years ago

$ buildah inspect --format "{{.FromImageDigest}}" debian:bookworm-slim sha256:35b43a17becf838cfabc13f278e9401c75f52e2837a0efd93fa83b7576e33c36

Exact the same

vrothberg commented 2 years ago

@giuseppe any suspicion what else could cause the issue?

giuseppe commented 2 years ago

are you using fuse-overlayfs? What is the underlying file system where you store the images?

The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.

Are you using multiple IDs for running your container (you can show the output of buildah unshare cat /proc/self/uid)?

g-braeunlich commented 2 years ago 
$ buildah info
{
    "host": {
        "CgroupVersion": "v1",
        "Distribution": {
            "distribution": "\"calculate\"",
            "version": "22.0.1"
        },
        "MemFree": 3491127296,
        "MemTotal": 16529133568,
        "OCIRuntime": "crun",
        "SwapFree": 14901223424,
        "SwapTotal": 16628494336,
        "arch": "amd64",
        "cpus": 8,
        "hostname": "calculate",
        "kernel": "5.15.29-calculate",
        "os": "linux",
        "rootless": true,
        "uptime": "61h 22m 2.71s (Approximately 2.54 days)",
        "variant": ""
    },
    "store": {
        "ContainerStore": {
            "number": 10
        },
        "GraphDriverName": "vfs",
        "GraphOptions": null,
        "GraphRoot": "/home/g/.local/share/containers/storage",
        "GraphStatus": {},
        "ImageStore": {
            "number": 23
        },
        "RunRoot": "/run/user/1000/containers"
    }
}

Therefore, I assume that I am using vfs and not fuse-overlayfs. Dont have /proc/self/uid but /proc/self/uid_map:

$ buildah unshare cat /proc/self/uid_map
         0       1000          1
         1    1065536      65536
moooeeeep commented 2 years ago

I have this problem on one computer, but not on another. I can reproduce this with:

docker run -it --rm debian:testing apt-get update

So I would agree that this might be not an issue with buildah

I found a possible solution here, but I didn't check if it solves the problem for me yet. Feel free to try it out.

g-braeunlich commented 2 years ago

Thx. What I tried so far was to completely delete .local/lib/containers and start from scratch. This did not help. And I have the same issue on 2 calculate linux boxes.

moooeeeep commented 2 years ago

Ok, we just fixed this by upgrading our docker version. We had version 19.03.x on that host. After installing 20.10.x the problem was gone.

Btw, we found that ubuntu:latest was also affected.

Related: https://stackoverflow.com/questions/66319610/gpg-error-in-ubuntu-21-04-after-second-apt-get-update-during-docker-build

Hope this helps!

g-braeunlich commented 2 years ago

The problem for me is that in my case docker is not affected, but podman / buildah is.

nalind commented 2 years ago

As @giuseppe noted, the crux of it is likely in the vicinity of this:

The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.

Can you re-run with --debug and paste the output? If we're running the command as a UID other than 0, this might shed some light on why. If we are using UID 0, which is the default for the base image, other differences in the configuration that the runtime gets might crop up.

g-braeunlich commented 2 years ago

Sure. Output:

DEBU[0000] Running [buildah-in-a-user-namespace build --debug -t test .] with environment [SHELL=/bin/bash SESSION_MANAGER=local/calculate:@/tmp/.ICE-unix/5439,unix/calculate:/tmp/.ICE-unix/5439 COLORTERM=truecolor XDG_CONFIG_DIRS=/etc/xdg LESS=-R -M --shift 5 HISTCONTROL=ignoreboth XDG_MENU_PREFIX=gnome- AUDIODEV=default JDK_HOME=/etc/java-config-2/current-system-vm GTK_IM_MODULE=uim CONFIG_PROTECT_MASK=/etc/sandbox.d /etc/fonts/fonts.conf /etc/gentoo-release /etc/terminfo /etc/dconf /etc/ca-certificates.conf /etc/texmf/web2c /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/revdep-rebuild R_HOME=/usr/lib64/R JAVA_HOME=/etc/java-config-2/current-system-vm SSH_AUTH_SOCK=/run/user/1000/keyring/ssh ANT_HOME=/usr/share/ant XMODIFIERS=@im=uim DESKTOP_SESSION=gnome LC_MONETARY=de_CH.UTF-8 EDITOR=/usr/bin/emacs PWD=/tmp/test CONFIG_PROTECT=/usr/share/gnupg/qualified.txt /usr/share/config XDG_SESSION_DESKTOP=gnome LOGNAME=g XDG_SESSION_TYPE=wayland MANPATH=/etc/java-config-2/current-system-vm/man:/usr/share/gcc-data/x86_64-pc-linux-gnu/11.3.0/man:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.38/man:/etc/java-config-2/current-system-vm/man/:/usr/local/share/man:/usr/share/man:/usr/lib/rust/man:/usr/lib/llvm/14/share/man XAUTHORITY=/run/user/1000/.mutter-Xwaylandauth.IYL8R1 OPENCL_PROFILE=ocl-icd GDM_LANG=en_US.utf8 HOME=/home/g USERNAME=g LC_PAPER=de_CH.UTF-8 LANG=en_US.utf8 LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.cfg=00;32:*.conf=00;32:*.diff=00;32:*.doc=00;32:*.ini=00;32:*.log=00;32:*.patch=00;32:*.pdf=00;32:*.ps=00;32:*.tex=00;32:*.txt=00;32:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36: XDG_CURRENT_DESKTOP=GNOME VTE_VERSION=6800 WAYLAND_DISPLAY=wayland-0 OPENGL_PROFILE=xorg-x11 GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/d68c66c7_21b9_4427_8423_64624634a739 QT_GRAPHICSSYSTEM=raster XZ_OPT=--threads=0 -8 PRELINK_PATH_MASK=/usr/lib64/gimp/2.0/plug-ins/hot PETSC_DIR=/usr/lib64/petsc INFOPATH=/usr/share/gcc-data/x86_64-pc-linux-gnu/11.3.0/info:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.38/info:/usr/share/info:/usr/share/info/emacs-28 MOZ_GMP_PATH=/usr/lib64/nsbrowser/plugins/gmp-gmpopenh264/system-installed GNOME_SETUP_DISPLAY=:1 JAVAC=/etc/java-config-2/current-system-vm/bin/javac XDG_SESSION_CLASS=user TERM=xterm-256color LESSOPEN=|lesspipe %s USER=g GNOME_TERMINAL_SERVICE=:1.275 MANPAGER=manpager DISPLAY=:0 SHLVL=1 PAGER=/usr/bin/less QT_IM_MODULE=uim LC_MEASUREMENT=de_CH.UTF-8 XDG_RUNTIME_DIR=/run/user/1000 XSESSION=cinnamon-session GCC_SPECS= GSETTINGS_BACKEND=dconf LC_TIME=de_CH.UTF-8 XDG_DATA_DIRS=/usr/local/share:/usr/share:/usr/share/gdm PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/lib/llvm/14/bin:/usr/lib64/julia-1.7.3/bin:/home/g/.local/bin/:/home/g/.local/opt/bin:/home/g/.cargo/bin VBOX_APP_HOME=/usr/lib64/virtualbox GDMSESSION=gnome DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-v2FqWmZYu1,guid=c1c03ba1c194bfda0916cb3563124f4a LV2_PATH=/usr/lib64/lv2 MAIL=/var/mail/g LC_NUMERIC=de_CH.UTF-8 _=/usr/bin/buildah OLDPWD=/tmp TMPDIR=/var/tmp _CONTAINERS_USERNS_CONFIGURED=1 BUILDAH_ISOLATION=rootless], UID map [{ContainerID:0 HostID:1000 Size:1} {ContainerID:1 HostID:1065536 Size:65536}], and GID map [{ContainerID:0 HostID:1001 Size:1} {ContainerID:1 HostID:1065536 Size:65536}] 
DEBU[0000] Pull Policy for pull [ifnewer]               
DEBU[0000] [graphdriver] trying provided driver "vfs"   
DEBU[0000] base for stage 0: "debian:bookworm-slim"     
DEBU[0000] FROM "debian:bookworm-slim"                  
STEP 1/2: FROM debian:bookworm-slim
DEBU[0000] Pulling image debian:bookworm-slim (policy: newer) 
DEBU[0000] Looking up image "debian:bookworm-slim" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Loading registries configuration "/home/g/.config/containers/registries.conf" 
DEBU[0000] Trying "docker.io/library/debian:bookworm-slim" ... 
DEBU[0000] parsed reference into "[vfs@/home/g/.local/share/containers/storage+/run/user/1000/containers]@74340468900a08c044d3498b7865f919f84626356a6d2acc69201105225b6375" 
DEBU[0000] Found image "debian:bookworm-slim" as "docker.io/library/debian:bookworm-slim" in local containers storage 
DEBU[0000] Found image "debian:bookworm-slim" as "docker.io/library/debian:bookworm-slim" in local containers storage ([vfs@/home/g/.local/share/containers/storage+/run/user/1000/containers]@74340468900a08c044d3498b7865f919f84626356a6d2acc69201105225b6375) 
DEBU[0000] exporting opaque data as blob "sha256:74340468900a08c044d3498b7865f919f84626356a6d2acc69201105225b6375" 
DEBU[0000] Image debian:bookworm-slim resolved to local image docker.io/library/debian:bookworm-slim which will be used for pulling 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Attempting to pull candidate docker.io/library/debian:bookworm-slim for docker.io/library/debian:bookworm-slim 
DEBU[0000] Using registries.d directory /etc/containers/registries.d 
DEBU[0000] Trying to access "docker.io/library/debian:bookworm-slim" 
DEBU[0000] No credentials matching docker.io/library/debian found in /run/user/1000/containers/auth.json 
DEBU[0000] No credentials matching docker.io/library/debian found in /home/g/.config/containers/auth.json 
DEBU[0000] No credentials matching docker.io/library/debian found in /home/g/.docker/config.json 
DEBU[0000] No credentials matching docker.io/library/debian found in /home/g/.dockercfg 
DEBU[0000] No credentials for docker.io/library/debian found 
DEBU[0000]  Lookaside configuration: using "default-docker" configuration 
DEBU[0000]  No signature storage configuration found for docker.io/library/debian:bookworm-slim, using built-in default file:///home/g/.local/share/containers/sigstore 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io 
DEBU[0000]  Sigstore attachments: using "default-docker" configuration 
DEBU[0000] GET https://registry-1.docker.io/v2/         
DEBU[0000] Ping https://registry-1.docker.io/v2/ status 401 
DEBU[0000] GET https://auth.docker.io/token?scope=repository%3Alibrary%2Fdebian%3Apull&service=registry.docker.io 
DEBU[0000] GET https://registry-1.docker.io/v2/library/debian/manifests/bookworm-slim 
DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json" 
DEBU[0001] GET https://registry-1.docker.io/v2/library/debian/manifests/sha256:35b43a17becf838cfabc13f278e9401c75f52e2837a0efd93fa83b7576e33c36 
DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json" 
DEBU[0001] Skipping pull candidate docker.io/library/debian:bookworm-slim as the image is not newer (pull policy newer) 
DEBU[0001] Looking up image "docker.io/library/debian:bookworm-slim" in local containers storage 
DEBU[0001] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0001] Trying "docker.io/library/debian:bookworm-slim" ... 
DEBU[0001] parsed reference into "[vfs@/home/g/.local/share/containers/storage+/run/user/1000/containers]@74340468900a08c044d3498b7865f919f84626356a6d2acc69201105225b6375" 
DEBU[0001] Found image "docker.io/library/debian:bookworm-slim" as "docker.io/library/debian:bookworm-slim" in local containers storage 
DEBU[0001] Found image "docker.io/library/debian:bookworm-slim" as "docker.io/library/debian:bookworm-slim" in local containers storage ([vfs@/home/g/.local/share/containers/storage+/run/user/1000/containers]@74340468900a08c044d3498b7865f919f84626356a6d2acc69201105225b6375) 
DEBU[0001] exporting opaque data as blob "sha256:74340468900a08c044d3498b7865f919f84626356a6d2acc69201105225b6375" 
DEBU[0002] exporting opaque data as blob "sha256:74340468900a08c044d3498b7865f919f84626356a6d2acc69201105225b6375" 
DEBU[0002] [graphdriver] trying provided driver "vfs"   
DEBU[0002] Container ID: f3432b2619bf8dd8b6b0a2f6bb15cf223f6d228dc34ee9d023a1fb186296c03b 
DEBU[0002] Parsed Step: {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:run Args:[apt-get update] Flags:[] Attrs:map[] Message:RUN apt-get update Original:RUN apt-get update} 
STEP 2/2: RUN apt-get update
DEBU[0002] RUN imagebuilder.Run{Shell:true, Args:[]string{"apt-get update"}, Mounts:[]string(nil)}, docker.Config{Hostname:"", Domainname:"", User:"", Memory:0, MemorySwap:0, MemoryReservation:0, KernelMemory:0, CPUShares:0, CPUSet:"", PortSpecs:[]string(nil), ExposedPorts:map[docker.Port]struct {}{}, PublishService:"", StopSignal:"", StopTimeout:0, Env:[]string{"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"}, Cmd:[]string{"bash"}, Shell:[]string{}, Healthcheck:(*docker.HealthConfig)(nil), DNS:[]string(nil), Image:"", Volumes:map[string]struct {}{}, VolumeDriver:"", WorkingDir:"", MacAddress:"", Entrypoint:[]string{}, SecurityOpts:[]string(nil), OnBuild:[]string{}, Mounts:[]docker.Mount(nil), Labels:map[string]string{}, AttachStdin:false, AttachStdout:false, AttachStderr:false, ArgsEscaped:false, Tty:false, OpenStdin:false, StdinOnce:false, NetworkDisabled:false, VolumesFrom:""} 
DEBU[0002] using "/var/tmp/buildah434580852" to hold bundle data 
DEBU[0002] Resources: &define.CommonBuildOptions{AddHost:[]string{}, OmitHistory:false, CgroupParent:"", CPUPeriod:0x0, CPUQuota:0, CPUShares:0x0, CPUSetCPUs:"", CPUSetMems:"", HTTPProxy:true, IdentityLabel:0x1, Memory:0, DNSSearch:[]string{}, DNSServers:[]string{}, DNSOptions:[]string{}, LabelOpts:[]string(nil), MemorySwap:0, NoHosts:false, OmitTimestamp:false, SeccompProfilePath:"/etc/crio/seccomp.json", ApparmorProfile:"", ShmSize:"65536k", Ulimit:[]string{}, Volumes:[]string{}, Secrets:[]string{}, SSHSources:[]string{}, OCIHooksDir:[]string{}} 
DEBU[0002] adding slirp4netns 10.0.2.3 built-in DNS server 
DEBU[0002] /etc/system-fips does not exist on host, not mounting FIPS mode subscription 
DEBU[0000] bind mounted "/home/g/.local/share/containers/storage/vfs/dir/84b9b0626ffc012121117e2de7756672b10fca9312cb3972530fe078a6732f01" to "/var/tmp/buildah434580852/mnt/rootfs" 
DEBU[0000] config = {"ociVersion":"1.0.2-dev","process":{"user":{"uid":0,"gid":0},"args":["/bin/sh","-c","apt-get update"],"env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","HOSTNAME=f3432b2619bf"],"cwd":"/","capabilities":{"bounding":["CAP_AUDIT_WRITE","CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_MKNOD","CAP_NET_BIND_SERVICE","CAP_NET_RAW","CAP_SETFCAP","CAP_SETGID","CAP_SETPCAP","CAP_SETUID","CAP_SYS_CHROOT"],"effective":["CAP_AUDIT_WRITE","CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_MKNOD","CAP_NET_BIND_SERVICE","CAP_NET_RAW","CAP_SETFCAP","CAP_SETGID","CAP_SETPCAP","CAP_SETUID","CAP_SYS_CHROOT"],"permitted":["CAP_AUDIT_WRITE","CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_MKNOD","CAP_NET_BIND_SERVICE","CAP_NET_RAW","CAP_SETFCAP","CAP_SETGID","CAP_SETPCAP","CAP_SETUID","CAP_SYS_CHROOT"],"ambient":["CAP_AUDIT_WRITE","CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_MKNOD","CAP_NET_BIND_SERVICE","CAP_NET_RAW","CAP_SETFCAP","CAP_SETGID","CAP_SETPCAP","CAP_SETUID","CAP_SYS_CHROOT"]},"rlimits":[{"type":"RLIMIT_NOFILE","hard":1024,"soft":1024},{"type":"RLIMIT_NPROC","hard":32768,"soft":32768}]},"root":{"path":"/var/tmp/buildah434580852/mnt/rootfs"},"hostname":"f3432b2619bf","mounts":[{"destination":"/sys","type":"bind","source":"/sys","options":["rprivate","nosuid","noexec","nodev","ro","rbind"]},{"destination":"/proc","type":"proc","source":"proc","options":["nosuid","noexec","nodev"]},{"destination":"/dev","type":"tmpfs","source":"tmpfs","options":["nosuid","noexec","strictatime","mode=755","size=65536k"]},{"destination":"/etc/hosts","type":"bind","source":"/var/tmp/buildah434580852/hosts","options":["rbind"]},{"destination":"/etc/hostname","type":"bind","source":"/var/tmp/buildah434580852/hostname","options":["rbind"]},{"destination":"/etc/resolv.conf","type":"bind","source":"/var/tmp/buildah434580852/resolv.conf","options":["rbind"]},{"destination":"/dev/pts","type":"devpts","source":"devpts","options":["nosuid","noexec","newinstance","ptmxmode=0666","mode=0620","gid=5"]},{"destination":"/dev/shm","type":"tmpfs","source":"shm","options":["private","nodev","noexec","nosuid","mode=1777","size=65536k"]},{"destination":"/dev/mqueue","type":"mqueue","source":"mqueue","options":["nosuid","noexec","nodev"]},{"destination":"/run/.containerenv","type":"bind","source":"/var/tmp/buildah434580852/run/.containerenv","options":["rbind"]}],"linux":{"resources":{"devices":[{"allow":false,"access":"rwm"}]},"namespaces":[{"type":"pid"},{"type":"network"},{"type":"ipc"},{"type":"uts"},{"type":"mount"}],"seccomp":{"defaultAction":"SCMP_ACT_ERRNO","architectures":["SCMP_ARCH_X86_64","SCMP_ARCH_X86","SCMP_ARCH_X32"],"syscalls":[{"names":["accept","accept4","access","adjtimex","alarm","bind","brk","capget","capset","chdir","chmod","chown","chown32","clock_getres","clock_gettime","clock_nanosleep","close","connect","copy_file_range","creat","dup","dup2","dup3","epoll_create","epoll_create1","epoll_ctl","epoll_ctl_old","epoll_pwait","epoll_wait","epoll_wait_old","eventfd","eventfd2","execve","execveat","exit","exit_group","faccessat","fadvise64","fadvise64_64","fallocate","fanotify_mark","fchdir","fchmod","fchmodat","fchown","fchown32","fchownat","fcntl","fcntl64","fdatasync","fgetxattr","flistxattr","flock","fork","fremovexattr","fsetxattr","fstat","fstat64","fstatat64","fstatfs","fstatfs64","fsync","ftruncate","ftruncate64","futex","futimesat","getcpu","getcwd","getdents","getdents64","getegid","getegid32","geteuid","geteuid32","getgid","getgid32","getgroups","getgroups32","getitimer","getpeername","getpgid","getpgrp","getpid","getppid","getpriority","getrandom","getresgid","getresgid32","getresuid","getresuid32","getrlimit","get_robust_list","getrusage","getsid","getsockname","getsockopt","get_thread_area","gettid","gettimeofday","getuid","getuid32","getxattr","inotify_add_watch","inotify_init","inotify_init1","inotify_rm_watch","io_cancel","ioctl","io_destroy","io_getevents","ioprio_get","ioprio_set","io_setup","io_submit","ipc","kill","lchown","lchown32","lgetxattr","link","linkat","listen","listxattr","llistxattr","_llseek","lremovexattr","lseek","lsetxattr","lstat","lstat64","madvise","memfd_create","mincore","mkdir","mkdirat","mknod","mknodat","mlock","mlock2","mlockall","mmap","mmap2","mprotect","mq_getsetattr","mq_notify","mq_open","mq_timedreceive","mq_timedsend","mq_unlink","mremap","msgctl","msgget","msgrcv","msgsnd","msync","munlock","munlockall","munmap","nanosleep","newfstatat","_newselect","open","openat","pause","pipe","pipe2","poll","ppoll","prctl","pread64","preadv","preadv2","prlimit64","pselect6","pwrite64","pwritev","pwritev2","read","readahead","readlink","readlinkat","readv","recv","recvfrom","recvmmsg","recvmsg","remap_file_pages","removexattr","rename","renameat","renameat2","restart_syscall","rmdir","rt_sigaction","rt_sigpending","rt_sigprocmask","rt_sigqueueinfo","rt_sigreturn","rt_sigsuspend","rt_sigtimedwait","rt_tgsigqueueinfo","sched_getaffinity","sched_getattr","sched_getparam","sched_get_priority_max","sched_get_priority_min","sched_getscheduler","sched_rr_get_interval","sched_setaffinity","sched_setattr","sched_setparam","sched_setscheduler","sched_yield","seccomp","select","semctl","semget","semop","semtimedop","send","sendfile","sendfile64","sendmmsg","sendmsg","sendto","setfsgid","setfsgid32","setfsuid","setfsuid32","setgid","setgid32","setgroups","setgroups32","setitimer","setpgid","setpriority","setregid","setregid32","setresgid","setresgid32","setresuid","setresuid32","setreuid","setreuid32","setrlimit","set_robust_list","setsid","setsockopt","set_thread_area","set_tid_address","setuid","setuid32","setxattr","shmat","shmctl","shmdt","shmget","shutdown","sigaltstack","signalfd","signalfd4","sigreturn","socket","socketcall","socketpair","splice","stat","stat64","statfs","statfs64","symlink","symlinkat","sync","sync_file_range","syncfs","sysinfo","syslog","tee","tgkill","time","timer_create","timer_delete","timerfd_create","timerfd_gettime","timerfd_settime","timer_getoverrun","timer_gettime","timer_settime","times","tkill","truncate","truncate64","ugetrlimit","umask","uname","unlink","unlinkat","utime","utimensat","utimes","vfork","vmsplice","wait4","waitid","waitpid","write","writev","mount","umount2","reboot","name_to_handle_at","unshare"],"action":"SCMP_ACT_ALLOW"},{"names":["personality"],"action":"SCMP_ACT_ALLOW","args":[{"index":0,"value":0,"op":"SCMP_CMP_EQ"}]},{"names":["personality"],"action":"SCMP_ACT_ALLOW","args":[{"index":0,"value":8,"op":"SCMP_CMP_EQ"}]},{"names":["personality"],"action":"SCMP_ACT_ALLOW","args":[{"index":0,"value":131072,"op":"SCMP_CMP_EQ"}]},{"names":["personality"],"action":"SCMP_ACT_ALLOW","args":[{"index":0,"value":131080,"op":"SCMP_CMP_EQ"}]},{"names":["personality"],"action":"SCMP_ACT_ALLOW","args":[{"index":0,"value":4294967295,"op":"SCMP_CMP_EQ"}]},{"names":["arch_prctl"],"action":"SCMP_ACT_ALLOW"},{"names":["modify_ldt"],"action":"SCMP_ACT_ALLOW"},{"names":["clone"],"action":"SCMP_ACT_ALLOW","args":[{"index":0,"value":2080505856,"op":"SCMP_CMP_MASKED_EQ"}]},{"names":["chroot"],"action":"SCMP_ACT_ALLOW"}]},"maskedPaths":["/proc/acpi","/proc/kcore","/proc/keys","/proc/latency_stats","/proc/timer_list","/proc/timer_stats","/proc/sched_debug","/proc/scsi","/sys/firmware","/sys/fs/selinux","/sys/dev"],"readonlyPaths":["/proc/asound","/proc/bus","/proc/fs","/proc/irq","/proc/sys","/proc/sysrq-trigger"]}} 
DEBU[0000] Running ["/usr/bin/crun" "create" "--bundle" "/var/tmp/buildah434580852" "--pid-file" "/var/tmp/buildah434580852/pid" "--no-new-keyring" "buildah-buildah434580852"] 
DEBU[0000] waiting for parent start message             
DEBU[0002] network namespace successfully setup, send start message to child 
DEBU[0000] Running ["/usr/bin/crun" "start" "buildah-buildah434580852"] 
DEBU[0000] closing stdin                                
Get:1 http://deb.debian.org/debian bookworm InRelease [157 kB]
Get:2 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:3 http://deb.debian.org/debian bookworm-updates InRelease [49.6 kB]
Err:1 http://deb.debian.org/debian bookworm InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 648ACFD622F3D138 NO_PUBKEY 0E98404D386FA1D9
Err:2 http://deb.debian.org/debian-security bookworm-security InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 112695A0E562B32A NO_PUBKEY 54404762BBB6E853
Err:3 http://deb.debian.org/debian bookworm-updates InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 648ACFD622F3D138 NO_PUBKEY 0E98404D386FA1D9
Reading package lists...
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://deb.debian.org/debian bookworm InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 648ACFD622F3D138 NO_PUBKEY 0E98404D386FA1D9
E: The repository 'http://deb.debian.org/debian bookworm InRelease' is not signed.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian-security/dists/bookworm-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://deb.debian.org/debian-security bookworm-security InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 112695A0E562B32A NO_PUBKEY 54404762BBB6E853
E: The repository 'http://deb.debian.org/debian-security bookworm-security InRelease' is not signed.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-bullseye-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-buster-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-security-automatic.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://deb.debian.org/debian/dists/bookworm-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-stable.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://deb.debian.org/debian bookworm-updates InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 648ACFD622F3D138 NO_PUBKEY 0E98404D386FA1D9
E: The repository 'http://deb.debian.org/debian bookworm-updates InRelease' is not signed.
E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
E: Sub-process returned an error code
DEBU[0001] "/var/tmp/buildah434580852/mnt/rootfs" is apparently not really mounted, skipping 
DEBU[0001] "/var/tmp/buildah434580852/mnt" is apparently not really mounted, skipping 
DEBU[0004] Error building at step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:run Args:[apt-get update] Flags:[] Attrs:map[] Message:RUN apt-get update Original:RUN apt-get update}: error while running runtime: exit status 100 
error building at STEP "RUN apt-get update": error while running runtime: exit status 100
DEBU[0004] shutting down the store                      
INFO[0004] failed to shutdown storage: "a layer is mounted: layer is in use by a container" 
DEBU[0004] exit status 100
nalind commented 2 years ago

Hmm, nothing jumps out as being unusual in there.

Apparently apt drops privileges to the _apt user when it's doing things that don't require privileges, but for whatever reason, it's not able to read files that are world-readable in the image. I can start a container based on that image manually, use su -s /bin/bash - _apt to open a shell as that user in the container, and read those files without issue here, so I'm going to assume that the image itself is fine.

Is your system doing something that mine isn't that would cause that access to fail? Is there a mandatory access control mechanism, like apparmor or SELinux, being used that the tool isn't aware of? The config blob we're passing to crun doesn't mention either.

g-braeunlich commented 2 years ago

No, I dont use apparmor or SELinux. Very strange

rhatdan commented 2 years ago

Does it work in rootful mode? Could be user namespace. Or fuse-overlay?

g-braeunlich commented 2 years ago

Just tried running buildah as root. Also does not work. 🤔

mazzz1y commented 2 years ago

the same for me

```yaml host: arch: amd64 buildahVersion: 1.28.0 cgroupControllers: - cpuset - cpu - io - memory - pids cgroupManager: systemd cgroupVersion: v2 conmon: package: /usr/bin/conmon is owned by conmon 1:2.1.5-1 path: /usr/bin/conmon version: 'conmon version 2.1.5, commit: c9f7f19eb82d5b8151fc3ba7fbbccf03fdcd0325' cpuUtilization: idlePercent: 94.64 systemPercent: 1.48 userPercent: 3.87 cpus: 16 distribution: distribution: arch version: unknown eventLogger: journald hostname: framework idMappings: gidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 - container_id: 65537 host_id: 231072 size: 65536 uidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 - container_id: 65537 host_id: 231072 size: 65536 kernel: 6.0.9-arch1-1 linkmode: dynamic logDriver: journald memFree: 45481086976 memTotal: 67131629568 networkBackend: netavark ociRuntime: name: runc package: /usr/bin/runc is owned by runc 1.1.4-1 path: /usr/bin/runc version: |- runc version 1.1.4 spec: 1.0.2-dev go: go1.19 libseccomp: 2.5.4 os: linux remoteSocket: path: /run/user/1000/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: true seccompEnabled: true seccompProfilePath: /etc/containers/seccomp.json selinuxEnabled: false serviceIsRemote: false slirp4netns: executable: /usr/bin/slirp4netns package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.0-1 version: |- slirp4netns version 1.2.0 commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383 libslirp: 4.7.0 SLIRP_CONFIG_VERSION_MAX: 4 libseccomp: 2.5.4 swapFree: 0 swapTotal: 0 uptime: 3h 29m 9.00s (Approximately 0.12 days) plugins: authorization: null log: - k8s-file - none - passthrough - journald network: - bridge - macvlan volume: - local registries: search: - docker.io - registry.fedoraproject.org - quay.io - registry.access.redhat.com - registry.centos.org store: configFile: /home/user/.config/containers/storage.conf containerStore: number: 1 paused: 0 running: 1 stopped: 0 graphDriverName: overlay graphOptions: overlay.mount_program: Executable: /usr/bin/fuse-overlayfs Package: /usr/bin/fuse-overlayfs is owned by fuse-overlayfs 1.9-1 Version: |- fusermount3 version: 3.12.0 fuse-overlayfs: version 1.9 FUSE library version 3.12.0 using FUSE kernel interface version 7.31 graphRoot: /home/user/.local/share/containers/storage graphRootAllocated: 33565814784 graphRootUsed: 85016576 graphStatus: Backing Filesystem: tmpfs Native Overlay Diff: "false" Supports d_type: "true" Using metacopy: "false" imageCopyTmpDir: /var/tmp imageStore: number: 1 runRoot: /run/user/1000/containers volumePath: /home/user/.local/share/containers/storage/volumes version: APIVersion: 4.3.1 Built: 1668092357 BuiltTime: Thu Nov 10 20:59:17 2022 GitCommit: 814b7b003cc630bf6ab188274706c383f9fb9915-dirty GoVersion: go1.19.3 Os: linux OsArch: linux/amd64 Version: 4.3.1 ```

For me helps change crun -> runc

Changing tmpfs to btrfs and fuse-overlayfs to overlayfs doesn't change anything, so just ignore it here.

g-braeunlich commented 1 year ago

For me helps change crun -> runc

Thx @mazzz1y I can confirm that this helps. Also after a fresh install, I can no longer reproduce the above issue.