search

debuerreotype / docker-debian-artifacts

Official builds of debuerreotype-generated Debian tarballs for use in Docker
https://docker.debian.net
Apache License 2.0
315 stars 103 forks source link

GPG issues using arm32v7/debian:buster-20190812 for cross-compilation on x86 machines #79

Closed anandadalton closed 5 years ago

anandadalton commented 5 years ago

Repro steps:

Using vanilla docker (works)

$ sudo apt update
$ sudo apt install qemu-user-static
...
qemu-user-static is already at the newest version (1:3.1+dfsg-7+build1)
$ sudo docker run -it -v /usr/bin/qemu-arm-static:/usr/bin/qemu-arm-static arm32v7/debian:buster-20190812 /bin/bash
# apt update
Get:1 http://cdn-fastly.deb.debian.org/debian buster InRelease [118 kB]                                                                                                                                                                                                                   
Get:3 http://security-cdn.debian.org/debian-security buster/updates InRelease [39.1 kB]                                                                                                                                                                                                   
Get:2 http://cdn-fastly.deb.debian.org/debian buster-updates InRelease [49.3 kB]                                                                                                                                                                                                          
Get:4 http://security-cdn.debian.org/debian-security buster/updates/main armhf Packages [73.1 kB]                                                                                                                                                                                         
Get:5 http://cdn-fastly.deb.debian.org/debian buster/main armhf Packages [7688 kB]                                                                                                                                                                                                        
Get:6 http://cdn-fastly.deb.debian.org/debian buster-updates/main armhf Packages [884 B]                                                                                                                                                                                                  
Fetched 7968 kB in 9s (925 kB/s)                                                                                                                                                                                                                                                          
Reading package lists... Done                                                                                                                                                                                                                                                             
Building dependency tree                                                                                                                                                                                                                                                                  
Reading state information... Done                                                                                                                                                                                                                                                         
1 package can be upgraded. Run 'apt list --upgradable' to see it.

Using cloud-build-local (google cloud thing) (fails)

$ sudo apt update
$ sudo apt install qemu-user-static google-cloud-sdk
...
qemu-user-static is already at the newest version (1:3.1+dfsg-7+build1)
$ mkdir /tmp/crossbuild && cd /tmp/crossbuild
$ cp /usr/bin/qemu-arm-static .
$ ./qemu-arm-static --version
qemu-arm version 3.1.0 (Debian 1:3.1+dfsg-7+build1)
Copyright (c) 2003-2018 Fabrice Bellard and the QEMU Project developers
$ cat > /tmp/crossbuild/Dockerfile << EOF
FROM arm32v7/debian:buster-20190812
COPY qemu-arm-static /usr/bin
RUN true \
   && apt update \
   && apt install -y nginx \
   && echo "daemon off;" >> /etc/nginx/nginx.conf \
   && true
CMD ["nginx"]
EOF
$ cat > /tmp/crossbuild/cloudbuild.yaml << EOF
steps:
- name: 'gcr.io/cloud-builders/docker'
  args: ['run', '--rm', '--privileged', 'multiarch/qemu-user-static:register', '--reset']
- name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', 'qemu-arm-demo', '.']
- name: 'gcr.io/cloud-builders/docker'
  args: ['save', 'qemu-arm-demo', '-o', './qemu-arm-demo.tar']
EOF
$ sudo cloud-build-local --config=cloudbuild.yaml --dryrun=false .
2019/08/28 13:48:12 Warning: The server docker version installed (18.09.3) is different from the one used in GCB (18.09.0)
2019/08/28 13:48:12 Warning: The client docker version installed (18.09.3) is different from the one used in GCB (18.09.0)
Using default tag: latest
latest: Pulling from cloud-builders/metadata
Digest: sha256:bf3eccb31e47edf25685fd03a28057c9b9ed552412c8230cb309a4e35926a3e1
Status: Image is up to date for gcr.io/cloud-builders/metadata:latest
2019/08/28 13:48:17 Started spoofed metadata server
2019/08/28 13:48:17 Build id = localbuild_607f2f0c-d39a-4b8d-bcfe-f9f6e867745d
2019/08/28 13:48:17 status changed to "BUILD"
BUILD
Starting Step #0
Step #0: Already have image (with digest): gcr.io/cloud-builders/docker
Step #0: Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
Step #0: Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
Step #0: Setting /usr/bin/qemu-armeb-static as binfmt interpreter for armeb
Step #0: Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
Step #0: Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
Step #0: Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
Step #0: Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
Step #0: Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
Step #0: Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
Step #0: Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
Step #0: Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
Step #0: Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
Step #0: Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
Step #0: Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
Step #0: Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
Step #0: Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
Step #0: Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
Step #0: Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
Step #0: Setting /usr/bin/qemu-aarch64_be-static as binfmt interpreter for aarch64_be
Step #0: Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
Step #0: Setting /usr/bin/qemu-riscv32-static as binfmt interpreter for riscv32
Step #0: Setting /usr/bin/qemu-riscv64-static as binfmt interpreter for riscv64
Step #0: Setting /usr/bin/qemu-xtensa-static as binfmt interpreter for xtensa
Step #0: Setting /usr/bin/qemu-xtensaeb-static as binfmt interpreter for xtensaeb
Step #0: Setting /usr/bin/qemu-microblaze-static as binfmt interpreter for microblaze
Step #0: Setting /usr/bin/qemu-microblazeel-static as binfmt interpreter for microblazeel
Step #0: Setting /usr/bin/qemu-or1k-static as binfmt interpreter for or1k
Finished Step #0
2019/08/28 13:48:20 Step Step #0 finished
Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/docker
Step #1: Sending build context to Docker daemon  6.064MB
Step #1: Step 1/4 : FROM arm32v7/debian:buster-20190812
Step #1:  ---> 28dcf7a068ee
Step #1: Step 2/4 : COPY qemu-arm-static /usr/bin
Step #1:  ---> Using cache
Step #1:  ---> 8b7a2a1941f8
Step #1: Step 3/4 : RUN true    && apt update    && apt install -y nginx    && echo "daemon off;" >> /etc/nginx/nginx.conf    && true
Step #1:  ---> Running in 93d5f2d5d65e
Step #1: 
Step #1: 
Step #1: WARNING: 
Step #1: apt does not have a stable CLI interface. Use with caution in scripts.
Step #1: 
Step #1: 
Step #1: Get:1 http://security-cdn.debian.org/debian-security buster/updates InRelease [39.1 kB]
Step #1: Get:2 http://cdn-fastly.deb.debian.org/debian buster InRelease [118 kB]
Step #1: Get:3 http://cdn-fastly.deb.debian.org/debian buster-updates InRelease [49.3 kB]
Step #1: Err:1 http://security-cdn.debian.org/debian-security buster/updates InRelease
Step #1:   Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/partial/security.debian.org_debian-security_dists_buster_updates_InRelease
Step #1: Err:2 http://cdn-fastly.deb.debian.org/debian buster InRelease
Step #1:   Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster_InRelease
Step #1: Err:3 http://cdn-fastly.deb.debian.org/debian buster-updates InRelease
Step #1:   Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster-updates_InRelease
Step #1: Reading package lists...
Step #1: 
Step #1: W: GPG error: http://security-cdn.debian.org/debian-security buster/updates InRelease: Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/partial/security.debian.org_debian-security_dists_buster_updates_InRelease
Step #1: E: The repository 'http://security.debian.org/debian-security buster/updates InRelease' is not signed.
Step #1: W: GPG error: http://cdn-fastly.deb.debian.org/debian buster InRelease: Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster_InRelease
Step #1: 
Step #1: E: The repository 'http://deb.debian.org/debian buster InRelease' is not signed.
Step #1: W: 
Step #1: GPG error: http://cdn-fastly.deb.debian.org/debian buster-updates InRelease: Couldn't execute /usr/bin/apt-key to check /var/lib/apt/lists/partial/deb.debian.org_debian_dists_buster-updates_InRelease
Step #1: E: The repository 'http://deb.debian.org/debian buster-updates InRelease' is not signed.
Step #1: 
Step #1: The command '/bin/sh -c true    && apt update    && apt install -y nginx    && echo "daemon off;" >> /etc/nginx/nginx.conf    && true' returned a non-zero code: 100
Finished Step #1
2019/08/28 13:48:24 Step Step #1 finished
2019/08/28 13:48:24 status changed to "ERROR"
ERROR
ERROR: build step 1 "gcr.io/cloud-builders/docker" failed: exit status 100
2019/08/28 13:48:26 Build finished with ERROR status

Using cloud-build-local (for alpine) (works??)

$ rm -rf /tmp/crossbuild && mkdir /tmp/crossbuild && cd /tmp/crossbuild
$ cp /usr/bin/qemu-arm-static .
$ cat > /tmp/crossbuild/Dockerfile << EOF
FROM arm32v7/alpine:3.10.2
COPY qemu-arm-static /usr/bin
RUN true \
    && apk update \
    && apk add nginx \
    && echo "daemon off;" >> /etc/nginx/nginx.conf \
    && true
CMD ["nginx"]
EOF
$ cat > /tmp/crossbuild/cloudbuild.yaml << EOF
steps:
- name: 'gcr.io/cloud-builders/docker'
  args: ['run', '--rm', '--privileged', 'multiarch/qemu-user-static:register', '--reset']
- name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', 'qemu-arm-demo', '.']
- name: 'gcr.io/cloud-builders/docker'
  args: ['save', 'qemu-arm-demo', '-o', './qemu-arm-demo.tar']
EOF
$ sudo cloud-build-local --config=cloudbuild.yaml --dryrun=false .
p #1: Executing nginx-1.16.1-r0.pre-install
Step #1: Executing busybox-1.30.1-r2.trigger
Step #1: OK: 5 MiB in 16 packages
Step #1: Removing intermediate container 20203594b4a7
Step #1:  ---> 18996ec9df2c
Step #1: Step 4/4 : CMD ["nginx"]
Step #1:  ---> Running in b0e9edebd97a
Step #1: Removing intermediate container b0e9edebd97a
Step #1:  ---> e04c25edf601
Step #1: Successfully built e04c25edf601
Step #1: Successfully tagged qemu-arm-demo:latest
Finished Step #1
2019/08/28 14:03:55 Step Step #1 finished
Starting Step #2
Step #2: Already have image (with digest): gcr.io/cloud-builders/docker
Finished Step #2
2019/08/28 14:03:57 Step Step #2 finished
2019/08/28 14:03:57 status changed to "DONE"
DONE

(incidentally, running the Debian on the actual google cloud build site has a similar problem and outputs the same error messages concerning inability to execute /usr/bin/apt-key.)

Debian cloud-build-local, but instead run /usr/bin/apt-key

$ mkdir /tmp/crossbuild && cd /tmp/crossbuild
$ cp /usr/bin/qemu-arm-static .
$ ./qemu-arm-static --version
qemu-arm version 3.1.0 (Debian 1:3.1+dfsg-7+build1)
Copyright (c) 2003-2018 Fabrice Bellard and the QEMU Project developers
$ cat > /tmp/crossbuild/Dockerfile << EOF
FROM arm32v7/debian:buster-20190812
COPY qemu-arm-static /usr/bin
RUN true \
   && /usr/bin/apt-key list \
   && true
CMD ["nginx"]
EOF
$ cat > /tmp/crossbuild/cloudbuild.yaml << EOF
steps:
- name: 'gcr.io/cloud-builders/docker'
  args: ['run', '--rm', '--privileged', 'multiarch/qemu-user-static:register', '--reset']
- name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', 'qemu-arm-demo', '.']
- name: 'gcr.io/cloud-builders/docker'
  args: ['save', 'qemu-arm-demo', '-o', './qemu-arm-demo.tar']
EOF
$ sudo cloud-build-local --config=cloudbuild.yaml --dryrun=false .
2019/08/28 14:09:53 Warning: The server docker version installed (18.09.3) is different from the one used in GCB (18.09.0)
2019/08/28 14:09:53 Warning: The client docker version installed (18.09.3) is different from the one used in GCB (18.09.0)
Using default tag: latest
latest: Pulling from cloud-builders/metadata
Digest: sha256:bf3eccb31e47edf25685fd03a28057c9b9ed552412c8230cb309a4e35926a3e1
Status: Image is up to date for gcr.io/cloud-builders/metadata:latest
2019/08/28 14:09:59 Started spoofed metadata server
2019/08/28 14:09:59 Build id = localbuild_c558c67a-10b5-4f6b-9b73-848b8e08c54f
2019/08/28 14:09:59 status changed to "BUILD"
BUILD
Starting Step #0
Step #0: Already have image (with digest): gcr.io/cloud-builders/docker
Step #0: Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
Step #0: Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
Step #0: Setting /usr/bin/qemu-armeb-static as binfmt interpreter for armeb
Step #0: Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
Step #0: Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
Step #0: Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
Step #0: Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
Step #0: Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
Step #0: Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
Step #0: Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
Step #0: Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
Step #0: Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
Step #0: Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
Step #0: Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
Step #0: Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
Step #0: Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
Step #0: Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
Step #0: Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
Step #0: Setting /usr/bin/qemu-aarch64_be-static as binfmt interpreter for aarch64_be
Step #0: Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
Step #0: Setting /usr/bin/qemu-riscv32-static as binfmt interpreter for riscv32
Step #0: Setting /usr/bin/qemu-riscv64-static as binfmt interpreter for riscv64
Step #0: Setting /usr/bin/qemu-xtensa-static as binfmt interpreter for xtensa
Step #0: Setting /usr/bin/qemu-xtensaeb-static as binfmt interpreter for xtensaeb
Step #0: Setting /usr/bin/qemu-microblaze-static as binfmt interpreter for microblaze
Step #0: Setting /usr/bin/qemu-microblazeel-static as binfmt interpreter for microblazeel
Step #0: Setting /usr/bin/qemu-or1k-static as binfmt interpreter for or1k
Finished Step #0
2019/08/28 14:10:02 Step Step #0 finished
Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/docker
Step #1: Sending build context to Docker daemon  6.064MB
Step #1: Step 1/4 : FROM arm32v7/debian:buster-20190812
Step #1:  ---> 28dcf7a068ee
Step #1: Step 2/4 : COPY qemu-arm-static /usr/bin
Step #1:  ---> Using cache
Step #1:  ---> 8b7a2a1941f8
Step #1: Step 3/4 : RUN true    && /usr/bin/apt-key list    && true
Step #1:  ---> Running in 8e26633e82fd
Step #1: E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation
Step #1: 
Step #1: The command '/bin/sh -c true    && /usr/bin/apt-key list    && true' returned a non-zero code: 255
Finished Step #1
2019/08/28 14:10:05 Step Step #1 finished
2019/08/28 14:10:06 status changed to "ERROR"
ERROR
ERROR: build step 1 "gcr.io/cloud-builders/docker" failed: exit status 255
2019/08/28 14:10:07 Build finished with ERROR status
tianon commented 5 years ago 
E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation

This is your money shot -- this is telling you that apt-key requires GnuPG installed, but it isn't by default anymore (APT and debootstrap use just the simpler gpgv by default for verification instead):

$ docker run -it --rm debian:buster-slim
root@fad5b5d46201:/# apt-key list
E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation
root@fad5b5d46201:/# apt-get update -qq
root@fad5b5d46201:/# apt-get install -yqq gnupg
...
root@fad5b5d46201:/# apt-key list
/etc/apt/trusted.gpg.d/debian-archive-buster-automatic.gpg
----------------------------------------------------------
pub   rsa4096 2019-04-14 [SC] [expires: 2027-04-12]
      80D1 5823 B7FD 1561 F9F7  BCDD DC30 D7C2 3CBB ABEE
uid           [ unknown] Debian Archive Automatic Signing Key (10/buster) <ftpmaster@debian.org>
sub   rsa4096 2019-04-14 [S] [expires: 2027-04-12]

/etc/apt/trusted.gpg.d/debian-archive-buster-security-automatic.gpg
-------------------------------------------------------------------
pub   rsa4096 2019-04-14 [SC] [expires: 2027-04-12]
      5E61 B217 265D A980 7A23  C5FF 4DFA B270 CAA9 6DFA
uid           [ unknown] Debian Security Archive Automatic Signing Key (10/buster) <ftpmaster@debian.org>
sub   rsa4096 2019-04-14 [S] [expires: 2027-04-12]

/etc/apt/trusted.gpg.d/debian-archive-buster-stable.gpg
-------------------------------------------------------
pub   rsa4096 2019-02-05 [SC] [expires: 2027-02-03]
      6D33 866E DD8F FA41 C014  3AED DCC9 EFBF 77E1 1517
uid           [ unknown] Debian Stable Release Key (10/buster) <debian-release@lists.debian.org>

/etc/apt/trusted.gpg.d/debian-archive-jessie-automatic.gpg
----------------------------------------------------------
pub   rsa4096 2014-11-21 [SC] [expires: 2022-11-19]
      126C 0D24 BD8A 2942 CC7D  F8AC 7638 D044 2B90 D010
uid           [ unknown] Debian Archive Automatic Signing Key (8/jessie) <ftpmaster@debian.org>

/etc/apt/trusted.gpg.d/debian-archive-jessie-security-automatic.gpg
-------------------------------------------------------------------
pub   rsa4096 2014-11-21 [SC] [expires: 2022-11-19]
      D211 6914 1CEC D440 F2EB  8DDA 9D6D 8F6B C857 C906
uid           [ unknown] Debian Security Archive Automatic Signing Key (8/jessie) <ftpmaster@debian.org>

/etc/apt/trusted.gpg.d/debian-archive-jessie-stable.gpg
-------------------------------------------------------
pub   rsa4096 2013-08-17 [SC] [expires: 2021-08-15]
      75DD C3C4 A499 F1A1 8CB5  F3C8 CBF8 D6FD 518E 17E1
uid           [ unknown] Jessie Stable Release Key <debian-release@lists.debian.org>

/etc/apt/trusted.gpg.d/debian-archive-stretch-automatic.gpg
-----------------------------------------------------------
pub   rsa4096 2017-05-22 [SC] [expires: 2025-05-20]
      E1CF 20DD FFE4 B89E 8026  58F1 E0B1 1894 F66A EC98
uid           [ unknown] Debian Archive Automatic Signing Key (9/stretch) <ftpmaster@debian.org>
sub   rsa4096 2017-05-22 [S] [expires: 2025-05-20]

/etc/apt/trusted.gpg.d/debian-archive-stretch-security-automatic.gpg
--------------------------------------------------------------------
pub   rsa4096 2017-05-22 [SC] [expires: 2025-05-20]
      6ED6 F5CB 5FA6 FB2F 460A  E88E EDA0 D238 8AE2 2BA9
uid           [ unknown] Debian Security Archive Automatic Signing Key (9/stretch) <ftpmaster@debian.org>
sub   rsa4096 2017-05-22 [S] [expires: 2025-05-20]

/etc/apt/trusted.gpg.d/debian-archive-stretch-stable.gpg
--------------------------------------------------------
pub   rsa4096 2017-05-20 [SC] [expires: 2025-05-18]
      067E 3C45 6BAE 240A CEE8  8F6F EF0F 382A 1A7B 6500
uid           [ unknown] Debian Stable Release Key (9/stretch) <debian-release@lists.debian.org>

root@fad5b5d46201:/#
anandadalton commented 5 years ago

Thanks for the workaround! I'm wondering if the Debian (and the Ubuntu) images could be rebuilt to avoid this issue? I'm not sure I want to write infrastructure that has the line "RUN... apt update -qq" in it, from a security standpoint.

tianon commented 5 years ago

No, GnuPG was removed from the essential set of both distributions on purpose -- if you want to use apt-key to manage keys, you'll need to install GnuPG yourself (-qq was used only for illustrative purposes -- I wouldn't put that in a real Dockerfile personally). So what I've provided isn't a "workaround" but rather "expected behavior" for these images.

What I'd recommend instead of apt-key is getting familiar with /etc/apt/trusted.gpg.d directly (which I find more reliable than using apt-key anyhow).

For further assistance/discussion, I'd recommend trying the Docker Community Forums, the Docker Community Slack, or Stack Overflow.

anandadalton commented 5 years ago

Thanks Tianon, I'll do that--sorry for the misunderstanding, and thank you for your clarification.

anandadalton commented 5 years ago

FYI to anyone who finds this bug later...

The nature of this bug has surprisingly little to do with the contents of /etc/apt/trusted.gpg.d/, which I verified to be the same, more or less, as on my host system. It has something to do with the fact that apt-key is no longer executed as root, but instead as another user, _apt, which has been introduced for this purpose. I can see this user by running "cat /etc/passwd". The relevance of this is that /usr/bin/qemu-arm-static does not have sufficient permissions when copied in. Running "chmod 0777 /usr/bin/qemu-arm-static" resolved the issue for me. Credit to this fix goes to: https://github.com/drtyhlpr/rpi23-gen-image/pull/85

Thanks again