error creating build container: error obtaining default signature policy: open /etc/containers/policy.json

x80486 commented 4 years ago

Description

I installed podman 1.9.3 and buildah 1.15.0 from Nix; everything works great so far. I then installed podman-compose 0.1.5 and I can't seem to be able to use any of the docker-compose.yml files that I have in any project. It always fails with this error: Error: error creating build container: error obtaining default signature policy: open /etc/containers/policy.json: no such file or directory (and I get the same error if I try to use podman to build any image).

I filed this one here because I saw that problems building images should be filed in the buildah repository, so I guess the problem here is podman — or probably some other configuration problem.

Steps to reproduce the issue:

Install podman-wrapper and buildah-wrapper
Have a multistage Docker image
Run podman-compose up --build --detach --remove-orphans

Describe the results you received

[x80486@archbook:~/Workshop/Development/erlang_cowboy]$ podman-compose up --build --detach --remove-orphans 
podman build -t acme/erlang-cowboy:latest -f ./Dockerfile .
STEP 1: FROM docker.io/library/erlang:23.0.2-alpine AS builder
Error: error creating build container: error obtaining default signature policy: open /etc/containers/policy.json: no such file or directory
125
podman pod create --name=erlang_cowboy --share net -p 9080:9080
Error: unable to create pod: error adding pod to state: name erlang_cowboy is in use: pod already exists
125
podman run --name=erlang-cowboy -d --pod=erlang_cowboy -l io.podman.compose.config-hash=123 -l io.podman.compose.project=erlang_cowboy -l io.podman.compose.version=0.0.1 -l com.docker.compose.container-number=1 -l com.docker.compose.service=api --add-host api:127.0.0.1 --add-host erlang-cowboy:127.0.0.1 acme/erlang-cowboy:latest
Trying to pull quay.io/acme/erlang-cowboy:latest...
  unauthorized: access to the requested resource is not authorized
Trying to pull docker.io/acme/erlang-cowboy:latest...
  denied: requested access to the resource is denied
Error: unable to pull acme/erlang-cowboy:latest: 2 errors occurred:
  * Error initializing source docker://quay.io/acme/erlang-cowboy:latest: Error reading manifest latest in quay.io/acme/erlang-cowboy: unauthorized: access to the requested resource is not authorized
  * Error initializing source docker://acme/erlang-cowboy:latest: Error reading manifest latest in docker.io/acme/erlang-cowboy: errors:
denied: requested access to the resource is denied
unauthorized: authentication required

I'm not using root for anything. Since I installed everything via Nix, I had to create manually: /etc/subuid, /etc/subgid, ~/.config/containers/registries.conf, and ~/.config/containers/policy.json.

Describe the results you expected:

For podman (or buildah if it's the one who does it) to be able to build the image(s), in the same way I was using Docker before.

Output of `buildah version`:

[x80486@archbook:~/Workshop/Development/erlang_cowboy]$ buildah version
Version:         1.15.0
Go Version:      go1.14.4
Image Spec:      1.0.1-dev
Runtime Spec:    1.0.2-dev
CNI Spec:        0.4.0
libcni Version:  v0.7.2-0.20190904153231-83439463f784
image Version:   5.5.1
Git Commit:      unknown
Built:           Wed Dec 31 19:00:01 1969
OS/Arch:         linux/amd64

Output of `podman version` if reporting a `podman build` issue:

[x80486@archbook:~/Workshop/Development/erlang_cowboy]$ podman version
Version:            1.9.3
RemoteAPI Version:  1
Go Version:         go1.14.4
Built:              Wed Dec 31 19:00:01 1969
OS/Arch:            linux/amd64

Output of `cat /etc/*release`:

[x80486@archbook:~/Workshop/Development/erlang_cowboy]$ cat /etc/*release
NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://www.archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
LOGO=archlinux

Output of `uname -a`:

[x80486@archbook:~/Workshop/Development/erlang_cowboy]$ uname -a
Linux uplink 5.4.48-1-lts #1 SMP Mon, 22 Jun 2020 08:10:21 +0000 x86_64 GNU/Linux

Output of `cat /etc/containers/storage.conf`:

[x80486@archbook:~/Workshop/Development/erlang_cowboy]$ cat /etc/containers/storage.conf
cat: /etc/containers/storage.conf: No such file or directory

I wouldn't want to configure anything in /etc/containers/**/*, unless there is no other way. So far, I have all my configuration files in my home directory:

[x80486@archbook:~/Workshop/Development/erlang_cowboy]$ cat ~/.config/containers/registries.conf 
[registries.search]
registries = ["quay.io", "docker.io"]

[registries.insecure]
registries = []

[registries.block]
registries = []

[x80486@archbook:~/Workshop/Development/erlang_cowboy]$ cat ~/.config/containers/policy.json 
{
  "default": [
    { "type": "insecureAcceptAnything" }
  ],
  "transports": {
    "docker-daemon": {
      "": [
        { "type": "insecureAcceptAnything" }
      ]
    }
  }
}

This is the output from what I'm doing:

[x80486@archbook:~/Workshop/Development/erlang_cowboy]$ podman-compose up --build --detach 
podman build -t acme/erlang-cowboy:latest -f ./Dockerfile .
STEP 1: FROM docker.io/library/erlang:23.0.2-alpine AS builder
Error: error creating build container: error obtaining default signature policy: open /etc/containers/policy.json: no such file or directory
125
podman pod create --name=erlang_cowboy --share net -p 9080:9080
7a55748f188d1f516efde08e08129dbae8efdf075a017e9438f4d8e83b8df6ef
0
podman run --name=erlang-cowboy -d --pod=erlang_cowboy -l io.podman.compose.config-hash=123 -l io.podman.compose.project=erlang_cowboy -l io.podman.compose.version=0.0.1 -l com.docker.compose.container-number=1 -l com.docker.compose.service=api --add-host api:127.0.0.1 --add-host erlang-cowboy:127.0.0.1 acme/erlang-cowboy:latest
Trying to pull quay.io/acme/erlang-cowboy:latest...
  unauthorized: access to the requested resource is not authorized
Trying to pull docker.io/acme/erlang-cowboy:latest...
  denied: requested access to the resource is denied
Error: unable to pull acme/erlang-cowboy:latest: 2 errors occurred:
  * Error initializing source docker://quay.io/acme/erlang-cowboy:latest: Error reading manifest latest in quay.io/acme/erlang-cowboy: unauthorized: access to the requested resource is not authorized
  * Error initializing source docker://acme/erlang-cowboy:latest: Error reading manifest latest in docker.io/acme/erlang-cowboy: errors:
denied: requested access to the resource is denied
unauthorized: authentication required

TomSweeneyRedHat commented 4 years ago

First off, love you icon! Thanks for the issue report too. Actually I don't believe this is a Buildah or a Podman issue, it's most likely an issue with containers/image. However, we'll keep the issue here until we can dredge up the location.

@QiWang19 can you take a look please?

@mheon or @mtrmac any thoughts?

mtrmac commented 4 years ago

Per-user policy.json requires c/image ≥ 5.5.0, which is not available in Podman 1.9.3 AFAICS; Podman 2 would be required.

As a possible workaround, several, but not all, Podman commands have an option to manually specify the path to policy.json; but even if that option were available for all the commands, passing that option through other software might be even more difficult.

x80486 commented 4 years ago

Thank you, @TomSweeneyRedHat! :wave: ...I imagined it was some setting(s) I was missing :man_shrugging:

OK, @mtrmac, if that's the case, I can wait for the Nix package to be updated. I don't have any rush; I'm just trying to see if I can replace the Docker machinery altogether with the podman[-compose] + buildah combination.

All of this looks great so far, you folks have been doing a phenomenal job! :revolving_hearts: :1st_place_medal: :tropical_drink: — feel free to close this one

TomSweeneyRedHat commented 4 years ago

Thx @x80486 , closing

x80486 commented 4 years ago

I just updated to version 2.0.1 and now I'm getting:

[x80486@archbook:~/Workshop/Development/erlang_cowboy]$ podman-compose up --build --detach 
podman build -t acme/erlang-cowboy:latest -f ./Dockerfile .
STEP 1: FROM docker.io/library/erlang:23.0.2-alpine AS builder
Getting image source signatures
Copying blob af1424f0cfcc done  
Copying blob cbdbe7a5bc2a done  
Copying config 7578713ee0 done  
Writing manifest to image destination
Storing signatures
STEP 2: WORKDIR /tmp/workspace/
--> c44f5f2933c
STEP 3: COPY . ./
--> 3821d4d663d
STEP 4: RUN set -eux;   mkdir release/;   rebar3 as prod tar;   tar -zxf _build/prod/rel/erlang_cowboy/erlang_cowboy-*.tar.gz -C /tmp/workspace/release/
ERRO[0000] systemd cgroup flag passed, but systemd support for managing cgroups is not available 
systemd cgroup flag passed, but systemd support for managing cgroups is not available
error running container: error creating container for [/bin/sh -c set -eux;   mkdir release/;   rebar3 as prod tar;   tar -zxf _build/prod/rel/erlang_cowboy/erlang_cowboy-*.tar.gz -C /tmp/workspace/release/]: : exit status 1
Error: error building at STEP "RUN set -eux;   mkdir release/;   rebar3 as prod tar;   tar -zxf _build/prod/rel/erlang_cowboy/erlang_cowboy-*.tar.gz -C /tmp/workspace/release/": error while running runtime: exit status 1
125
podman pod create --name=erlang_cowboy --share net -p 9080:9080
62b39104c83b6a8868edb0a399cc21971884dd3477b8bf30e7ef2825c9a830cd
0
podman run --name=erlang-cowboy -d --pod=erlang_cowboy -l io.podman.compose.config-hash=123 -l io.podman.compose.project=erlang_cowboy -l io.podman.compose.version=0.0.1 -l com.docker.compose.container-number=1 -l com.docker.compose.service=api --add-host api:127.0.0.1 --add-host erlang-cowboy:127.0.0.1 acme/erlang-cowboy:latest
Trying to pull quay.io/acme/erlang-cowboy:latest...
  unauthorized: access to the requested resource is not authorized
Trying to pull docker.io/acme/erlang-cowboy:latest...
  denied: requested access to the resource is denied
Error: unable to pull acme/erlang-cowboy:latest: 2 errors occurred:
  * Error initializing source docker://quay.io/acme/erlang-cowboy:latest: Error reading manifest latest in quay.io/acme/erlang-cowboy: unauthorized: access to the requested resource is not authorized
  * Error initializing source docker://acme/erlang-cowboy:latest: Error reading manifest latest in docker.io/acme/erlang-cowboy: errors:
denied: requested access to the resource is denied
unauthorized: authentication required

125

Is this a know issue or something specific to my environment/operating system? I searched for this error but there is not much information about it, and the only "relevant" issue that I see here is closed. Any clues?

TomSweeneyRedHat commented 4 years ago

@x80486 That looks like something is wonky in your cgroups or systemd setup. What OS are you on and are you using runc (cgroup v1) or crun (cgroup v2)? @vrothberg or @giuseppe any thoughts?

x80486 commented 4 years ago

I think you are right; I didn't know if this was an issue indeed with podman or not, but judging for the results on the search, it hasn't happened to many people.

I'm using Arch Linux (LTS kernel) Linux uplink 5.4.48-1-lts #1 SMP Mon, 22 Jun 2020 08:10:21 +0000 x86_64 GNU/Linux.

vrothberg commented 4 years ago

The error comes from pulling acme/erlang-cowboy:latest. The image doesn't specify a registry, so podman is going through the "unqualified search registries" in /etc/containers/registries.conf and tries to pull it from them - first successful pull would win. However, none of the pulls succeeded as the image a) doesn't exist or b) requires authentication where no or the wrong credentials were set.

x80486 commented 4 years ago

I see an error before that, but the image you were referring to is the actual one that's getting built; this is my docker-compose.yml file:

version: "3"

services:
  api:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: erlang-cowboy
    image: acme/erlang-cowboy:latest
    networks:
      - global-network
    ports:
      - "9080:9080"

networks:
  global-network:
    driver: bridge
    name: cowboy-network

...and this is the Dockerfile being referenced:

#
# Dockerfile for erlang_cowboy
#
#   $ buildah build-using-dockerfile --tag acme/erlang-cowboy:latest ./
#   $ podman run --detach --name erlang-cowboy --publish 9080:9080 --tty acme/erlang-cowboy:latest

#
# https://hub.docker.com/_/erlang/
FROM docker.io/library/erlang:23.0.2-alpine AS builder

WORKDIR /tmp/workspace/

COPY . ./

RUN set -eux; \
  mkdir release/; \
  rebar3 as prod tar; \
  tar -zxf _build/prod/rel/erlang_cowboy/erlang_cowboy-*.tar.gz -C /tmp/workspace/release/

#
# https://hub.docker.com/_/alpine/
FROM docker.io/library/alpine:3.11

ENV COOKIE YW5MV2IyeXZWS0dSVmRya0VNTFJZNkxxZQ==
ENV RELX_OUT_FILE_PATH /tmp

RUN set -eux; \
  apk update; \
  apk add --no-cache ncurses; \
  rm -fR /var/cache/apk/*

WORKDIR /opt/application/

COPY --from=builder /tmp/workspace/release/ ./

ENTRYPOINT ["/opt/application/bin/erlang_cowboy"]

CMD ["foreground"]

I know that it works because I used to use it with Docker before; also, if I use the buildah + podman combination it works also (commands on the top of the Dockerfile).

It just doesn't work with podman-compose or just building it with podman.

vrothberg commented 4 years ago

Does a podman pull acme/erlang-cowboy:latest work?

x80486 commented 4 years ago

No it does not; this is the result of a full run by using buildah and podman (notice that podman pull fails, but podmanrun` does not :man_shrugging:):

[x80486@archbook:~/Workshop/Development/erlang_cowboy]$ flush_containers --prune
==> Stopping and removing pods (if any)...

==> Stopping and removing containers (if any)...

==> Destroying the world...
Deleted Pods
Deleted Containers
Deleted Volumes
Deleted Images

==> Results
REPOSITORY  TAG     IMAGE ID  CREATED  SIZE

POD ID  NAME    STATUS  CREATED  NAMES   INFRA ID

CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES

[x80486@archbook:~/Workshop/Development/erlang_cowboy]$ buildah build-using-dockerfile --tag acme/erlang-cowboy:latest ./ 
STEP 1: FROM docker.io/library/erlang:23.0.2-alpine AS builder
Getting image source signatures
Copying blob cbdbe7a5bc2a done  
Copying blob af1424f0cfcc done  
Copying config 7578713ee0 done  
Writing manifest to image destination
Storing signatures
STEP 2: WORKDIR /tmp/workspace/
STEP 3: COPY . ./
STEP 4: RUN set -eux;   mkdir release/;   rebar3 as prod tar;   tar -zxf _build/prod/rel/erlang_cowboy/erlang_cowboy-*.tar.gz -C /tmp/workspace/release/
+ mkdir release/
+ rebar3 as prod tar
===> Verifying dependencies...
===> Fetching cowboy v2.7.0
===> Downloaded package, caching at /root/.cache/rebar3/hex/hexpm/packages/cowboy-2.7.0.tar
===> Linking _build/default/lib/cowboy to _build/prod/lib/cowboy
===> Fetching jsone v1.5.2
===> Downloaded package, caching at /root/.cache/rebar3/hex/hexpm/packages/jsone-1.5.2.tar
===> Linking _build/default/lib/jsone to _build/prod/lib/jsone
===> Fetching lager v3.8.0
===> Downloaded package, caching at /root/.cache/rebar3/hex/hexpm/packages/lager-3.8.0.tar
===> Linking _build/default/lib/lager to _build/prod/lib/lager
===> Fetching cowlib v2.8.0
===> Downloaded package, caching at /root/.cache/rebar3/hex/hexpm/packages/cowlib-2.8.0.tar
===> Linking _build/default/lib/cowlib to _build/prod/lib/cowlib
===> Fetching goldrush v0.1.9
===> Downloaded package, caching at /root/.cache/rebar3/hex/hexpm/packages/goldrush-0.1.9.tar
===> Linking _build/default/lib/goldrush to _build/prod/lib/goldrush
===> Fetching ranch v1.7.1
===> Downloaded package, caching at /root/.cache/rebar3/hex/hexpm/packages/ranch-1.7.1.tar
===> Linking _build/default/lib/ranch to _build/prod/lib/ranch
===> Compiling cowlib
===> Compiling ranch
===> Compiling goldrush
===> Compiling lager
===> Compiling jsone
===> Compiling cowboy
===> Compiling erlang_cowboy
===> Starting relx build process ...
===> Resolving OTP Applications from directories:
          /tmp/workspace/_build/prod/lib
          /usr/local/lib/erlang/lib
===> Resolved erlang_cowboy-0.1.0
===> Including Erts from /usr/local/lib/erlang
===> release successfully created!
===> Starting relx build process ...
===> Resolving OTP Applications from directories:
          /tmp/workspace/_build/prod/lib
          /usr/local/lib/erlang/lib
          /tmp/workspace/_build/prod/rel
===> Resolved erlang_cowboy-0.1.0
===> tarball /tmp/workspace/_build/prod/rel/erlang_cowboy/erlang_cowboy-0.1.0.tar.gz successfully created!
+ tar -zxf _build/prod/rel/erlang_cowboy/erlang_cowboy-0.1.0.tar.gz -C /tmp/workspace/release/
STEP 5: FROM docker.io/library/alpine:3.11
Getting image source signatures
Copying blob cbdbe7a5bc2a [--------------------------------------] 0.0b / 0.0b
Copying config f70734b6a2 done  
Writing manifest to image destination
Storing signatures
STEP 6: ENV COOKIE YW5MV2IyeXZWS0dSVmRya0VNTFJZNkxxZQ==
STEP 7: ENV RELX_OUT_FILE_PATH /tmp
STEP 8: RUN set -eux;   apk update;   apk add --no-cache ncurses;   rm -fR /var/cache/apk/*
+ apk update
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
v3.11.6-91-gd05b39805b [http://dl-cdn.alpinelinux.org/alpine/v3.11/main]
v3.11.6-90-g318b6c3504 [http://dl-cdn.alpinelinux.org/alpine/v3.11/community]
OK: 11271 distinct packages available
+ apk add --no-cache ncurses
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
(1/3) Installing ncurses-terminfo-base (6.1_p20200118-r4)
(2/3) Installing ncurses-libs (6.1_p20200118-r4)
(3/3) Installing ncurses (6.1_p20200118-r4)
Executing busybox-1.31.1-r9.trigger
OK: 7 MiB in 17 packages
+ rm -fR /var/cache/apk/APKINDEX.70f61090.tar.gz /var/cache/apk/APKINDEX.ca2fea5b.tar.gz
STEP 9: WORKDIR /opt/application/
STEP 10: COPY --from=builder /tmp/workspace/release/ ./
STEP 11: ENTRYPOINT ["/opt/application/bin/erlang_cowboy"]
STEP 12: CMD ["foreground"]
STEP 13: COMMIT acme/erlang-cowboy:latest
Getting image source signatures
Copying blob 3e207b409db3 skipped: already exists  
Copying blob d8f72e1f9b64 done  
Copying config c43ff5126d done  
Writing manifest to image destination
Storing signatures
--> c43ff5126d1
c43ff5126d1502a9016e2cf47795add7e4b04a443eec5ef4939531de66f3766a

[x80486@archbook:~/Workshop/Development/erlang_cowboy]$ podman pull acme/erlang-cowboy:latest 
Trying to pull quay.io/acme/erlang-cowboy:latest...
  unauthorized: access to the requested resource is not authorized
Trying to pull docker.io/acme/erlang-cowboy:latest...
  denied: requested access to the resource is denied
Error: unable to pull acme/erlang-cowboy:latest: 2 errors occurred:
  * Error initializing source docker://quay.io/acme/erlang-cowboy:latest: Error reading manifest latest in quay.io/acme/erlang-cowboy: unauthorized: access to the requested resource is not authorized
  * Error initializing source docker://acme/erlang-cowboy:latest: Error reading manifest latest in docker.io/acme/erlang-cowboy: errors:
denied: requested access to the resource is denied
unauthorized: authentication required

[x80486@archbook:~/Workshop/Development/erlang_cowboy]$ podman run --detach --name erlang-cowboy --publish 9080:9080 --tty acme/erlang-cowboy:latest 
416cf4ad893dc9346a1aa4f1b707017c0c1b5d774c7dafb04e54df76ae2c0bdf
[x80486@archbook:~/Workshop/Development/erlang_cowboy]$ podman images 
REPOSITORY                    TAG            IMAGE ID      CREATED         SIZE
localhost/acme/erlang-cowboy  latest         c43ff5126d15  25 seconds ago  17 MB
docker.io/library/erlang      23.0.2-alpine  7578713ee0f1  4 weeks ago     70.9 MB
docker.io/library/alpine      3.11           f70734b6a266  2 months ago    5.88 MB
[x80486@archbook:~/Workshop/Development/erlang_cowboy]$ podman ps --all 
CONTAINER ID  IMAGE                                COMMAND     CREATED         STATUS             PORTS                   NAMES
416cf4ad893d  localhost/acme/erlang-cowboy:latest  foreground  15 seconds ago  Up 14 seconds ago  0.0.0.0:9080->9080/tcp  erlang-cowboy

This is how my registries.conf file looks like:

[x80486@archbook:~/Workshop/Development/erlang_cowboy]$ cat ~/.config/containers/registries.conf 
# The only valid categories are: 'registries.search', 'registries.insecure',
# and 'registries.block'.

[registries.search]
registries = ["quay.io", "docker.io"]

[registries.insecure]
registries = []

#
# Docker only
[registries.block]
registries = []

vrothberg commented 4 years ago

Thanks! This looks like a regression in podman run not using the right credentials. @x80486, could you open a new issue including the reproducer and last comment?

Cc: @baude @mheon @rhatdan

x80486 commented 4 years ago

Alright...see #2431. Thanks for the help! :1st_place_medal:

containers / buildah