search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.57k stars 2.4k forks source link

container build via TCP gives error Content-Type: application/json is not supported. #11012

Closed mattymo closed 3 years ago

mattymo commented 3 years ago

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I am trying to build using docker-maven-plugin pointing at podman running in service mode listening on a TCP port. It fails with a content type error. I thought this was already fixed based on the age of the comments in this other issue: https://github.com/fabric8io/docker-maven-plugin/issues/1330

Steps to reproduce the issue:

  1. Run podman in service mode

  2. Run a docker build from another host, specifying DOCKER_HOST pointing to podman

Describe the results you received: Relevant output from podman in debug mode:

DEBU[0120] Applying tar in /var/lib/containers/storage/overlay/2775bac72cec67a28cdde119e557f74643e82304a6e0148db98d3a9c662c28ad/diff 
DEBU[0123] setting image creation date to 2021-04-08 07:22:24.324057841 +0000 UTC 
DEBU[0123] created new image ID "b14538981bc5d275204434ca07abafa3aab421f7be6a71c6727dd845d0c636c6" 
DEBU[0123] set names of image "b14538981bc5d275204434ca07abafa3aab421f7be6a71c6727dd845d0c636c6" to [docker-hub.mycompany/centos7-with-security-fixes:latest] 
DEBU[0123] saved image metadata "{\"signatures-sizes\":{\"sha256:358470d32fd9c6ec3003ff2472d25feed2d4b3d5a83dcb87c289eb314aef047d\":[]}}" 
DEBU[0123] Pulled candidate docker-hub.mycompany/centos7-with-security-fixes:latest successfully 
DEBU[0123] Looking up image "docker-hub.mycompany/centos7-with-security-fixes:latest" in local containers storage 
DEBU[0123] Trying "docker-hub.mycompany/centos7-with-security-fixes:latest" ... 
DEBU[0123] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/var/lib/shared,overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mountopt=nodev,fsync=0]@b14538981bc5d275204434ca07abafa3aab421f7be6a71c6727dd845d0c636c6" 
DEBU[0123] Found image "docker-hub.mycompany/centos7-with-security-fixes:latest" as "docker-hub.mycompany/centos7-with-security-fixes:latest" in local containers storage 
DEBU[0123] APIHandler(1fce8af2-ef3e-4826-9063-dea384217f64) -- POST /v1.18/images/create?fromImage=docker-hub.mycompany%2Fcentos7-with-security-fixes&tag=latest END 
DEBU[0123] IdleTracker 0xc0005f6000:idle 1m+0h/1t connection(s) 
DEBU[0123] IdleTracker 0xc0005f6000:active 1m+0h/1t connection(s) 
INFO[0123] APIHandler(fabc3c63-4d90-4dad-be5b-b2e2268082cb) -- GET /v1.18/images/internalrepo.mycompany%2Fprj-db%3A21.0.0.0-dev-8123c6c/json BEGIN 
DEBU[0123] APIHandler(fabc3c63-4d90-4dad-be5b-b2e2268082cb) -- Header: Accept-Encoding=[gzip,deflate] 
DEBU[0123] APIHandler(fabc3c63-4d90-4dad-be5b-b2e2268082cb) -- Header: Accept=[*/*] 
DEBU[0123] APIHandler(fabc3c63-4d90-4dad-be5b-b2e2268082cb) -- Header: Content-Type=[application/json] 
DEBU[0123] APIHandler(fabc3c63-4d90-4dad-be5b-b2e2268082cb) -- Header: Connection=[Keep-Alive] 
DEBU[0123] APIHandler(fabc3c63-4d90-4dad-be5b-b2e2268082cb) -- Header: User-Agent=[Apache-HttpClient/4.5.5 (Java/11.0.8)] 
DEBU[0123] Looking up image "internalrepo.mycompany/prj-db:21.0.0.0-dev-8123c6c" in local containers storage 
DEBU[0123] Trying "internalrepo.mycompany/prj-db:21.0.0.0-dev-8123c6c" ... 
DEBU[0123] Trying "internalrepo.mycompany/prj-db:21.0.0.0-dev-8123c6c" ... 
DEBU[0123] Trying "internalrepo.mycompany/prj-db:21.0.0.0-dev-8123c6c" ... 
INFO[0123] Request Failed(Not Found): failed to find image internalrepo.mycompany/prj-db:21.0.0.0-dev-8123c6c: internalrepo.mycompany/prj-db:21.0.0.0-dev-8123c6c: No such image 
DEBU[0123] APIHandler(fabc3c63-4d90-4dad-be5b-b2e2268082cb) -- GET /v1.18/images/internalrepo.mycompany%2Fprj-db%3A21.0.0.0-dev-8123c6c/json END 
DEBU[0123] IdleTracker 0xc0005f6000:idle 1m+0h/1t connection(s) 
DEBU[0124] IdleTracker 0xc0005f6000:active 1m+0h/1t connection(s) 
INFO[0124] APIHandler(dc70df91-b04f-493f-b9b1-228074f937f0) -- POST /v1.18/build?dockerfile=Dockerfile&forcerm=1&nocache=0&t=internalrepo.mycompany%2Fprj-db%3A21.0.0.0-dev-8123c6c BEGIN 
DEBU[0124] APIHandler(dc70df91-b04f-493f-b9b1-228074f937f0) -- Header: Content-Type=[application/json] 
DEBU[0124] APIHandler(dc70df91-b04f-493f-b9b1-228074f937f0) -- Header: Content-Length=[2539520] 
DEBU[0124] APIHandler(dc70df91-b04f-493f-b9b1-228074f937f0) -- Header: Connection=[Keep-Alive] 
DEBU[0124] APIHandler(dc70df91-b04f-493f-b9b1-228074f937f0) -- Header: User-Agent=[Apache-HttpClient/4.5.5 (Java/11.0.8)] 
DEBU[0124] APIHandler(dc70df91-b04f-493f-b9b1-228074f937f0) -- Header: Accept-Encoding=[gzip,deflate] 
DEBU[0124] APIHandler(dc70df91-b04f-493f-b9b1-228074f937f0) -- Header: Accept=[*/*] 
INFO[0124] Request Failed(Bad Request): failed to parse query parameter 'Content-Type': "application/json": Content-Type: application/json is not supported. Should be "application/x-tar" 
DEBU[0124] APIHandler(dc70df91-b04f-493f-b9b1-228074f937f0) -- POST /v1.18/build?dockerfile=Dockerfile&forcerm=1&nocache=0&t=internalrepo.mycompany%2Fprj-db%3A21.0.0.0-dev-8123c6c END

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      3.2.2
API Version:  3.2.2
Go Version:   go1.16.4
Built:        Fri Jun 25 23:49:19 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.21.0
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - memory
  - devices
  - freezer
  - net_cls
  - blkio
  - perf_event
  - hugetlb
  - pids
  - net_prio
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.27-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 40
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: file
  hostname: celeste.int.zone
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 3.10.0-1127.10.1.el7.x86_64
  linkmode: dynamic
  memFree: 221974728704
  memTotal: 270205313024
  ociRuntime:
    name: crun
    package: crun-0.20.1-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.20.1
      commit: 0d42f1109fd73548f44b01b3e84d04a279e99d2e
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 2195h 11m 16.53s (Approximately 91.46 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /var/lib/shared
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.5.0-1.fc34.x86_64
      Version: |-
        fusermount3 version: 3.10.4
        fuse-overlayfs: version 1.5
        FUSE library version 3.10.4
        using FUSE kernel interface version 7.31
    overlay.mountopt: nodev,fsync=0
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.2.2
  Built: 1624664959
  BuiltTime: Fri Jun 25 23:49:19 2021
  GitCommit: ""
  GoVersion: go1.16.4
  OsArch: linux/amd64
  Version: 3.2.2

Package info (e.g. output of rpm -q podman or apt list podman):

v3.2.2 (docker image)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

mheon commented 3 years ago

@jwhonce PTAL, especially at the linked Maven plugin issue - it's saying that we don't send Version headers on all requests. I thought I remember us fixing that - but it seems like this is still happening on the latest code?

mattymo commented 3 years ago

Is there any update here on this issue?

mheon commented 3 years ago

@jwhonce PTAL

jwhonce commented 3 years ago

@mattymo @mheon This is as designed from the docker API build documentation, Content-Type has one valid value application/x-tar.

From https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type :

In requests, (such as POST or PUT), the client tells the server what type of data is actually sent.

We validate that header. After reviewing the docker code, they do not. So you may use any value for that header with docker, but only a tar file will be supported hence our validation.

fabric8io/docker-maven-plugin comment asks for a PR to change their behavior.

I'm going to leave this issue open for discussion. /cc @baude @rhatdan

jwhonce commented 3 years ago

@mheon As of 87a78c059dc2fb72ba54805d5488f3e619ae46f1, the API-Version and Libpod-API-Version headers are written on every response before the endpoint handler code is run. https://github.com/fabric8io/docker-maven-plugin/issues/1330#issuecomment-690291669 pre-dates this change in our code.

mattymo commented 3 years ago

@mheon @jwhonce Please reopen. I tested on the latest podman master (commit94886d4abad7787cb920614ed26955ca36b13846) and it is still failing on /build endpoint:

docker-maven-plugin error:

12:41:25 [el7] Caused by: io.fabric8.maven.docker.access.hc.http.HttpRequestException: {"cause":"Content-Type: application/json is not supported. Should be \"application/x-tar\"","message":"failed to parse query parameter 'Content-Type': \"application/json\": Content-Type: application/json is not supported. Should be \"application/x-tar\"","response":400} (Bad Request: 400)

podman logs in debug mode:

time="2021-08-23T09:50:59Z" level=info msg="APIHandler(5d61e467-72ce-4bd0-bf72-264ad5c53ef3) -- GET /v1.40/images/myimage/json BEGIN"
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(5d61e467-72ce-4bd0-bf72-264ad5c53ef3) -- Header: Accept=[*/*]"
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(5d61e467-72ce-4bd0-bf72-264ad5c53ef3) -- Header: Content-Type=[application/json]"
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(5d61e467-72ce-4bd0-bf72-264ad5c53ef3) -- Header: Connection=[Keep-Alive]"
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(5d61e467-72ce-4bd0-bf72-264ad5c53ef3) -- Header: User-Agent=[Apache-HttpClient/4.5.5 (Java/11.0.8)]"
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(5d61e467-72ce-4bd0-bf72-264ad5c53ef3) -- Header: Accept-Encoding=[gzip,deflate]"
time="2021-08-23T09:50:59Z" level=debug msg="Looking up image \"myimage\" in local containers storage"
time="2021-08-23T09:50:59Z" level=debug msg="Trying \"myimage\" ..."
time="2021-08-23T09:50:59Z" level=debug msg="Trying \"localhost/myimage:latest\" ..."
time="2021-08-23T09:50:59Z" level=debug msg="Trying \"registry.fedoraproject.org/myimage:latest\" ..."
time="2021-08-23T09:50:59Z" level=debug msg="Trying \"registry.access.redhat.com/myimage:latest\" ..."
time="2021-08-23T09:50:59Z" level=debug msg="Trying \"docker.io/library/myimage:latest\" ..."
time="2021-08-23T09:50:59Z" level=debug msg="Trying \"quay.io/myimage:latest\" ..."
time="2021-08-23T09:50:59Z" level=debug msg="Trying \"docker.io/library/myimage:latest\" ..."
time="2021-08-23T09:50:59Z" level=info msg="Request Failed(Not Found): failed to find image myimage: myimage: No such image"
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(5d61e467-72ce-4bd0-bf72-264ad5c53ef3) -- GET /v1.40/images/myimage/json END"
time="2021-08-23T09:50:59Z" level=debug msg="IdleTracker 0xc000748000:idle 1m+0h/2t connection(s)"
time="2021-08-23T09:50:59Z" level=debug msg="IdleTracker 0xc000748000:active 1m+0h/2t connection(s)"
time="2021-08-23T09:50:59Z" level=info msg="APIHandler(39b6922a-4e7a-42b8-bb77-8cfb261fdd2a) -- POST /v1.40/build?dockerfile=Dockerfile&forcerm=1&nocache=0&t=myimage BEGIN"
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(39b6922a-4e7a-42b8-bb77-8cfb261fdd2a) -- Header: Accept=[*/*]"
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(39b6922a-4e7a-42b8-bb77-8cfb261fdd2a) -- Header: Content-Type=[application/json]"
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(39b6922a-4e7a-42b8-bb77-8cfb261fdd2a) -- Header: Content-Length=[10240]"
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(39b6922a-4e7a-42b8-bb77-8cfb261fdd2a) -- Header: Connection=[Keep-Alive]"
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(39b6922a-4e7a-42b8-bb77-8cfb261fdd2a) -- Header: User-Agent=[Apache-HttpClient/4.5.5 (Java/11.0.8)]"
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(39b6922a-4e7a-42b8-bb77-8cfb261fdd2a) -- Header: Accept-Encoding=[gzip,deflate]"
time="2021-08-23T09:50:59Z" level=info msg="Request Failed(Bad Request): failed to parse query parameter 'Content-Type': \"application/json\": Content-Type: application/json is not supported. Should be \"application/x-tar\""
time="2021-08-23T09:50:59Z" level=debug msg="APIHandler(39b6922a-4e7a-42b8-bb77-8cfb261fdd2a) -- POST /v1.40/build?dockerfile=Dockerfile&forcerm=1&nocache=0&t=myimage END"
mheon commented 3 years ago

What Podman version? This should definitely be fixed in 3.3.

mattymo commented 3 years ago

@mheon
quay.io/podman/stable@sha256:bc6545c7b3e0d9ab06aa1c7b3303211c15fae50738673834a6f20c5cd1b61814

"Labels": {
    "license": "MIT",
    "name": "fedora",
    "org.opencontainers.image.created": "2021-08-23T08:47:34+00:00",
    "org.opencontainers.image.revision": "2de56a5f41473e8f759972ec568089e9cddc0f98",
    "org.opencontainers.image.source": "https://github.com/containers/podman.git",
    "org.opencontainers.image.version": "3.2.3",
    "vendor": "Fedora Project",
    "version": "34"
}

I don't see any 3.3 builds on quay.io. Is the release coming soon?

mheon commented 3 years ago

The release is complete. @TomSweeneyRedHat When will images be rebuilt?

TomSweeneyRedHat commented 3 years ago

@mattymo the Podman v3.3.0 (or any new version) lands on quay.io after it gets set to stable on bodhi. After being set to stable, it sometimes can take up to an additional day for the CI system to note that and build the image.

I just checked, and it looks like the Fedora 34 and Fedora 35 variants have gone to stable, but 33 and 32 have not yet. @cevich does the new CI job check Fedora 33? I thought that had been bumped to 34....

cevich commented 3 years ago

The multi-arch image build log is here: https://github.com/containers/podman/actions/workflows/multi-arch-build.yaml

@TomSweeneyRedHat the Dockerfile specifies 'latest' Fedora so I think that's F34, no?

Hmm, yes it seems it is pushing v3.2.3 for stable. I just looked and it doesn't appear to have reached the stable repos. yet:

$ podman run -it --rm registry.fedoraproject.org/fedora:latest
Trying to pull registry.fedoraproject.org/fedora:latest...
Getting image source signatures
Copying blob ecfb9899f4ce done
Copying config 37e5619f4a done
Writing manifest to image destination
Storing signatures
[root@087dc3f5ca63 /]# dnf list available podman
Fedora 34 - x86_64                                     6.2 MB/s |  74 MB     00:12
Fedora 34 openh264 (From Cisco) - x86_64               6.0 kB/s | 2.5 kB     00:00
Fedora Modular 34 - x86_64                             2.5 MB/s | 4.9 MB     00:01
Fedora 34 - x86_64 - Updates                            16 MB/s |  25 MB     00:01
Fedora Modular 34 - x86_64 - Updates                   3.7 MB/s | 4.6 MB     00:01
Last metadata expiration check: 0:00:01 ago on Tue Aug 24 20:10:46 2021.
Available Packages
podman.x86_64                          3:3.2.3-2.fc34                           updates
[root@087dc3f5ca63 /]#
TomSweeneyRedHat commented 3 years ago

@cevich I just checked the variant of Fedora used in the build, and it's F34. I just double-checked bodhi and Podman v3.3.0 is stable for F35 and F36. For F33 and F34, they're both still in testing. Apparently, my eyeballs failed me last night when I checked bodhi. So this is working as expected.

cevich commented 3 years ago

Ahh, yes that makes sense given F35 isn't released yet :grin: