podman push to aws manifest failure

[michael-c-hoffman]

Description Podman push to authenticated aws registry fails.

Steps to reproduce the issue:

1. login to aws registery with podman

2. create and tag image

3. docker push /image tag:latest

Describe the results you received: podman push foobar.dkr.ecr.us-east-1.amazonaws.com/helloworld:latest Getting image source signatures Copying blob sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f 199.14 MB / 199.14 MB [==================================================] 1m0s Copying blob sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef 1024 B / 1024 B [==========================================================] 1s Copying config sha256:e83afaabb1f41afb22d7638022ad561cb3cb69f6e014eecafb9664f9e04fb1e5 0 B / 1.21 KB [------------------------------------------------------------] 0s Writing manifest to image destination Error copying image to the remote destination: Error writing manifest: Error uploading manifest latest to foobar.dkr.ecr.us-east-1.amazonaws.com/helloworld: unsupported: Invalid parameter at 'ImageManifest' failed to satisfy constraint: 'Invalid JSON syntax'

Describe the results you expected: Successful upload of image and manifest

Additional information you deem important (e.g. issue happens only occasionally): AWS is expecting a different json format (https://docs.aws.amazon.com/AmazonECR/latest/APIReference/API_PutImage.html) "imageManifest": "string",

podman is sending ImageManifest and errors

Output of podman version:

podman version
Version:       0.10.1.3
Go Version:    go1.10.4
OS/Arch:       linux/amd64

Output of podman info:

podman info
host:
  BuildahVersion: 1.5-dev
  Conmon:
    package: podman-0.10.1.3-1.gitdb08685.fc28.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: 4a03e555ee4105308fa4d814d4ce02a059af0b7f-dirty'
  Distribution:
    distribution: fedora
    version: "28"
  MemFree: 215162880
  MemTotal: 8235417600
  OCIRuntime:
    package: runc-1.0.0-56.dev.git78ef28e.fc28.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 3918188544
  SwapTotal: 3997167616
  arch: amd64
  cpus: 4
  hostname: mhoffman.localdomain
  kernel: 4.18.14-200.fc28.x86_64
  os: linux
  uptime: 24h 12m 32.99s (Approximately 1.00 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ContainerStore:
    number: 5
  GraphDriverName: vfs
  GraphOptions: []
  GraphRoot: /home/mhoffman/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 2
  RunRoot: /run/user/1000/run

Additional environment details (AWS, VirtualBox, physical, etc.): AWS

[rhatdan]

@mtrmac PTAL

[mtrmac]

Thanks for your report.

AWS is expecting a different json format (https://docs.aws.amazon.com/AmazonECR/latest/APIReference/API_PutImage.html) "imageManifest": "string",

podman is not calling that API directly, though it does seem quite possible that the underlying implementation of the registry calls it.

As a random guess, can you try this with podman push --format=v2s2 …, please?

If that fails, mkdir $sometmpdir; podman push foobar.dkr.ecr.us-east-1.amazonaws.com/helloworld:latest dir:$sometmpdir and attach $sometmpdir/manifest.json, please — though it seems extremely unlikely that the JSON is actually syntactically invalid.

---

Either way, the output of podman --log-level=debug push … could be useful as well.

[michael-c-hoffman]

podman push --format=v2s2 036143827064.dkr.ecr.us-east-1.amazonaws.com/ecstasktest:latest
--compress and --format can be set only when pushing to a directory using the 'dir' transport

manifest.json

{"schemaVersion":2,"mediaType":"application/vnd.docker.distribution.manifest.v2+json","config":{"mediaType":"application/vnd.docker.container.image.v1+json","size":1243,"digest":"sha256:e83afaabb1f41afb22d7638022ad561cb3cb69f6e014eecafb9664f9e04fb1e5"},"layers":[{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":208811520,"digest":"sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f"},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":1024,"digest":"sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"}]}

podman push to aws with debug

podman --log-level=debug push 036143827064.dkr.ecr.us-east-1.amazonaws.com/ecstasktest:latest
INFO[0000] running as rootless                          
DEBU[0000] [graphdriver] trying provided driver "vfs"   
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
DEBU[0000] Initializing boltdb state at /home/mhoffman/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] parsed reference into "[vfs@/home/mhoffman/.local/share/containers/storage+/run/user/1000/run]036143827064.dkr.ecr.us-east-1.amazonaws.com/ecstasktest:latest" 
DEBU[0000] parsed reference into "[vfs@/home/mhoffman/.local/share/containers/storage+/run/user/1000/run]@e83afaabb1f41afb22d7638022ad561cb3cb69f6e014eecafb9664f9e04fb1e5" 
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]   Using file:///var/lib/atomic/sigstore      
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/036143827064.dkr.ecr.us-east-1.amazonaws.com 
DEBU[0000] IsRunningImageAllowed for image containers-storage:[vfs@/home/mhoffman/.local/share/containers/storage]@e83afaabb1f41afb22d7638022ad561cb3cb69f6e014eecafb9664f9e04fb1e5 
DEBU[0000]  Using default policy section                
DEBU[0000]  Requirement 0: allowed                      
DEBU[0000] Overall: allowed                             
Getting image source signatures
DEBU[0000] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0000] ... will first try using the original manifest unmodified 
DEBU[0000] Checking /v2/ecstasktest/blobs/sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f 
DEBU[0000] GET https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ 
DEBU[0000] Ping https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ status 401 
DEBU[0000] HEAD https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ecstasktest/blobs/sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f 
DEBU[0000] ... not authorized                           
ERRO[0000] Error copying image to the remote destination: Error checking for blob sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f at destination: Error checking whether a blob sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f exists in 036143827064.dkr.ecr.us-east-1.amazonaws.com/ecstasktest: unauthorized: authentication required 
[mhoffman@mhoffman container]$ podman
podman           podman_awslogin  
[mhoffman@mhoffman container]$ podman_awslogin 
Login Succeeded!
[mhoffman@mhoffman container]$ podman --log-level=debug push 036143827064.dkr.ecr.us-east-1.amazonaws.com/ecstasktest:latest
INFO[0000] running as rootless                          
DEBU[0000] [graphdriver] trying provided driver "vfs"   
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
DEBU[0000] Initializing boltdb state at /home/mhoffman/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] parsed reference into "[vfs@/home/mhoffman/.local/share/containers/storage+/run/user/1000/run]036143827064.dkr.ecr.us-east-1.amazonaws.com/ecstasktest:latest" 
DEBU[0000] parsed reference into "[vfs@/home/mhoffman/.local/share/containers/storage+/run/user/1000/run]@e83afaabb1f41afb22d7638022ad561cb3cb69f6e014eecafb9664f9e04fb1e5" 
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]   Using file:///var/lib/atomic/sigstore      
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/036143827064.dkr.ecr.us-east-1.amazonaws.com 
DEBU[0000] IsRunningImageAllowed for image containers-storage:[vfs@/home/mhoffman/.local/share/containers/storage]@e83afaabb1f41afb22d7638022ad561cb3cb69f6e014eecafb9664f9e04fb1e5 
DEBU[0000]  Using default policy section                
DEBU[0000]  Requirement 0: allowed                      
DEBU[0000] Overall: allowed                             
Getting image source signatures
DEBU[0000] Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0000] ... will first try using the original manifest unmodified 
DEBU[0000] Checking /v2/ecstasktest/blobs/sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f 
DEBU[0000] GET https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ 
DEBU[0000] Ping https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ status 401 
DEBU[0000] HEAD https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ecstasktest/blobs/sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f 
DEBU[0000] ... not present                              
Copying blob sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f
DEBU[0000] exporting filesystem layer "f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f" without compression for blob "sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f" 
DEBU[0000] No compression detected                      
 0 B / 199.14 MB [-------------------------------------------------------------]DEBU[0000] Compressing blob on the fly                  
DEBU[0000] Uploading /v2/ecstasktest/blobs/uploads/     
DEBU[0000] POST https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ecstasktest/blobs/uploads/ 
 8 B / 199.14 MB [>------------------------------------------------------------]DEBU[0001] PATCH https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ecstasktest/blobs/uploads/58b35b66-3aac-795e-bb10-0f025460ac4f 
 199.14 MB / 199.14 MB [=======================================================]DEBU[0060] PUT https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ecstasktest/blobs/uploads/58b35b66-3aac-795e-bb10-0f025460ac4f?digest=sha256%3Aaeb7866da422acc7e93dcf7323f38d7646f6269af33bcdb6647f2094fc4b3bf7 
DEBU[0061] Upload of layer sha256:aeb7866da422acc7e93dcf7323f38d7646f6269af33bcdb6647f2094fc4b3bf7 complete 
 199.14 MB / 199.14 MB [==================================================] 1m0s
DEBU[0061] Checking /v2/ecstasktest/blobs/sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef 
DEBU[0061] HEAD https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ecstasktest/blobs/sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef 
DEBU[0061] ... not present                              
Copying blob sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef
DEBU[0061] exporting filesystem layer "164f60be2ec77141184112c3cc51f398754c0dbf22696f9973d0445275042ba2" without compression for blob "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef" 
DEBU[0061] No compression detected                      
 0 B / 1024 B [----------------------------------------------------------------]DEBU[0061] Compressing blob on the fly                  
DEBU[0061] Uploading /v2/ecstasktest/blobs/uploads/     
DEBU[0061] POST https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ecstasktest/blobs/uploads/ 
 8 B / 1024 B [>---------------------------------------------------------------]DEBU[0061] PATCH https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ecstasktest/blobs/uploads/deb35b66-b143-821b-7c63-7bd412473b1b 
DEBU[0062] PUT https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ecstasktest/blobs/uploads/deb35b66-b143-821b-7c63-7bd412473b1b?digest=sha256%3A4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 
 1024 B / 1024 B [=============================================================]DEBU[0062] Upload of layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 complete 
 1024 B / 1024 B [==========================================================] 1s
Copying config sha256:e83afaabb1f41afb22d7638022ad561cb3cb69f6e014eecafb9664f9e04fb1e5
DEBU[0062] exporting opaque data as blob "sha256:e83afaabb1f41afb22d7638022ad561cb3cb69f6e014eecafb9664f9e04fb1e5" 
DEBU[0062] No compression detected                      
 0 B / 1.21 KB [---------------------------------------------------------------]DEBU[0062] Using original blob without modification     
DEBU[0062] Checking /v2/ecstasktest/blobs/sha256:e83afaabb1f41afb22d7638022ad561cb3cb69f6e014eecafb9664f9e04fb1e5 
DEBU[0062] HEAD https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ecstasktest/blobs/sha256:e83afaabb1f41afb22d7638022ad561cb3cb69f6e014eecafb9664f9e04fb1e5 
DEBU[0063] ... already exists                           
 0 B / 1.21 KB [------------------------------------------------------------] 0s
Writing manifest to image destination
DEBU[0063] PUT https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ecstasktest/manifests/latest 
DEBU[0063] Writing manifest using preferred type application/vnd.oci.image.manifest.v1+json failed: Error writing manifest: Error uploading manifest latest to 036143827064.dkr.ecr.us-east-1.amazonaws.com/ecstasktest: unsupported: Invalid parameter at 'ImageManifest' failed to satisfy constraint: 'Invalid JSON syntax' 
ERRO[0063] Error copying image to the remote destination: Error writing manifest: Error uploading manifest latest to 036143827064.dkr.ecr.us-east-1.amazonaws.com/ecstasktest: unsupported: Invalid parameter at 'ImageManifest' failed to satisfy constraint: 'Invalid JSON syntax' 

[mtrmac]

--compress and --format can be set only when pushing to a directory using the 'dir' transport

Oops, sorry about that. --format AFAICS should in principle work with other transports though, @umohnani8 can you remember why it is dir:-only? #126 does not say. [It might have been my reluctance about exposing this at all, I vaguely remember being reluctant. But I think I’d have been reluctant in writing :) ]

AWS rejecting OCI is still my leading hypothesis, although https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-manifest-formats.html claims that OCI is natively supported.

Until podman push --format works, can you try

$ mkdir $sometmpdir
$ podman push --format=v2s2 foobar.dkr.ecr.us-east-1.amazonaws.com/helloworld:latest dir:$sometmpdir
$ skopeo --debug copy dir:$sometmpdir docker://foobar.dkr.ecr.us-east-1.amazonaws.com/helloworld:latest

, please?

[michael-c-hoffman]

skopeo output shows it successful and was able to verify that via aws console

latestview all sha256:9c9fc1dfcc096c60e0b770fc612ba542f3ef5220aab089fb0bcc6117e39756a7 71.24 2018-10-27 16:10:21 -0400

skopeo --debug copy dir:./image docker://036143827064.dkr.ecr.us-east-1.amazonaws.com/ecstasktest:latest
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]   Using file:///var/lib/atomic/sigstore      
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/036143827064.dkr.ecr.us-east-1.amazonaws.com 
DEBU[0000] IsRunningImageAllowed for image dir:/home/mhoffman/Documents/code/gitlab_michaelchoffman/aws_michaelchoffman/ecstasktest/container/image 
DEBU[0000]  Using default policy section                
DEBU[0000]  Requirement 0: allowed                      
DEBU[0000] Overall: allowed                             
Getting image source signatures
DEBU[0000] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0000] ... will first try using the original manifest unmodified 
DEBU[0000] Checking /v2/ecstasktest/blobs/sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f 
DEBU[0000] GET https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ 
DEBU[0000] Ping https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ err <nil> 
DEBU[0000] Ping https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ status 401 
DEBU[0000] HEAD https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ecstasktest/blobs/sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f 
DEBU[0000] ... not present                              
Copying blob sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f
DEBU[0000] No compression detected                      
 0 B / 199.14 MB [-------------------------------------------------------------]DEBU[0000] Compressing blob on the fly                  
DEBU[0000] Uploading /v2/ecstasktest/blobs/uploads/     
DEBU[0000] POST https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ecstasktest/blobs/uploads/ 
 8 B / 199.14 MB [>------------------------------------------------------------]DEBU[0001] PATCH https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ecstasktest/blobs/uploads/5eb35b92-9770-8087-b937-5fa90ccb69b1 
 199.14 MB / 199.14 MB [=======================================================]DEBU[0060] PUT https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ecstasktest/blobs/uploads/5eb35b92-9770-8087-b937-5fa90ccb69b1?digest=sha256%3Aaeb7866da422acc7e93dcf7323f38d7646f6269af33bcdb6647f2094fc4b3bf7 
DEBU[0061] Upload of layer sha256:aeb7866da422acc7e93dcf7323f38d7646f6269af33bcdb6647f2094fc4b3bf7 complete 
 199.14 MB / 199.14 MB [==================================================] 1m0s
DEBU[0061] Checking /v2/ecstasktest/blobs/sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef 
DEBU[0061] HEAD https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ecstasktest/blobs/sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef 
DEBU[0061] ... not present                              
Copying blob sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef
DEBU[0061] No compression detected                      
 0 B / 1024 B [----------------------------------------------------------------]DEBU[0061] Compressing blob on the fly                  
DEBU[0061] Uploading /v2/ecstasktest/blobs/uploads/     
DEBU[0061] POST https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ecstasktest/blobs/uploads/ 
 8 B / 1024 B [>---------------------------------------------------------------]DEBU[0061] PATCH https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ecstasktest/blobs/uploads/96b35b93-0dae-7a7e-06e9-f71a7ca65354 
 1024 B / 1024 B [=============================================================]DEBU[0062] PUT https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ecstasktest/blobs/uploads/96b35b93-0dae-7a7e-06e9-f71a7ca65354?digest=sha256%3A4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 
DEBU[0062] Upload of layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 complete 
 1024 B / 1024 B [==========================================================] 1s
Copying config sha256:e83afaabb1f41afb22d7638022ad561cb3cb69f6e014eecafb9664f9e04fb1e5
DEBU[0062] No compression detected                      
 0 B / 1.21 KB [---------------------------------------------------------------]DEBU[0062] Using original blob without modification     
DEBU[0062] Checking /v2/ecstasktest/blobs/sha256:e83afaabb1f41afb22d7638022ad561cb3cb69f6e014eecafb9664f9e04fb1e5 
DEBU[0062] HEAD https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ecstasktest/blobs/sha256:e83afaabb1f41afb22d7638022ad561cb3cb69f6e014eecafb9664f9e04fb1e5 
DEBU[0062] ... already exists                           
 0 B / 1.21 KB [------------------------------------------------------------] 0s
Writing manifest to image destination
DEBU[0062] PUT https://036143827064.dkr.ecr.us-east-1.amazonaws.com/v2/ecstasktest/manifests/latest 
Storing signatures

[mtrmac]

Thanks, that confirms the hunch that AWS does not like our OCI manifests. Now, what to do about it…

@rhatdan do we have a contact at Amazon to contact about their OCI support to ask why it fails with Invalid parameter at 'ImageManifest' failed to satisfy constraint: 'Invalid JSON syntax'?

In the worst case, we can of course teach c/image/docker.isManifestInvalidError to recognize the “unsupported” errors so that we fall back to v2s2, but I’d prefer making the OCI support interoperable.

[michael-c-hoffman]

it looks like it is more of a camel case vs aws expecting lowercase then camel case

[mtrmac]

it looks like it is more of a camel case vs aws expecting lowercase then camel case

What points to that? and camel case vs. lowercase of what?

[umohnani8]

@mtrmac yeah I believe --format is for dir only because that is the one that can have either manifest. If you want v2s2 just use the docker-archive transport, if you want oci just use either the oci or oci-archive transport. I guess it makes sense to make format work for the docker:// transport as well.

[michael-c-hoffman]

The aws doc is erroring on pdman sending ImageManifest and aws expecting imageManifest. At least that is what I got from the error and the aws reference docs

On Mon, Oct 29, 2018, 11:29 Miloslav Trmač notifications@github.com wrote:

it looks like it is more of a camel case vs aws expecting lowercase then camel case

What points to that? and camel case vs. lowercase of what?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/containers/libpod/issues/1719#issuecomment-433953701, or mute the thread https://github.com/notifications/unsubscribe-auth/AB35qnbL5EW79XK-7sfDPKj_1yZ961iIks5upx7PgaJpZM4X9I-F .

[mtrmac]

@mtrmac yeah I believe --format is for dir only because that is the one that can have either manifest. If you want v2s2 just use the docker-archive transport, if you want oci just use either the oci or oci-archive transport. I guess it makes sense to make format work for the docker:// transport as well.

(FWIW, dir: is actually not great for alternative formats because it does not record the MIME type. docker-archive is not using v2s2, the tarball format has its own manifest format (only the transport lies and pretends it’s v2s2; in principle, maybe we should stop and support that format natively, but integrating paths vs. blob digests could be rather difficult for little gain.))

[mtrmac]

The aws doc is erroring on pdman sending ImageManifest and aws expecting imageManifest. At least that is what I got from the error and the aws reference docs

That’s interesting, but, again, this code does not use that AWS API at all; so the writer of ImageManifest is also somewhere inside AWS.

[michael-c-hoffman]

so are is it my understanding that podman pushing to aws ecr repositories will not be addressed? Even though it works using skopeo? Is the permanent work around to dump to a directory and use skopeo??? I'm not sure what the next steps for this issue are thanks

[mtrmac]

By far the most preferable outcome is that this works automatically and transparently; for that we need to figure out what is going on and what ECR expects, and that may take some time (i.e. not be done this week).

Modifying podman to allow --format=v2s2 is easy and quick, and should probably happen anyway, but that’s not a real solution and can make the true transparently-working fix less urgent.

[umohnani8]

@mtrmac iirc, I think the main reason for --format was so that the satelite team (I might be forgetting which team) can convert from one format to another easily and they were mainly using the dir: transport so we implemented it for dir. You think we should implement it for docker:// also to support this scenario in podman push?

[mtrmac]

You think we should implement it for docker:// also to support this scenario in podman push?

Overall, yes. The underlying implementation is generic, this is an artificial limitation in podman push that can just be removed.

(I’d personally slightly prefer to solve the ECR incompatibility before we relax this, just so that there remains notable motivation to actually solving the problem in the best way, but we know right now how to relax the option and we don’t know yet how to solve the incompatibility, so it seems a risky bet to delay relaxing the option — and making users’ lives more painful so that we have to care about something is overall the wrong balance.)

[michael-c-hoffman]

Oki was trying to get a sense if the issue would get marked closed not fix and end here. For the moment while not ideal skopea gets me what I need for the moment. Thanks and will continue to follow and provide testing if needed.

On Tue, Oct 30, 2018, 12:23 Miloslav Trmač notifications@github.com wrote:

You think we should implement it for docker:// also to support this scenario in podman push?

Overall, yes. The underlying implementation is generic, this is an artificial limitation in podman push that can just be removed.

(I’d personally slightly prefer to solve the ECR incompatibility before we relax this, just so that there remains notable motivation to actually solving the problem in the best way, but we know right now how to relax the option and we don’t know yet how to solve the incompatibility, so it seems a risky bet to delay relaxing the option — and making users’ lives more painful so that we have to care about something is overall the wrong balance.)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/containers/libpod/issues/1719#issuecomment-434368672, or mute the thread https://github.com/notifications/unsubscribe-auth/AB35qr9bs6MDHTOn_ch0uWF1nfGv21qhks5uqH0JgaJpZM4X9I-F .

[rhatdan]

@mtrmac any update on this?

[mtrmac]

@rhatdan

@rhatdan do we have a contact at Amazon to contact about their OCI support to ask why it fails with Invalid parameter at 'ImageManifest' failed to satisfy constraint: 'Invalid JSON syntax'?

Also, as a possible workaround:

Modifying podman to allow --format=v2s2 is easy and quick, and should probably happen anyway, but that’s not a real solution and can make the true transparently-working fix less urgent.

[rhatdan]

@nalind WDYT?

[markmc]

It looks like ECR is rejecting any OCI manifest without a top-level media type - this needs to be fixed by AWS. See also opencontainers/image-spec#411

I verified this by adding a media type back to the OCI manifest generated by buildah, and a push to ECR works fine now

diff --git a/image.go b/image.go
index 163d3426..577aa035 100644
--- a/image.go
+++ b/image.go
@@ -198,6 +198,7 @@ func (i *containerImageRef) createConfigsAndManifests() (v1.Image, v1.Manifest,
        omanifest := v1.Manifest{
                Versioned: specs.Versioned{
                        SchemaVersion: 2,
+                       MediaType:     v1.MediaTypeImageManifest,
                },
                Config: v1.Descriptor{
                        MediaType: v1.MediaTypeImageConfig,
diff --git a/vendor/github.com/opencontainers/image-spec/specs-go/versioned.go b/vendor/github.com/opencontainers/image-spec/specs-go/versioned.go
index 58a1510f..d01a1a89 100644
--- a/vendor/github.com/opencontainers/image-spec/specs-go/versioned.go
+++ b/vendor/github.com/opencontainers/image-spec/specs-go/versioned.go
@@ -20,4 +20,7 @@ package specs
 type Versioned struct {
        // SchemaVersion is the image manifest schema that this image follows
        SchemaVersion int `json:"schemaVersion"`
+
+       // MediaType is the media type of this schema.
+       MediaType string `json:"mediaType"`
 }

[mtrmac]

It looks like ECR is rejecting any OCI manifest without a top-level media type - this needs to be fixed by AWS.

Thanks.

c/image/docker.isManifestInvalidError can, in that case, be extended to detect this AWS error message, and things could work transparently by downgrading to v2s2 — but, still, AWS needs to be fixed for true OCI pushes to work.

[markmc]

See containers/image#584

Quickly checking out testing options, it looks like the skopeo integration tests might exercise to two existing isManifestInvalidError() cases? So the right thing to do would be to somehow add a test registry that emulates ECR's buggy behavior?

[mtrmac]

I wouldn’t bother — if we are going to fake a non-realistic registry, faking a very limited amount of data for isManifestInvalidError would give us exactly the same information, and would be much easier to implement.

Having a long-term integration test that works against the current version of ECR as it is being deployed in practice would be nice, but that’s another thing entirely, and absolutely not blocking the merging of that small PR.

[markmc]

On Mon, Feb 11, 2019 at 2:54 PM Miloslav Trmač notifications@github.com wrote:

I wouldn’t bother — if we are going to fake a non-realistic registry, faking a very limited amount of data for isManifestInvalidError would give us exactly the same information, and would be much easier to implement.

Cool - I'm not sure what that would look like either, but if you have any hints where to plug such a test in, I might get to it

[mtrmac]

On Mon, Feb 11, 2019 at 2:54 PM Miloslav Trmač @.***> wrote: I wouldn’t bother — if we are going to fake a non-realistic registry, faking a very limited amount of data for isManifestInvalidError would give us exactly the same information, and would be much easier to implement. Cool - I'm not sure what that would look like either, but if you have any hints where to plug such a test in, I might get to it

Add a c/image/docker/docker_image_dest_test.go with a simple Go unit test for that function, nothing fancy. (As you can see, there isn’t a unit test for the other cases, so it is not fair to block https://github.com/containers/image/pull/584 on adding a unit test for the new case either. Just please say whether you plan to do so, so that I can merge it now or wait.)

[markmc]

On Mon, Feb 11, 2019 at 3:00 PM Miloslav Trmač notifications@github.com wrote:

On Mon, Feb 11, 2019 at 2:54 PM Miloslav Trmač @.***> wrote: I wouldn’t bother — if we are going to fake a non-realistic registry, faking a very limited amount of data for isManifestInvalidError would give us exactly the same information, and would be much easier to implement. Cool - I'm not sure what that would look like either, but if you have any hints where to plug such a test in, I might get to it

Add a c/image/docker/docker_image_dest_test.go with a simple Go unit test for that function, nothing fancy. (As you can see, there isn’t a unit test for the other cases, so it is not fair to block containers/image#584 https://github.com/containers/image/pull/584 on adding a unit test for the new case either. Just please say whether you plan to do so, so that I can merge it now or wait.)

Thanks. No immediate time to get to it, so I wouldn't wait.

[mtrmac]

Should be fixed in c/image by https://github.com/containers/image/pull/584 .

[rhatdan]

Fixed in master.