search

containers / skopeo

Work with remote images registries - retrieving information, images, signing content
Apache License 2.0
8.33k stars 785 forks source link

error uploading imagine to openshift integrated registry #258

Closed raffaelespazzoli closed 7 years ago

raffaelespazzoli commented 7 years ago

I'm using openshift 3.3.0 (cdk) with integrated registry. when I upload an image I get the following error: I'm running skopeo on fedora 24

skopeo version:

[rspazzol@localhost tmp]$ skopeo -v
skopeo version 0.1.17-dev commit: 980ff3eadd57f46479c28078c2489058081b0165

skopeo command and log:

[rspazzol@localhost tmp]$ skopeo --debug --tls-verify=false copy --dest-creds `oc whoami`:`oc whoami -t` --src-creds `oc whoami`:`oc whoami -t` docker://hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/jenkins2/docker:latest docker://hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/jenkins2/docker:ciao2
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]   Using file:///var/lib/atomic/sigstore      
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]  No signature storage configuration found for hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/jenkins2/docker:latest 
DEBU[0000] IsRunningImageAllowed for image docker:hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/jenkins2/docker:latest 
DEBU[0000]  Using default policy section                
DEBU[0000]  Requirement 0: allowed                      
DEBU[0000] Overall: allowed                             
DEBU[0000] GET https://hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/v2/ 
DEBU[0000] Ping https://hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/v2/ err <nil> 
DEBU[0000] Ping https://hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/v2/ status 401 
DEBU[0000] GET https://hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/v2/jenkins2/docker/manifests/latest 
Getting image source signatures
DEBU[0000] Manifest MIME type application/vnd.docker.distribution.manifest.v2+json is declared supported by the destination 
Copying blob sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092f6862f
DEBU[0000] Downloading jenkins2/docker/blobs/sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092f6862f 
DEBU[0000] GET https://hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/v2/jenkins2/docker/blobs/sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092f6862f 
DEBU[0001] Detected compression format gzip             
 0 B / 2.21 MB [---------------------------------------------------------------]DEBU[0001] Using original blob without modification     
DEBU[0001] Checking jenkins2/docker/blobs/sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092f6862f 
DEBU[0001] GET https://hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/v2/ 
DEBU[0001] Ping https://hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/v2/ err <nil> 
DEBU[0001] Ping https://hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/v2/ status 401 
DEBU[0001] HEAD https://hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/v2/jenkins2/docker/blobs/sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092f6862f 
DEBU[0001] ... already exists, not uploading            

Copying blob sha256:7601c2ad1cd11213e66512af4a8f4372a4870541b8cf79a9a1058742bf65d6e7
DEBU[0001] Downloading jenkins2/docker/blobs/sha256:7601c2ad1cd11213e66512af4a8f4372a4870541b8cf79a9a1058742bf65d6e7 
DEBU[0001] GET https://hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/v2/jenkins2/docker/blobs/sha256:7601c2ad1cd11213e66512af4a8f4372a4870541b8cf79a9a1058742bf65d6e7 
DEBU[0003] Detected compression format gzip             
 0 B / 893.61 KB [-------------------------------------------------------------]DEBU[0003] Using original blob without modification     
DEBU[0003] Checking jenkins2/docker/blobs/sha256:7601c2ad1cd11213e66512af4a8f4372a4870541b8cf79a9a1058742bf65d6e7 
DEBU[0003] HEAD https://hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/v2/jenkins2/docker/blobs/sha256:7601c2ad1cd11213e66512af4a8f4372a4870541b8cf79a9a1058742bf65d6e7 
DEBU[0003] ... already exists, not uploading            

Copying blob sha256:a2fa41e1947e55748e6bb64fdc8decb92de95bf1f7f69c9da3ed022983953a9f
DEBU[0003] Downloading jenkins2/docker/blobs/sha256:a2fa41e1947e55748e6bb64fdc8decb92de95bf1f7f69c9da3ed022983953a9f 
DEBU[0003] GET https://hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/v2/jenkins2/docker/blobs/sha256:a2fa41e1947e55748e6bb64fdc8decb92de95bf1f7f69c9da3ed022983953a9f 
DEBU[0004] Detected compression format gzip             
 0 B / 27.43 MB [--------------------------------------------------------------]DEBU[0004] Using original blob without modification     
DEBU[0004] Checking jenkins2/docker/blobs/sha256:a2fa41e1947e55748e6bb64fdc8decb92de95bf1f7f69c9da3ed022983953a9f 
DEBU[0004] HEAD https://hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/v2/jenkins2/docker/blobs/sha256:a2fa41e1947e55748e6bb64fdc8decb92de95bf1f7f69c9da3ed022983953a9f 
DEBU[0005] ... already exists, not uploading            

Copying blob sha256:7ee74a896d835287baba1aa769cdc88ec2b48879aabd9e1bfb3382de803495d0
DEBU[0005] Downloading jenkins2/docker/blobs/sha256:7ee74a896d835287baba1aa769cdc88ec2b48879aabd9e1bfb3382de803495d0 
DEBU[0005] GET https://hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/v2/jenkins2/docker/blobs/sha256:7ee74a896d835287baba1aa769cdc88ec2b48879aabd9e1bfb3382de803495d0 
DEBU[0005] Detected compression format gzip             
 0 B / 489 B [-----------------------------------------------------------------]DEBU[0005] Using original blob without modification     
DEBU[0005] Checking jenkins2/docker/blobs/sha256:7ee74a896d835287baba1aa769cdc88ec2b48879aabd9e1bfb3382de803495d0 
DEBU[0005] HEAD https://hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/v2/jenkins2/docker/blobs/sha256:7ee74a896d835287baba1aa769cdc88ec2b48879aabd9e1bfb3382de803495d0 
DEBU[0006] ... already exists, not uploading            

Copying config sha256:cf693ec9b5c76a8d05bb5e0c38257d5a575b0e1a7af56037cea8f54e9a751cab
DEBU[0006] Downloading jenkins2/docker/blobs/sha256:cf693ec9b5c76a8d05bb5e0c38257d5a575b0e1a7af56037cea8f54e9a751cab 
DEBU[0006] GET https://hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/v2/jenkins2/docker/blobs/sha256:cf693ec9b5c76a8d05bb5e0c38257d5a575b0e1a7af56037cea8f54e9a751cab 
DEBU[0007] No compression detected                      
 0 B / 3.26 KB [---------------------------------------------------------------]DEBU[0007] Using original blob without modification     
DEBU[0007] Checking jenkins2/docker/blobs/sha256:cf693ec9b5c76a8d05bb5e0c38257d5a575b0e1a7af56037cea8f54e9a751cab 
DEBU[0007] HEAD https://hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/v2/jenkins2/docker/blobs/sha256:cf693ec9b5c76a8d05bb5e0c38257d5a575b0e1a7af56037cea8f54e9a751cab 
DEBU[0007] ... already exists, not uploading            

Writing manifest to image destination
DEBU[0007] PUT https://hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/v2/jenkins2/docker/manifests/ciao2 
DEBU[0007] Error body {"errors":[{"code":"UNKNOWN","message":"unknown error","detail":{"code":"MANIFEST_INVALID","message":"manifest invalid","detail":{}}}]}

DEBU[0007] Error uploading manifest, status 500, &http.Response{Status:"500 Internal Server Error", StatusCode:500, Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Content-Type":[]string{"application/json; charset=utf-8"}, "Docker-Distribution-Api-Version":[]string{"registry/2.0"}, "Date":[]string{"Thu, 01 Dec 2016 01:49:25 GMT"}, "Content-Length":[]string{"136"}}, Body:(*http.bodyEOFSignal)(0xc8201320c0), ContentLength:136, TransferEncoding:[]string(nil), Close:true, Trailer:http.Header(nil), Request:(*http.Request)(0xc8204c1340), TLS:(*tls.ConnectionState)(0xc8200c91e0)} 
FATA[0007] Error writing manifest: Error uploading manifest to jenkins2/docker/manifests/ciao2, status 500

registry log:

time="2016-11-30T20:49:25.991156037-05:00" level=debug msg="Origin auth: checking for access to repository:jenkins2/docker:push" go.version=go1.6.2 http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.request.host="hub.openshift.rhel-cdk.10.1.2.2.xip.io:443" http.request.id=7e0c2a0d-165b-4c9c-9d1b-c1c1fb3391a5 http.request.method=PUT http.request.remoteaddr="172.17.0.1:41536" http.request.uri="/v2/jenkins2/docker/manifests/ciao2" http.request.useragent="Go-http-client/1.1" instance.id=ea1c8051-02f8-410f-b4fb-a81bd8cec902 vars.name="jenkins2/docker" vars.reference=ciao2 
time="2016-11-30T20:49:25.993886421-05:00" level=debug msg=PutImageManifest go.version=go1.6.2 http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.request.host="hub.openshift.rhel-cdk.10.1.2.2.xip.io:443" http.request.id=7e0c2a0d-165b-4c9c-9d1b-c1c1fb3391a5 http.request.method=PUT http.request.remoteaddr="172.17.0.1:41536" http.request.uri="/v2/jenkins2/docker/manifests/ciao2" http.request.useragent="Go-http-client/1.1" instance.id=ea1c8051-02f8-410f-b4fb-a81bd8cec902 vars.name="jenkins2/docker" vars.reference=ciao2 
172.17.0.1 - - [30/Nov/2016:20:49:25 -0500] "PUT /v2/jenkins2/docker/manifests/ciao2 HTTP/1.1" 500 136 "" "Go-http-client/1.1"
time="2016-11-30T20:49:25.994113514-05:00" level=warning msg="the ResponseWriter does not implement CloseNotifier (type: *context.instrumentedResponseWriter)" go.version=go1.6.2 http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.request.host="hub.openshift.rhel-cdk.10.1.2.2.xip.io:443" http.request.id=7e0c2a0d-165b-4c9c-9d1b-c1c1fb3391a5 http.request.method=PUT http.request.remoteaddr="172.17.0.1:41536" http.request.uri="/v2/jenkins2/docker/manifests/ciao2" http.request.useragent="Go-http-client/1.1" instance.id=ea1c8051-02f8-410f-b4fb-a81bd8cec902 vars.name="jenkins2/docker" vars.reference=ciao2 
time="2016-11-30T20:49:25.99436406-05:00" level=error msg="response completed with error" err.code=unknown err.detail="manifest invalid: manifest invalid" err.message="unknown error" go.version=go1.6.2 http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.request.host="hub.openshift.rhel-cdk.10.1.2.2.xip.io:443" http.request.id=7e0c2a0d-165b-4c9c-9d1b-c1c1fb3391a5 http.request.method=PUT http.request.remoteaddr="172.17.0.1:41536" http.request.uri="/v2/jenkins2/docker/manifests/ciao2" http.request.useragent="Go-http-client/1.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=7.452421ms http.response.status=500 http.response.written=136 instance.id=ea1c8051-02f8-410f-b4fb-a81bd8cec902 vars.name="jenkins2/docker" vars.reference=ciao2

I'm not sure if I'm doing something wrong or I'm finding a bug. Thanks.

runcom commented 7 years ago

Just to narrow this down a bit since I'm seeing your using these new *creds flags. Are you able to make this work by not using the flags and just oc login?

raffaelespazzoli commented 7 years ago

@runcom , Antonio, I don't understand the question. did you mean docker login? Because I didn't think that an oc login had any bearing on skopeo behavior...

Anyway in order to do an oc whoami, which is part of the command, I had to previously login to oc.

Plus why would the login phase have anything to do with the manigest format, which seems to be the reason why the command is failing?

runcom commented 7 years ago

Plus why would the login phase have anything to do with the manigest format, which seems to be the reason why the command is failing?

never mind, I replied too fast w/o looking at the logs - @mtrmac PTAL

mtrmac commented 7 years ago 
DEBU[0000] Manifest MIME type application/vnd.docker.distribution.manifest.v2+json is declared supported by the destination

Yeah, this is #86. For now, using atomic: instead of docker:// in the destination specification should work around this. Does that work?

raffaelespazzoli commented 7 years ago

I changed my command line to 

skopeo --debug --tls-verify=false copy --dest-creds `oc whoami`:`oc whoami -t` --src-creds `oc whoami`:`oc whoami -t` atomic:hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/jenkins2/docker:latest atomic:hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/jenkins2/docker:ciao2

as instructed. I now get this error

DEBU[0000] cmdConfig: &openshift.deferredLoadingClientConfig{loadingRules:(*openshift.clientConfigLoadingRules)(0xc82027df00), clientConfig:openshift.clientConfig(nil)} 
DEBU[0000] restConfig: &openshift.restConfig{Host:"https://10.1.2.2:8443", Username:"", Password:"", BearerToken:"ePPzhRzfxj_pSdv3C19tIuWFafcYuVVkR3NwDPA5elg", restTLSClientConfig:openshift.restTLSClientConfig{CertFile:"", KeyFile:"", CAFile:"", CertData:[]uint8(nil), KeyData:[]uint8(nil), CAData:[]uint8(nil)}, Insecure:true} 
DEBU[0000] URL: url.URL{Scheme:"https", Opaque:"", User:(*url.Userinfo)(nil), Host:"10.1.2.2:8443", Path:"", RawPath:"", RawQuery:"", Fragment:""} 
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]   Using file:///var/lib/atomic/sigstore      
DEBU[0000] cmdConfig: &openshift.deferredLoadingClientConfig{loadingRules:(*openshift.clientConfigLoadingRules)(0xc820352320), clientConfig:openshift.clientConfig(nil)} 
DEBU[0000] restConfig: &openshift.restConfig{Host:"https://10.1.2.2:8443", Username:"", Password:"", BearerToken:"ePPzhRzfxj_pSdv3C19tIuWFafcYuVVkR3NwDPA5elg", restTLSClientConfig:openshift.restTLSClientConfig{CertFile:"", KeyFile:"", CAFile:"", CertData:[]uint8(nil), KeyData:[]uint8(nil), CAData:[]uint8(nil)}, Insecure:true} 
DEBU[0000] URL: url.URL{Scheme:"https", Opaque:"", User:(*url.Userinfo)(nil), Host:"10.1.2.2:8443", Path:"", RawPath:"", RawQuery:"", Fragment:""} 
DEBU[0000] IsRunningImageAllowed for image atomic:hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/jenkins2/docker:latest 
DEBU[0000]  Using default policy section                
DEBU[0000]  Requirement 0: allowed                      
DEBU[0000] Overall: allowed                             
DEBU[0000] GET {https  <nil> 10.1.2.2:8443 /oapi/v1/namespaces/jenkins2/imagestreams/docker   } 
DEBU[0000] Got body: {"kind":"ImageStream","apiVersion":"v1","metadata":{"name":"docker","namespace":"jenkins2","selfLink":"/oapi/v1/namespaces/jenkins2/imagestreams/docker","uid":"87f04d18-b1e7-11e6-8796-5254007f3655","resourceVersion":"130672","generation":10,"creationTimestamp":"2016-11-24T01:44:25Z","labels":{"app":"docker"},"annotations":{"openshift.io/generated-by":"OpenShiftNewApp","openshift.io/image.dockerRepositoryCheck":"2016-11-26T05:20:11Z"}},"spec":{"tags":[{"name":"latest","annotations":{"openshift.io/imported-from":"docker"},"from":{"kind":"DockerImage","name":"docker"},"generation":10,"importPolicy":{}}]},"status":{"dockerImageRepository":"172.30.144.245:5000/jenkins2/docker","tags":[{"tag":"latest","items":[{"created":"2016-11-26T05:20:11Z","dockerImageReference":"docker@sha256:83795a384fd7321e1a58d06c2fa2b86a0f0ce9e4a75c91aa382317b3c453671d","image":"sha256:83795a384fd7321e1a58d06c2fa2b86a0f0ce9e4a75c91aa382317b3c453671d","generation":10},{"created":"2016-11-26T04:39:00Z","dockerImageReference":"docker@sha256:9cfeaf7b1d09b5a388b8e98fcd132fdf272edd13751b2a1057a781daac07cab7","image":"sha256:9cfeaf7b1d09b5a388b8e98fcd132fdf272edd13751b2a1057a781daac07cab7","generation":8},{"created":"2016-11-26T04:37:35Z","dockerImageReference":"docker@sha256:83795a384fd7321e1a58d06c2fa2b86a0f0ce9e4a75c91aa382317b3c453671d","image":"sha256:83795a384fd7321e1a58d06c2fa2b86a0f0ce9e4a75c91aa382317b3c453671d","generation":6},{"created":"2016-11-26T04:36:01Z","dockerImageReference":"docker@sha256:9cfeaf7b1d09b5a388b8e98fcd132fdf272edd13751b2a1057a781daac07cab7","image":"sha256:9cfeaf7b1d09b5a388b8e98fcd132fdf272edd13751b2a1057a781daac07cab7","generation":4},{"created":"2016-11-24T01:44:26Z","dockerImageReference":"docker@sha256:83795a384fd7321e1a58d06c2fa2b86a0f0ce9e4a75c91aa382317b3c453671d","image":"sha256:83795a384fd7321e1a58d06c2fa2b86a0f0ce9e4a75c91aa382317b3c453671d","generation":2}]},{"tag":"ciao","items":[{"created":"2016-11-25T16:21:50Z","dockerImageReference":"172.30.144.245:5000/jenkins2/docker@sha256:4cef65809efb24dc9616fbe7dc880e6e3687689b91dbafeb93fcf36654b0da68","image":"sha256:4cef65809efb24dc9616fbe7dc880e6e3687689b91dbafeb93fcf36654b0da68","generation":10}]}]}}

DEBU[0000] Got content-type: application/json           
DEBU[0000] tag event &openshift.tagEvent{DockerImageReference:"docker@sha256:83795a384fd7321e1a58d06c2fa2b86a0f0ce9e4a75c91aa382317b3c453671d", Image:"sha256:83795a384fd7321e1a58d06c2fa2b86a0f0ce9e4a75c91aa382317b3c453671d"} 
FATA[0000] Error initializing image from source atomic:hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/jenkins2/docker:latest: Invalid format of docker reference docker@sha256:83795a384fd7321e1a58d06c2fa2b86a0f0ce9e4a75c91aa382317b3c453671d: missing '/'
sabre1041 commented 7 years ago

Using the following command, I was able to copy between two OpenShift registries

./skopeo --policy default-policy.json  --tls-verify=false copy --src-creds=openshift:WepBAoU6sY4_reJStK29GewgzYx6YoE3-nSdPWDhQus --dest-creds=openshift:WepBAoU6sY4_reJStK29GewgzYx6YoE3-nSdPWDhQus docker://hub.openshift.rhel-cdk.10.1.2.2.xip.io/test-promote/httpd-app:latest docker://hub.openshift.rhel-cdk.10.1.2.2.xip.io/skopeo-test/httpd-app:latest

CDK with OCP 3.3 (pre release) and built and running skopeo on RHEL 7.2

mtrmac commented 7 years ago 
skopeo --debug --tls-verify=false copy --dest-creds `oc whoami`:`oc whoami -t` --src-creds `oc whoami`:`oc whoami -t` atomic:hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/jenkins2/docker:latest atomic:hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/jenkins2/docker:ciao2

@aweiteka in this case, the source image (hub.openshift.rhel-cdk.10.1.2.2.xip.io:443/jenkins2/docker:latest) has a tagEvent value:

DEBU[0000] tag event &openshift.tagEvent{DockerImageReference:"docker@sha256:83795a384fd7321e1a58d06c2fa2b86a0f0ce9e4a75c91aa382317b3c453671d", Image:"sha256:83795a384fd7321e1a58d06c2fa2b86a0f0ce9e4a75c91aa382317b3c453671d"}

Any idea why the DockerImageReference is a bare docker without a hostname (not even the incorrect internal IP address a usual)?

@sabre1041 as another workaround, try using atomic: only for the destination, not the source.

mtrmac commented 7 years ago

Using the following command, I was able to copy between two OpenShift registries

Is the difference only in the image names? Do they differ in schema versions perhaps?

aweiteka commented 7 years ago

Hmm, I don't know, although I'm not sure I completely follow the entire issue.

sabre1041 commented 7 years ago

@mtrmac It was working successful for my test. I will attempt to build a bunch of different images and try a number of different command to see if i experience any unsuccessful copy activities

mtrmac commented 7 years ago

The original issue (missing support for schema1-embedded name:tag) has just been fixed in #337 . Are there any outstanding failures?

aweiteka commented 7 years ago

The original issue (missing support for schema1-embedded name:tag) has just been fixed in #337 . Are there any outstanding failures?

@raffaelespazzoli ^^ can we close?

raffaelespazzoli commented 7 years ago

we can close it thanks.

2017-05-11 14:22 GMT-04:00 Aaron Weitekamp notifications@github.com:

The original issue (missing support for schema1-embedded name:tag) has just been fixed in #337 https://github.com/projectatomic/skopeo/pull/337 . Are there any outstanding failures?

@raffaelespazzoli https://github.com/raffaelespazzoli ^^ can we close?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/projectatomic/skopeo/issues/258#issuecomment-300875486, or mute the thread https://github.com/notifications/unsubscribe-auth/AF5I3F4ppseS7_3wH3JwszRCjd4Q7Zcuks5r41HugaJpZM4LA60d .

-- ciao/bye Raffaele