Closed TomSweeneyRedHat closed 4 years ago
@mtrmac Any ideas?
This really looks like the server is replying with HTML. That might be the case if the registry is set up correctly but incorrectly reporting errors (like Quay.io does), e.g. on a missing image, or it just might be an incorrect server altogether.
Are you absolutely sure this is the same configuration? What is the Docker command that succeeds?
Failing that, full debug logs (podman --log-level=debug …
) and possibly applying https://github.com/mtrmac/image/tree/http-dump might be useful, but my 95% bet is on the above.
The same command with docker, I only change podman by docker:
~$ podman pull registry:5000/security/alpine:latest
~$ docker pull registry:5000/security/alpine:latest
The full log is the following:
~$ podman --log-level=debug pull registry:5000/security/alpine:latest
DEBU[0000] using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/rodri/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver vfs
DEBU[0000] Using graph root /home/rodri/.local/share/containers/storage
DEBU[0000] Using run root /run/user/1000
DEBU[0000] Using static dir /home/rodri/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp
DEBU[0000] Using volume path /home/rodri/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "vfs"
DEBU[0000] Initializing event backend journald
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument
DEBU[0000] using runtime "/usr/lib/cri-o-runc/sbin/runc"
DEBU[0000] Failed to add podman to systemd sandbox cgroup: Process org.freedesktop.systemd1 exited with status 1
INFO[0000] running as rootless
DEBU[0000] using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/rodri/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver vfs
DEBU[0000] Using graph root /home/rodri/.local/share/containers/storage
DEBU[0000] Using run root /run/user/1000
DEBU[0000] Using static dir /home/rodri/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp
DEBU[0000] Using volume path /home/rodri/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] Initializing event backend journald
DEBU[0000] using runtime "/usr/lib/cri-o-runc/sbin/runc"
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument
DEBU[0000] parsed reference into "[vfs@/home/rodri/.local/share/containers/storage+/run/user/1000]registry:5000/security/alpine:latest"
Trying to pull registry:5000/security/alpine:latest...
DEBU[0000] reference rewritten from 'registry:5000/security/alpine:latest' to 'registry:5000/security/alpine:latest'
DEBU[0000] Trying to pull "registry:5000/security/alpine:latest"
DEBU[0000] Returning credentials from /run/user/1000/auth.json
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration
DEBU[0000] Using "default-docker" configuration
DEBU[0000] No signature storage configuration found for registry:5000/security/alpine:latest
DEBU[0000] error accessing certs directory due to permissions: stat /etc/docker/certs.d/registry:5000: permission denied
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry:5000
DEBU[0000] Skipping scan of /etc/docker/certs.d/registry:5000 due to permission error: open /etc/docker/certs.d/registry:5000: permission denied
DEBU[0000] GET https://registry:5000/v2/
DEBU[0003] Ping https://registry:5000/v2/ status 200
DEBU[0003] GET https://registry:5000/v2/security/alpine/manifests/latest
DEBU[0009] Using blob info cache at /home/rodri/.local/share/containers/cache/blob-info-cache-v1.boltdb
DEBU[0009] IsRunningImageAllowed for image docker:registry:5000/security/alpine:latest
DEBU[0009] Using default policy section
DEBU[0009] Requirement 0: allowed
DEBU[0009] Overall: allowed
DEBU[0009] Error pulling image ref //registry:5000/security/alpine:latest: Error initializing image from source docker://registry:5000/security/alpine:latest: invalid character '<' looking for beginning of value
invalid character '<' looking for beginning of value
DEBU[0009] [graphdriver] trying provided driver "vfs"
ERRO[0009] error pulling image "registry:5000/security/alpine:latest": unable to pull registry:5000/security/alpine:latest: unable to pull image: Error initializing image from source docker://registry:5000/security/alpine:latest: invalid character '<' looking for beginning of value
This is my registries.conf:
~$ cat /home/rodri/.config/containers/registries.conf
This is a system-wide configuration file used to
keep track of registries for various container backends.
It adheres to TOML format and does not support recursive
lists of registries.
The default location for this configuration file is /etc/containers/registries.conf.
The only valid categories are: 'registries.search', 'registries.insecure',
and 'registries.block'.
[registries.search]
registries = ['myregistry.com', 'docker.io', 'registry.fedoraproject.org', 'quay.io', egistry.access.redhat.com', 'registry.centos.org']
If you need to access insecure registries, add the registry's fully-qualified name.
An insecure registry is one that does not have a valid SSL certificate or only does HTTP.
[registries.insecure]
registries = ['myregistry.com']
\# If you need to block pull access from a registry, uncomment the section below
and add the registries fully-qualified name.
[registries.block]
registries = []
But I don't know if the private registry must have some special configuration for podman
~$ podman --log-level=debug pull registry:5000/security/alpine:latest … DEBU[0000] GET https://registry:5000/v2/ DEBU[0003] Ping https://registry:5000/v2/ status 200 DEBU[0003] GET https://registry:5000/v2/security/alpine/manifests/latest … DEBU[0009] Error pulling image ref //registry:5000/security/alpine:latest: Error initializing image from source docker://registry:5000/security/alpine:latest: invalid character '<' looking for beginning of value
Thanks, nothing in there looks unexpected.
skopeo inspect --raw docker://registry:5000/security/alpine:latest
should (with a new enough skopeo, after https://github.com/containers/skopeo/pull/708 ) print the manifest that is returned by the registry; if my guess is right that it is actually HTML, that could help show what is going on.
This is the inspect return:
$skopeo inspect --creds=user:pass --raw docker://myregistry:5000/security/alpine:latest
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 1512,
"digest": "sha256:37eec16f187294a31cf56273bd544eaf75f7972e309dce838c18be2dd1aa0a45"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 1928586,
"digest": "sha256:30064267e5b89281ce906c124096c47516d73ac7219e30d678b934eedc91816c"
}
]
}
Thanks to you.
OK, I’m afraid that makes no sense to me at all. Can anyone else see what is going on?
Any chance you can replicate this on a public artifactory where we could play with it?
The only niggle I can see in the above is registry:5000
vs. myregistry:5000
vs. myregistry.com
.
I guess none of these hostnames are real, are they? That should be fine, as long as the podman
and skopeo
commands really use exactly the same host name, and are not contacting a different server or a different virtual host or something like that. (There should be an exactly the same
DEBU[0003] GET https://registry:5000/v2/security/alpine/manifests/latest
entry in podman --log-level=debug pull
and skopeo --debug inspect
.)
yes, the hostnames aren't reals. I've launched the following commands:
~$ **skopeo --debug inspect --creds=user:pass --raw docker://myregistry.com:5000/security/alpine:latest**
DEBU[0000] reference rewritten from 'myregistry.com:5000/security/alpine:latest' to 'myregistry.com:5000/security/alpine:latest'
DEBU[0000] Trying to pull "myregistry.com:5000/security/alpine:latest"
DEBU[0000] Returning credentials from DockerAuthConfig
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration
DEBU[0000] Using "default-docker" configuration
DEBU[0000] No signature storage configuration found for myregistry.com:5000/security/alpine:latest
DEBU[0000] error accessing certs directory due to permissions: stat /etc/docker/certs.d/myregistry.com:5000: permission denied
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/myregistry.com:5000
DEBU[0000] Skipping scan of /etc/docker/certs.d/myregistry.com:5000 due to permission error: open /etc/docker/certs.d/myregistry.com:5000: permission denied
DEBU[0000] GET https://myregistry.com:5000/v2/
DEBU[0005] Ping https://myregistry.com:5000/v2/ status 401
DEBU[0005] GET https://myregistry.com:5000/artifactory/api/docker/docker-repo/v2/token?account=user-bot&scope=repository%3Asecurity%2Falpine%3Apull&service=myregistry.com%3A5000
DEBU[0005] GET https://myregistry.com:5000/v2/security/alpine/manifests/latest
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 1512,
"digest": "sha256:37eec16f187294a31cf56273bd544eaf75f7972e309dce838c38be2dd2aa0a45"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 1928586,
"digest": "sha256:30064267e5b89281ce906c124096c47516d73ac7219e30d678b934eedc93826c"
}
]
}
and:
~$ **podman --log-level=debug pull myregistry.com:5000/security/alpine:latest**
DEBU[0000] using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/rodri/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver vfs
DEBU[0000] Using graph root /home/rodri/.local/share/containers/storage
DEBU[0000] Using run root /run/user/1000
DEBU[0000] Using static dir /home/rodri/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp
DEBU[0000] Using volume path /home/rodri/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "vfs"
DEBU[0000] Initializing event backend journald
DEBU[0000] using runtime "/usr/lib/cri-o-runc/sbin/runc"
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument
DEBU[0000] Failed to add podman to systemd sandbox cgroup: Process org.freedesktop.systemd1 exited with status 1
INFO[0000] running as rootless
DEBU[0000] using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/rodri/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver vfs
DEBU[0000] Using graph root /home/rodri/.local/share/containers/storage
DEBU[0000] Using run root /run/user/1000
DEBU[0000] Using static dir /home/rodri/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp
DEBU[0000] Using volume path /home/rodri/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] Initializing event backend journald
DEBU[0000] using runtime "/usr/lib/cri-o-runc/sbin/runc"
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument
DEBU[0000] parsed reference into "[vfs@/home/rodri/.local/share/containers/storage+/run/user/1000]myregistry.com:5000/security/alpine:latest"
Trying to pull myregistry.com:5000/security/alpine:latest...
DEBU[0000] reference rewritten from 'myregistry.com:5000/security/alpine:latest' to 'myregistry.com:5000/security/alpine:latest'
DEBU[0000] Trying to pull "myregistry.com:5000/security/alpine:latest"
DEBU[0000] Returning credentials from /run/user/1000/containers/auth.json
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration
DEBU[0000] Using "default-docker" configuration
DEBU[0000] No signature storage configuration found for myregistry.com:5000/security/alpine:latest
DEBU[0000] error accessing certs directory due to permissions: stat /etc/docker/certs.d/myregistry.com:5000: permission denied
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/myregistry.com:5000
DEBU[0000] Skipping scan of /etc/docker/certs.d/myregistry.com:5000 due to permission error: open /etc/docker/certs.d/myregistry.com:5000: permission denied
DEBU[0000] GET https://myregistry.com:5000/v2/
DEBU[0001] Ping https://myregistry.com:5000/v2/ status 200
DEBU[0001] **GET https://myregistry.com:5000/v2/security/alpine/manifests/latest**
DEBU[0002] Using blob info cache at /home/rodri/.local/share/containers/cache/blob-info-cache-v1.boltdb
DEBU[0002] IsRunningImageAllowed for image docker:myregistry.com:5000/security/alpine:latest
DEBU[0002] Using default policy section
DEBU[0002] Requirement 0: allowed
DEBU[0002] Overall: allowed
DEBU[0002] Error pulling image ref //myregistry.com:5000/security/alpine:latest: Error initializing image from source docker://myregistry.com:5000/security/alpine:latest: invalid character '<' looking for beginning of value
invalid character '<' looking for beginning of value
DEBU[0002] [graphdriver] trying provided driver "vfs"
ERRO[0002] error pulling image "myregistry.com:5000/security/alpine:latest": unable to pull myregistry.com:5000/security/alpine:latest: unable to pull image: Error initializing image from source docker://myregistry.com:5000/security/alpine:latest: invalid character '<' looking for beginning of value
The hostsnames are the same in both.
Any chance you can replicate this on a public artifactory where we could play with it?
I pulled an image from a docker hub with Podman, this image is pushed to a my private registry and when i try to pull it... returns the error
~$ **skopeo --debug inspect --creds=user:pass --raw docker://myregistry.com:5000/security/alpine:latest** … DEBU[0000] GET https://myregistry.com:5000/v2/ DEBU[0005] Ping https://myregistry.com:5000/v2/ status 401 DEBU[0005] GET https://myregistry.com:5000/artifactory/api/docker/docker-repo/v2/token?account=user-bot&scope=repository%3Asecurity%2Falpine%3Apull&service=myregistry.com%3A5000 DEBU[0005] GET https://myregistry.com:5000/v2/security/alpine/manifests/latest
and:
~$ **podman --log-level=debug pull myregistry.com:5000/security/alpine:latest** DEBU[0000] GET https://myregistry.com:5000/v2/ DEBU[0001] Ping https://myregistry.com:5000/v2/ status 200 DEBU[0001] **GET https://myregistry.com:5000/v2/security/alpine/manifests/latest**
The hostsnames are the same in both.
How is it possible, then that /v2/
has status 401 (“provide credentials”) with Skopeo, but status 200 (”OK, no credentials necessary”) with Podman? Correspondingly, Skopeo does send credentials, but Podman does not, and the HTTP may well be an “unauthorized” message or something like that.
(It might still be a behavior difference between Skopeo and Podman, but there’s rather little that can vary in this initial request.)
Before the podman pull I sent a podman login to myregistry:
~$ podman login -u user -p pass myregistry:5000
Login Succeeded!
with the same user:pass that i used for skopeo command
Yes, but that’s irrelevant because the registry does not ask for any credentials, when contacted by Podman.
This issue had no activity for 30 days. In the absence of activity or the "do-not-close" label, the issue will be automatically closed within 7 days.
@TomSweeneyRedHat @mtrmac Should this issue be closed?
I've added the do-not-close for now, I'd like to keep it open at least until early next year to see if we can chew on it a bit.
I am going to close, since I don't see any movement on this. Reopen if I am mistaken.
Might be related (or not), I'm getting the following on pushing in Artifactory + Docker:
unauthorized: Pushing Docker images with manifest v2 schema 1 to this repository is blocked.
Does podman
support manifest v2 schema 2? Thanks.
@sectorsize512 That’s completely unrelated; close to https://github.com/containers/skopeo/issues/651 but probably ultimately different. Either way, please file a separate issue.
I also have this problem with podman 2.0.2 and runc 1.0.0-rc91. Exactly the behavior already described in the previous comments on this issue. I did the podman login, etc. does anyone has any hints to get this working?
@gdiazlo Does it work with Skopeo?
@rhatdan yes, i tried the inspect command and it works
Can you pull it with skopeo though?
This works:
skopeo copy docker://artifactory/image dir:/tmp/image
The real helper here would be to get an image we could play with. Does anyone have an image that exhibits the behaviour that we could test with.
Is there a public artifactory instance we can play with?
Ideally both a registry/image/credentials, and complete steps to recreate the environment. If the immediate difference is Ping https://registry:5000/v2/ status
200 vs. 401, it’s not very clear to me how that could happen.
… unless — the only difference between Podman and Skopeo in that code path that I can see right now is that Podman sets an User-Agent
header, and that somehow causes the registry to behave differently. Can you try commenting out https://github.com/containers/podman/blob/26410e4f4e2a93f2e9ecc2cda7ef59d888285b39/libpod/image/docker_registry_options.go#L67 , please?
@mtrmac that did it. I commented out that line and the fmt and podmanVersion imports and it started working. Thanks :)
I tested the 2.0.2 release source code package.
@gdiazlo Thanks!
Can you paste the full output of
curl -iL -H 'Docker-Distribution-API-Version: registry/2.0' -H 'User-Agent: libpod/2.0.2' 'https://registry.fedoraproject.org/v2/'
replacing registry.fedoraproject.org
with the right host name, please? Feel free to sanitize host names / identifiers, as long as it is clear what was edited and the structure is preserved.
Hello
Here it is. There is an oauth dance, and then a web page source. I think the editions are evident. But let me know if there is anything I should include.
HTTP/1.1 302 Moved Temporarily
Content-Type: text/html
Date: Wed, 22 Jul 2020 09:53:37 GMT
Location: https://my-internal-registry
Server: nginx
Strict-Transport-Security: max-age=15768000
Content-Length: 138
Connection: keep-alive
HTTP/1.1 302 Found
Content-Type: text/html; charset=utf-8
Date: Wed, 22 Jul 2020 09:53:37 GMT
Location: /oauth2/start?rd=%2F
Server: nginx
Strict-Transport-Security: max-age=15768000
Content-Length: 43
Connection: keep-alive
HTTP/1.1 302 Found
Content-Type: text/html; charset=utf-8
Date: Wed, 22 Jul 2020 09:53:37 GMT
Location: https://oauth-provider-url.....
Server: nginx
Strict-Transport-Security: max-age=15768000
Content-Length: 332
Connection: keep-alive
HTTP/2 302
content-type: application/binary
cache-control: no-cache, no-store, max-age=0, must-revalidate
pragma: no-cache
expires: Mon, 01 Jan 1990 00:00:00 GMT
date: Wed, 22 Jul 2020 09:53:37 GMT
location: https://oauth-provider-url.....
server: ESF
content-length: 0
x-xss-protection: 0
x-frame-options: SAMEORIGIN
alt-svc: h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
HTTP/2 302
content-type: text/html; charset=UTF-8
x-frame-options: DENY
cache-control: no-cache, no-store, max-age=0, must-revalidate
pragma: no-cache
expires: Mon, 01 Jan 1990 00:00:00 GMT
date: Wed, 22 Jul 2020 09:53:37 GMT
location:https://oauth-provider-url.....
content-security-policy: script-src 'report-sample' 'nonce-ESnpZsdbkFgRTIRcXeQZRA' 'unsafe-inline' 'unsafe-eval';object-src 'none';base-uri 'self';report-uri /cspreport
content-length: 839
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
server: GSE
set-cookie: auth-cookie
set-cookie: auth-cookie
HTTP/2 302
content-type: text/html; charset=UTF-8
x-frame-options: DENY
cache-control: no-cache, no-store, max-age=0, must-revalidate
pragma: no-cache
expires: Mon, 01 Jan 1990 00:00:00 GMT
date: Wed, 22 Jul 2020 09:53:37 GMT
location: https://oauth-provider-url.....
content-security-policy: script-src 'report-sample' 'nonce-NK13VV9EGEovZzDn8tRCFw' 'unsafe-inline' 'unsafe-eval';object-src 'none';base-uri 'self';report-uri /cspreport
content-length: 858
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
server: GSE
set-cookie: auth-cookie
set-cookie: auth-cookie
HTTP/2 200
content-type: text/html; charset=UTF-8
x-frame-options: DENY
x-auto-login: realm=auth-realm&args=continue%3Dhttps%253A%252F%252Foauth-provider-url.....
cache-control: no-cache, no-store, max-age=0, must-revalidate
pragma: no-cache
expires: Mon, 01 Jan 1990 00:00:00 GMT
date: Wed, 22 Jul 2020 09:53:37 GMT
strict-transport-security: max-age=31536000; includeSubDomains
content-security-policy: script-src 'report-sample' 'nonce-on6KZUPUV2FPmZIArPEsqQ' 'unsafe-inline' 'unsafe-eval';object-src 'none';base-uri 'self';report-uri /cspreport
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
server: GSE
set-cookie: auth-cookie
set-cookie: auth-cookie
<!DOCTYPE html>
<html lang="es">
<head>
<meta charset="utf-8">
<meta content="width=300, initial-scale=1" name="viewport">
......
</body>
</html>
HTTP/1.1 302 Moved Temporarily Content-Type: text/html Date: Wed, 22 Jul 2020 09:53:37 GMT Location: https://my-internal-registry
So I guess the registry just decides that the client is not a registry client at all, and assumes it is a browser with long-term cookies and so on?
From a quick check of the 6.20.1 source code, I can’t find anything that would be triggered by the Podman User-Agent value.
@rhatdan do we have someone to contact about this?
@gdiazlo it might be useful to contact the vendor directly about this, if you are a customer.
I don't know of anyone, but I sent out a note on Techlist to see if anyone knows of anyone?
I also sent out on the podman.io mailing list.
@rhatdan Sorry for stating obvious but they are on github. Seven contacts are listed on their front page.
Thanks, I don't have access to jfrog support, the support on my side told me "use docker, it works", and it worked yes, but I do prefer podman over docker. I am not sure also if the oauth dance might make the server behave the way it does, or may be a transparent proxy :-?
It might be a proxy… maybe a product, because we have another similar report. But that’s up to whoever deployed that proxy.
For now, I can use the work-around you suggested, I am not sure what are the implications of making such a change permanent, but if there is no impact I would propose to make it, or at least make it optional, taking into account there are others like skopeo who actually do not have this problem. Again, thanks a lot for taking your time to look into this.
If I remember well, jfrog was using a jira tracker for all their products and services.
Could someone open an issue with jfrog. We are attempting to reach out to their engineering team to see what is going on.
I would do it but I so not know the details. https://www.jfrog.com/jira/projects/
Since this is an issue with Artifactory and issues have been opened over there I am going to close. Users with this problem, please keep nagging Jfrog to properly support OCI Images.
In https://github.com/containers/podman.io/issues/135, @rodris-gimba reported:
"When i try to pull any image from artifactory, I received the error:
The same image pulled with Docker doesn't return this error and the result is succesful.
Does anyone have the same problem or a solution for this? Thanks."
I'm closing the podman.io issue.