Closed Gaibhne closed 8 months ago
Gaibhne
commented
1 year ago I have narrowed down the point of the regression to the move from 4.3 to 4.4. With 4.3.1 it was still possible to pull images with a podman login via docker-compose, with 4.4 it stops being possible, and I tried 4.5, still broken there.
rhatdan
commented
1 year ago @vrothberg @mtrmac PTAL
vrothberg
commented
1 year ago I am running out of time to look into this issue but we need to track it down.
vrothberg
commented
1 year ago I cannot reproduce with either docker.io or quay.io.
@Gaibhne, are you running docker-compose as root?
vrothberg
commented
1 year ago Are you able to pull the images or is the problem that you are hitting the rate limit after some time?
dzirg44
commented
1 year ago In my case, I can build a docker image using private registries after log-in into the private ECR repo
aws ecr get-login-password --region us-west-2 | podman login --username AWS(podman build), but I can't use them in my docker-compose configuration, because I am getting 401 Unauthorized.
vrothberg
commented
1 year ago @dzirg44, does it work with docker-compose when you pass the --authfile flag?
podman login --authfile ~/.docker/config.json
dzirg44
commented
1 year ago @vrothberg It will not work because this file should be created(copied, generated, linked) in advance, podman doesn't create a symlink for this file (tested on macos and linux fedora). Temporary solution is disabling buildkit
export DOCKER_BUILDKIT=0we have a bunch of issues related to incompatibilities with buildkit
also thanks to @vrothberg another solution can be
ln -s ~/.config/containers/auth.json ~/.docker/config.jsonI hope it will help somebody
Gaibhne
commented
1 year ago Are you able to pull the images or is the problem that you are hitting the rate limit after some time?
Yes, that is what is happening. We have a company account, into which we log in with podman login docker.io, and which is then used for podman pull but not via docker-compose pull.
The suggestions by @vrothberg and @dzirg44 did not work for me; as I wrote in the initial post, I already tried that. They worked fine in 4.4.
dstaley
commented
1 year ago I am also experiencing the same issue.
host:
arch: amd64
buildahVersion: 1.30.0
cgroupControllers:
- cpuset
- cpu
- io
- memory
- hugetlb
- pids
- rdma
- misc
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: conmon-2.1.7-2.1.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.1.7, commit: unknown'
cpuUtilization:
idlePercent: 99.9
systemPercent: 0.04
userPercent: 0.06
cpus: 32
databaseBackend: boltdb
distribution:
distribution: '"opensuse-microos"'
version: "20230706"
eventLogger: journald
hostname: euporie
idMappings:
gidmap: null
uidmap: null
kernel: 6.3.9-1-default
linkmode: dynamic
logDriver: journald
memFree: 340746240
memTotal: 16348246016
networkBackend: cni
ociRuntime:
name: runc
package: runc-1.1.7-1.3.x86_64
path: /usr/bin/runc
version: |-
runc version 1.1.7
commit: v1.1.7-0-g860f061b76bb
spec: 1.0.2-dev
go: go1.20.5
libseccomp: 2.5.4
os: linux
remoteSocket:
exists: true
path: /run/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: false
seccompEnabled: true
seccompProfilePath: /etc/containers/seccomp.json
selinuxEnabled: true
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.2.0-1.3.x86_64
version: |-
slirp4netns version 1.2.0
commit: unknown
libslirp: 4.7.0
SLIRP_CONFIG_VERSION_MAX: 5
libseccomp: 2.5.4
swapFree: 0
swapTotal: 0
uptime: 15h 26m 25.00s (Approximately 0.62 days)
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries:
search:
- registry.opensuse.org
- registry.suse.com
- docker.io
store:
configFile: /etc/containers/storage.conf
containerStore:
number: 1
paused: 0
running: 1
stopped: 0
graphDriverName: btrfs
graphOptions: {}
graphRoot: /var/lib/containers/storage
graphRootAllocated: 234047737856
graphRootUsed: 61385965568
graphStatus:
Build Version: Btrfs v6.3
Library Version: "102"
imageCopyTmpDir: /var/tmp
imageStore:
number: 11
runRoot: /var/run/containers/storage
transientStore: false
volumePath: /var/lib/containers/storage/volumes
version:
APIVersion: 4.5.1
Built: 1685318400
BuiltTime: Mon May 29 00:00:00 2023
GitCommit: ""
GoVersion: go1.20.5
Os: linux
OsArch: linux/amd64
Version: 4.5.1I stored credentials in ~/.docker/config.json as well, but still experience the same issue.
# docker-compose --version
Docker Compose version v2.19.1Thanks everybody for providing all the data. I am taking a look at the issue and hope to have a fix for the upcoming Podman v4.6 release.
vrothberg
commented
1 year ago Bisect revealed commit 3718ac8e969b for having introduced the regression.
vrothberg
commented
1 year ago OK, I think this - so far - has just worked by pure chance and luck.
https://github.com/containers/image/commit/b3748aab5c90 changed the logic of looking up credentials to stop checking any other paths than the specified auth file. This is the "breaking" change.
The actual problem is that podman login --authfile ~/docker/config.json doesn't produce something useful for docker or docker-compose: "WARNING: Error loading config file: /home/vrothberg/.docker/config.json/config.json". Prior it worked by pure luck since the system service happened to lookup ~/docker/config.json but docker-compose never set the auth header as it cannot read the JSON created by Podman.
Docker and docker-compose don't seem to be able to use Podman's config.json correctly. For instance, they expect the registry to be "https://index.docker.io/v1/" and not docker.io. There's probably more to it.
@mtrmac @baude - I think we need to reflect whether podman login should be able to generate docker-compatible auth json. The remote UX would otherwise be impacted.
rhatdan
commented
1 year ago is docker.io a special case?
$ docker login quay.io/rhatdan
Username: rhatdan
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
sh-5.2# cat /root/.docker/config.json
{
"auths": {
"quay.io": {
"auth": "foobar="
}
}#podman login quay.io/rhatdan
Authenticating with existing credentials for quay.io/rhatdan
Existing credentials are valid. Already logged in to quay.io
sh-5.2# rm /root/.docker/config.json
sh-5.2# podman login quay.io/rhatdan
Username: rhatdan
Password:
Login Succeeded!
# cat /run/containers/0/auth.json
{
"auths": {
"quay.io/rhatdan": {
"auth": "cmhhdGRhbjpmRUQ5Z01CNHV1YmZ1UjY="
}
}
mtrmac
commented
1 year ago The actual problem is that
podman login --authfile ~/docker/config.jsondoesn't produce something useful fordockerordocker-compose:
Yeah, --authfile …/.docker/config.json is just not expected at all. The code intends to write to an auth.json file (an extension of the config.json format); .docker/config.json is only considered as a read-only fallback.
pkg/docker/config currently doesn’t have any concept of “writing in the Docker-compatible-format”; the implemented compatibility is one-way, read-only.
Prior it worked by pure luck since the
system servicehappened to lookup~/docker/config.jsonbutdocker-composenever set the auth header as it cannot read the JSON created by Podman.
(If the whole system is intentionally built around 1. podman login, and 2. a podman-based backend, an option might be to possibly lean into this accidentally-occurring design. Notably having the backend read the credentials directly would allow using mirror credentials, which a Docker .docker/config.json consumer is never going to send over the API.
OTOH that would make the Podman users of compose rather different from Docker users, and that would make things harder to understand / debug.)
Docker and docker-compose don't seem to be able to use Podman's config.json correctly. For instance, they expect the registry to be
"https://index.docker.io/v1/"and notdocker.io. There's probably more to it.
quay.io/vendor/ns) are one major feature difference.auth.json is transient, so the default helper is set in registries.conf; whereas for Docker it is stored in config.json). If c/image were to write to config.json, would it completely ignore registries.conf and instead look for a helper in the config.json field?@mtrmac @baude - I think we need to reflect whether
podman loginshould be able to generate docker-compatible auth json.
What would podman login quay.io/vendor do? Failing is not good. So I think such a behavior can’t be the default, we can’t update both files automatically. (Also, error handling if one write succeeds and the other one fails would be a pain.)
podman login --docker-config-file ~/.docker/config.json would make sense to me.
Alternatively, if we are wrapping compose in a Podman-specific glue anyway (are we?), generating a $TMPDIR/podman-compose-credentials.json for that one command, based on `config.GetAllCredentials, might make sense… Either way, we would need to implement a writer for that format.
vrothberg
commented
1 year ago podman login --docker-config-file ~/.docker/config.json would make sense to me.
I think we're going in the same direction. I had a login --format in mind that can be configured in containers.conf. This way, Podman Desktop can drop a .conf file forcing the legacy format.
Either way, we would need to implement a writer for that format
Agreed. I'd like to make this a priority in the next quarter as I think this issue will hit more and more users.
mtrmac
commented
1 year ago I had a
login --formatin mind that can be configured incontainers.conf.
I’m not very enthusiastic about that.
podman login quay.io/vendor” option. Suppose one software package requires it to be on, and another one to be off — the two will now not be possible to install simultaneously.--authfile …/.docker/config.json, and that’s not a containers.conf option AFAICS. So I think users of podman login --authfile …/config.json should be using a new option to use the Docker format, in some script. (Alternatively, we could make --authfile also a containers.conf option, but the first point would apply … even more.)vrothberg
commented
1 year ago Alternatively, we could make --authfile also a containers.conf option, but the first point would apply … even more.
I like that idea :+1:
mtrmac
commented
1 year ago I… don’t :) but I could live with it. The c/image code would be exactly the same either way.
rhatdan
commented
1 year ago I think handle it in script/option first before we move this to containers.conf. I am still wondering if this is only an issue for docker.io images.
mtrmac
commented
1 year ago IIRC from a quick check, Docker:
docker login
docker login (or I missed it… now that I think about it, the ability to successfully login implies some kind of a valid format)docker.io with a specific https://… url on docker logindocker.io, requires that specific https://… url to match exactlySo it might be the case that only docker.io matters.
If we are serious about interoperability, we should have unit tests vs. the Docker implementation… we don’t have them today for the read path.
vrothberg
commented
1 year ago If we are serious about interoperability, we should have unit tests vs. the Docker implementation… we don’t have them today for the read path.
Agreed.
vrothberg
commented
1 year ago @mtrmac, for compose to work smoothly we need also some new API to query for the authfile path that podman login would use in the specific environment.
mtrmac
commented
12 months ago I mean, podman login would always use an auth.json by default.
If it were to use a Docker format, that would only happen with a podman login --some-option; so I’m not quite sure what you mean here precisely. Is the idea that callers would set “use the Docker format and files” boolean, and the implementation would choose the right path automatically? That could work.
Alternatively, IIRC Compose is intended to work though a Podman wrapper anyway. So, would it work for that wrapper to define its own (single-use?) file path, and explicitly set it in SystemContext, along with the option to use a Docker-compatible format?
I think an important factor in designing this is, for users that use credential helpers, where that configuration is set. If it is in one of the Docker files (because auth.json is not persistent) — i.e. not in registries.conf — then using a single-use empty config.json could be inconvenient.
That’s … also an important point. For the purpose of “compose working smoothly”, does it matter if credentials are stored in credential helpers? Does that affect anything about the flow?
A c/image/pkg/docker/config.GetDockerConfigPath[UsingTheNewFormatNotDockercfg] is an option. That gets a tiny bit complicated in that we can, currently, read from two possible locations; I’m not immediately sure whether that matches Docker’s behavior, or which one is the right one to write to. That’s a question that can trivially be answered with a bit of research — just research that I think needs to happen before committing to an API.
Either way, it is an API that can be added, sure.
vrothberg
commented
11 months ago If it were to use a Docker format, that would only happen with a
podman login --some-option; so I’m not quite sure what you mean here precisely. Is the idea that callers would set “use the Docker format and files” boolean, and the implementation would choose the right path automatically? That could work.
What I think we need is a way to instruct podman login to create something Docker clients can read. That is pretty much the scenario we're running into here. docker-compose doesn't fully understand Podman's auth.json.
Alternatively, IIRC Compose is intended to work though a Podman wrapper anyway. So, would it work for that wrapper to define its own (single-use?) file path, and explicitly set it in
SystemContext, along with the option to use a Docker-compatible format?
Yes, I think that's we need for podman login.
I think an important factor in designing this is, for users that use credential helpers, where that configuration is set. If it is in one of the Docker files (because
auth.jsonis not persistent) — i.e. not inregistries.conf— then using a single-use emptyconfig.jsoncould be inconvenient.That’s … also an important point. For the purpose of “compose working smoothly”, does it matter if credentials are stored in credential helpers? Does that affect anything about the flow?
At the moment, I don't think it matters. Maybe I am missing the point but I expect Docker clients to handle credential helpers.
A
c/image/pkg/docker/config.GetDockerConfigPath[UsingTheNewFormatNotDockercfg]is an option. That gets a tiny bit complicated in that we can, currently, read from two possible locations; I’m not immediately sure whether that matches Docker’s behavior, or which one is the right one to write to. That’s a question that can trivially be answered with a bit of research — just research that I think needs to happen before committing to an API.
Docker reads from one path only AFAIK.
What I currently desire is a "mode" for Podman remote to always generate auth files Docker clients can read. I think that's a compatibility scenario we need to support.
Gaibhne
commented
9 months ago @vrothberg is there a way to kick the issue back up to triage or whatever is the right 'state' ? It seems after you removed yourself, it didn't go back on anyones radar. Or is that not how it works in this repo ?
mtrmac
commented
8 months ago
podman login --authfile ~/.docker/config.json docker.io
With #20621 (as it is now, subject to change), this will generate a warning, pointing to a new podman login --compat-auth-file ~/.docker/config.json … option.
Issue Description
All of the following relates to rootless podman. I have no setup to test with rootful podman
Up until at least
podman-4.2.0-11.el9_1.x86_64we were able to useDocker Compose version v2.17.3to pull images from an authenticated registry such asdocker.io(viapodman login --authfile ~/.docker/config.json docker.io). Starting with at leastpodman-4.4.1-9.el9_2.x86_64, this is no longer possible. Neither with the authentication stored in~/.docker/config.jsonnor with it stored in~/.config/containers/auth.json(nor with it stored in both, via multiple calls topodman login) will the credentials be accessed/applied when images are pulled viadocker-compose. Identical auth files anddocker-composeversions were used for testing to narrow the problem down to a change inpodmanbetween the two versions listed above.podman pullitself correctly uses the credentials.Steps to reproduce the issue
Steps to reproduce the issue
podman login --authfile ~/.docker/config.json docker.iodocker-compose pull <some service with an image definition like image: docker.io/library/redis:6.2.6>Describe the results you received
Cry because you've hit the rate limits and your account credentials are not used for pulling
Describe the results you expected
The joy of successfully pulling the image with my provided credentials, as was the case with podman 4.2.
podman info output
Podman in a container
No
Privileged Or Rootless
Rootless
Upstream Latest Release
No
Additional environment details
Behaviour tried in different virtual machines (kvm, virtualbox), always reproducible
Additional information
I am aware of https://github.com/containers/podman/issues/15620 and the solution there does not apply, I have confirmed that the credential files are created correctly