rootless podman supports credHelper

[QiWang19]

c/image and podman already support "credHelpers", but only works for config in /run/user/0/auth.json and sudo podman login

with config in /run/user/1000/auth.json and podman login got error

{
        "credHelpers": {
                "docker.io": "secretservice"
        }
}

$ podman login docker.io
Error: error reading auth file: error getting credentials - err: exit status 1, out: `Exhausted all available authentication mechanisms (tried: EXTERNAL) (available: EXTERNAL)`

[sdouche]

Hi, The lack of non-root use is the blocking point to switch to Podman from Docker on some use-cases. Highly interested in this feature. Do you know a workaround to bypass this restriction?

[mtrmac]

I can’t see why this should not work in principle, c/image does nothing interesting about privileges/credentials.

That bug is almost certainly reported by the secretservice helper itself, and needs to be diagnosed within it. (As a wild guess, it needs some local secrets that are available to root but not UID=1000?)

[github-actions[bot]]

This issue had no activity for 30 days. In the absence of activity or the "do-not-close" label, the issue will be automatically closed within 7 days.

[TomSweeneyRedHat]

Going to keep this one alive.

[sdouche]

Hi:). Do you think this issue will be included in the short-term roadmap? To know if should switch back on Docker.

[vrothberg]

@QiWang19, do you have cycles to look into it?

[rhatdan]

@sdouche do you have a specific credhelper you are looking for?

[QiWang19]

@QiWang19, do you have cycles to look into it?

yes, I want to look into this

[sdouche]

@sdouche do you have a specific credhelper you are looking for?

Hi @rhatdan. For now, to retrieve containers from AWS ECR:

{
  "credHelpers": {
    "xxx.dkr.ecr.eu-west-1.amazonaws.com": "ecr-login"
  }
}

Does that answer your question?

[rhatdan]

Yes, that is exactly what I was looking for.

[sdouche]

I forgot, feel free to ping me for testing :). Thanks for your work.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@QiWang19 Can you look into this?

[rhatdan]

@QiWang19 Working on this one?

[rhatdan]

@QiWang19 What is the status on this one?

[QiWang19]

@giuseppe verified this is an issue with secret-tools using d-bus hardcodes the euid() inside the EXTERNAL request itself. I think he has a patch for fixing this.

[giuseppe]

Merge Request here: https://gitlab.gnome.org/GNOME/glib/-/merge_requests/1403

[QiWang19]

Thanks. Should this issue be closed now or should wait for the fix to be included in some release? For now the rootless podman with credential helper configuration still can't work.

Merge Request here: https://gitlab.gnome.org/GNOME/glib/-/merge_requests/1403

[rhatdan]

Keep it open. If this gets merged we are going to have to ask for a back port to RHEL8 as well.

[giuseppe]

I got some comments on the merge request but I am not able at the moment to take care of them.

@alexlarsson do you know who could help us to move the gnome MR forward?

[alexlarsson]

@giuseppe Seems like simon is looking at it? He is the upstream dbus maintainer, and also often involved in gnome stuff.

[rhatdan]

@giuseppe @alexlarsson @QiWang19 Any update on this? Has dbus been fixed?

[alexlarsson]

I'll ping some people about it

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@alexlarsson Did anyone respond?

[svdHero]

What's the status on this? In my company we are using docker with password manager pass and docker credentials-helper docker-credential-pass. We would love to migrate to Podman, if this feature - including the corresponding documentation - was available.

[mtrmac]

@svdHero This bug report is specifically about the secretservice helper not working in Podman. If docker-credential-pass fails, and the reason is not obviously the same, please file a separate report.

[svdHero]

@mtrmac I came here from #7254 which describes my problem, but was closed as a duplicate of #4119. In #4119 @QiWang19 mentioned this issue here. So, I assumed this would be the place where he was documenting his progress on credential-helpers. Maybe this was a wrong assumption.

[QiWang19]

@svdHero The secretservice helper can't work rootless caused by secret-tools with dbus issue. What error you came across with docker-credential-pass? Does it work with rootful podman?

[svdHero]

There are no error messages. #7254 states that credential-helpers are not supported by podman. So, to be more precise, my question would be: what is the status of credential-helpers in podman and how can I use them? Do I just install the docker-credential-pass and use it with podman or is there a separate podman-credential-pass?

Concerning rootless, I have always used the podman command with my normal user account and without sudo. That's what I considered "rootless". Is that what you mean or is there more to rootlesness that I don't understand?

Sorry if I ended up in the wrong issue with my noob questions. Maybe adding some documentation on podman and credential-helpers might avoid this in the future?

[QiWang19]

@svdHero Here's the documentation https://github.com/containers/image/blob/master/docs/containers-auth.json.5.md#format. you can just install the docker-credential-pass and use it with podman. The credential helper "registry.example.com": "pass" can be configured under "credHelpers": field.

Concerning rootless, I have always used the podman command with my normal user account and without sudo. That's what I considered "rootless". Is that what you mean or is there more to rootlesness that I don't understand?

Yes. Use podman without sudo, that's what I mean rootless.

[mtrmac]

To be a bit more explicit, the concept of credential helpers is supported; but only the credHelpers config field is implemented, not yet the credsStore field.

[svdHero]

@mtrmac Now I am totally confused. How would I use credential helpers without specifying it in the credsStore field? Coming from https://docs.docker.com/engine/reference/commandline/login/#credentials-store and https://github.com/docker/docker-credential-helpers#usage this does not make sense at all. At least not to a container beginner like myself.

Could you elaborate a bit on this, please? What would be the recommended way to use credential helpers then? The link provided by @QiWang19 didn't really help, because the article just refers to the docker-credential-helpers github page which again uses the credsStorefield.

[mtrmac]

@svdHero Search for credHelpers on the first link.

[tjanez]

@QiWang19,

I've tried setting the pass credentials helper for the docker.io registry:

$ cat ~/.config/containers/auth.json 
{
    "credHelpers": {
        "docker.io": "pass"
    }
}

But it doesn't work:

$ podman search --list-tags docker.io/PRIVATE-ORG/PRIVATE-REPO
ERRO[0001] error listing registry tags "docker.io": error getting repository tags: unable to retrieve auth token: invalid username/password: errors:
denied: requested access to the resource is denied
unauthorized: authentication required

Am I missing something?

I'm using podman-2.2.1-1.fc33.x86_64 from Fedora 33.

I have docker-credentials-pass installed and it has 1 entry:

$ docker-credential-pass list
{"https://index.docker.io/v1/":"tjanez"}

[QiWang19]

@tjanez docker-credential-pass never works for me. Although I can get/list to see my credentials, it failed to use the credentials from there.

$ docker-credential-pass get
docker.io
{"ServerURL":"docker.io","Username":"qiwanredhat","Secret":"..."}
$ podman search --list-tags docker.io/qiwanredht/<repo>
ERRO[0000] error listing registry tags "docker.io": error getting repository tags: failed to create client: error getting username and password: error getting credentials - err: exit status 1, out: `exit status 2: gpg: decryption failed: No secret key`

[tjanez]

@tjanez docker-credential-pass never works for me. Although I can get/list to see my credentials, it failed to use the credentials from there.

Huh, I was under the impression that this was working...

$ docker-credential-pass get
docker.io
{"ServerURL":"docker.io","Username":"qiwanredhat","Secret":"..."}
$ podman search --list-tags docker.io/qiwanredht/<repo>
ERRO[0000] error listing registry tags "docker.io": error getting repository tags: failed to create client: error getting username and password: error getting credentials - err: exit status 1, out: `exit status 2: gpg: decryption failed: No secret key`

Thanks for this example. Apparently, docker command worked with "https://index.docker.io/v1/" as the ServerURL, but podman doesn't.

I've added a new entry with "docker.io" as the ServerURL:

$ docker-credential-pass store
{"ServerURL":"docker.io","Username":"tjanez","Secret":"...removed..."}

And now I get the same error as you:

$ podman search --list-tags docker.io/PRIVATE-ORG/PRIVATE-REPO
ERRO[0000] error listing registry tags "docker.io": error getting repository tags: failed to create client: error getting username and password: error getting credentials - err: exit status 1, out: `exit status 2: gpg: decryption failed: No secret key` 

[tjanez]

@QiWang19, the thing that helped me solve the issue (suggested in https://github.com/docker/docker-credential-helpers/issues/118#issuecomment-483952635) was to set:

export GPG_TTY=$(tty)

so that GPG could ask me for my YubiKey's PIN to decrypt the credentials stored via pass.

Afterwards, listing worked as expected:

$ podman search --list-tags docker.io/PRIVATE-ORG/PRIVATE-REPO
NAME                                 TAG
docker.io/PRIVATE-ORG/PRIVATE-REPO  0.0.1
docker.io/PRIVATE-ORG/PRIVATE-REPO  0.0.10
docker.io/PRIVATE-ORG/PRIVATE-REPO  0.0.11
docker.io/PRIVATE-ORG/PRIVATE-REPO  0.0.12
docker.io/PRIVATE-ORG/PRIVATE-REPO  0.0.13
docker.io/PRIVATE-ORG/PRIVATE-REPO  0.0.14
docker.io/PRIVATE-ORG/PRIVATE-REPO  0.0.15
docker.io/PRIVATE-ORG/PRIVATE-REPO  0.0.16
docker.io/PRIVATE-ORG/PRIVATE-REPO  0.0.2
docker.io/PRIVATE-ORG/PRIVATE-REPO  0.0.3
docker.io/PRIVATE-ORG/PRIVATE-REPO  0.0.4
docker.io/PRIVATE-ORG/PRIVATE-REPO  0.0.5
docker.io/PRIVATE-ORG/PRIVATE-REPO  0.0.6
docker.io/PRIVATE-ORG/PRIVATE-REPO  0.0.7
docker.io/PRIVATE-ORG/PRIVATE-REPO  0.0.8
docker.io/PRIVATE-ORG/PRIVATE-REPO  0.0.9
docker.io/PRIVATE-ORG/PRIVATE-REPO  0.1.0
docker.io/PRIVATE-ORG/PRIVATE-REPO  0.1.1
docker.io/PRIVATE-ORG/PRIVATE-REPO  0.1.13
docker.io/PRIVATE-ORG/PRIVATE-REPO  0.1.2
docker.io/PRIVATE-ORG/PRIVATE-REPO  0.2.0
docker.io/PRIVATE-ORG/PRIVATE-REPO  0.4.0
docker.io/PRIVATE-ORG/PRIVATE-REPO  0.4.1
docker.io/PRIVATE-ORG/PRIVATE-REPO  0.4.10
docker.io/PRIVATE-ORG/PRIVATE-REPO  0.4.11

[QiWang19]

@tjanez That works. Thanks!

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@QiWang19 What should happen with this one?

[MikeKroell]

We also have an interest in this. We use ECR with HPC, and want to use rootless podman for security. Though this is a show-stopper for us.

[tjanez]

I have it working on my system but I guess there are no complete instructions for using docker-credentials-pass with Podman (yet).

Let me write them here.

Guide on using docker-credentials-pass with Podman

Install docker-credential-pass docker credential helper

Download the latest release from https://github.com/docker/docker-credential-helpers/releases.

Make sure docker-credentials-pass binary is in your PATH.

For example, symlink the binary to some directory in your path (e.g. ~/.local/bin):

ln -s <path-to-docker-credentials-helper-executable> ~/.local/bin/docker-credential-pass

Initialize docker-credentials-pass

Run:

pass init -p docker-credential-helpers <GPG-key-id>

For example:

pass init -p docker-credential-helpers myname@mydomain.com

If everything was successful, you should see output similar to:

mkdir: created directory '/home/myname/.password-store/docker-credential-helpers'
Password store initialized for myname@mydomain.com (docker-credential-helpers)
[master 1faa999] Set GPG id to myname@mydomain.com (docker-credential-helpers).
 1 file changed, 1 insertion(+)
 create mode 100644 docker-credential-helpers/.gpg-id

Add credentials for a registry

Add the following to ~/.config/containers/auth.json (create the file if necessary) (e.g. for the docker.io registry):

{
    "credHelpers": {
        "docker.io": "pass"
    }
}

NOTE: Podman currently only implements support for the credHelpers config field. The global credsStore field is not yet supported. See https://github.com/containers/podman/issues/4123#issuecomment-733677392 for more details.

Add credentials for the registry (e.g. docker.io) to pass by running:

docker-credential-pass store

Enter your registry credentials in the following format:

{"ServerURL":"docker.io","Username":"myname","Secret":"<password>"}

And press Ctrl+D after you've finished.

NOTE: If your password contains a backslash (\) character, you need to escape it with another backslash, i.e. \\.

The docker-credentials-pass helper will store your registry credentials as docker-credential-helpers/<encoded-registry-URL>/<registry-username>, e.g. docker-credential-helpers/ZG9ja2VyLmlv/myname.

_SECURITY WARNING: Adding credentials via podman login doesn't work at the moment. It will store your registry credentials unencrypted in ~/$XDG_RUNTIME_DIR/containers/auth.json file!_

List stored credentials (without passwords)

Run:

docker-credential-pass list

If should return:

{"docker.io":"myname"}

Configure GPG

To tell GPG to ask the user for his YubiKey's PIN to decrypt the credentials stored via pass, add the following to your ~/.bashrc:

export GPG_TTY=$(tty)

For more details, see: https://github.com/containers/podman/issues/4123#issuecomment-772763894.

[mtrmac]

_SECURITY WARNING: Adding credentials via podman login doesn't work at the moment. It will store your registry credentials unencrypted in ~/$XDG_RUNTIME_DIR/containers/auth.json file!_

Yeah, that’s one of the downsides of using an ephemeral (swap-only) credential file as the default: it can’t reasonably contain a credential helper configuration because it is erased on every reboot. (It shouldn’t be much of a risk, because it is supposed to be only stored in RAM and on the swap; assuming encrypted swap, both is protected using exactly the same kinds of credentials necessary to attack a credential helper process running under the same UID — but that depends on details of your threat model and mitigations, if any.)

An alternative to this is to configure the credential helper in registries.conf(.d) ’s credential-helpers option; that allows changing the global default, or per-user-default, to a helper instead of a config file, in a persistent way. (Full disclosure: I didn’t test this now to verify it works as expected, I’m only pointing out the existence of the option.)

[tjanez]

_SECURITY WARNING: Adding credentials via podman login doesn't work at the moment. It will store your registry credentials unencrypted in ~/$XDG_RUNTIME_DIR/containers/auth.json file!_

(It shouldn’t be much of a risk, because it is supposed to be only stored in RAM and on the swap; assuming encrypted swap, both is protected using exactly the same kinds of credentials necessary to attack a credential helper process running under the same UID — but that depends on details of your threat model and mitigations, if any.)

Yeah, that is true.

Hopefully, using a credential helper like pass in combination with a smartcard-backed GPG key (e.g. YubiKey), one would need to physically connect the smartcard to the system and unlock it via a PIN and the credential helper would stop working after the smartcard is disconnected, so the secrets would be "exposed" to processes running under the same UID for less time.

An alternative to this is to configure the credential helper in registries.conf(.d) ’s credential-helpers option; that allows changing the global default, or per-user-default, to a helper instead of a config file, in a persistent way. (Full disclosure: I didn’t test this now to verify it works as expected, I’m only pointing out the existence of the option.)

Thanks for mentioning this, I see this is quite a new thing: https://github.com/containers/image/pull/1193.

[rhatdan]

This screams out for a Blog to be written. Podman and Credential helpers.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@lsm5 @mtrmac @vrothberg PTAL

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.