Open patoarvizu opened 2 years ago
Some additional information: I had a hunch and tried using a custom image (instead of the upstream grafana/tanka
) that copied /usr/local/bin/argocd
from quay.io/argoproj/argocd:v2.3.1
. When I did this, I got the following error message:
... level=fatal msg="rpc error: code = NotFound desc = unknown nonce" error: unable to read askpass response from 'argocd' fatal: could not read Username...
Which sounds like there's some misconfiguration somewhere, I just don't know where.
One thing I do want to point out though is that if this is the way that sidecars have to be configured to be able to get git credentials, then I feel like this cancels out the benefit of using sidecar plugins, which is not having to create custom images. For example, in this case I wouldn't be able to use the upstream grafana/tanka
image on a sidecar cleanly, and would have to create my own image that includes argocd
. Or am I missing something?
also having this issue with classic cmp after upgrading to v2.3 before v2.3 git clone (https) works without any special code seems to be the same cause for #7995
I'm also seeing this issue after updating to v2.3+, any apps using our tanka plugin fail with the same error message reported by @patoarvizu. Watching for updates on https://github.com/argoproj/argo-cd/pull/8389. Ty.
@crenshaw-dev Is #8389 the issue to track now for the resolution to this?
@patoarvizu that's the PR to track (just merged), https://github.com/argoproj/argo-cd/issues/7995 is the issue.
I'm going to put up a PR to cherry-pick this onto the next 2.3 release.
Awesome, thank you! 👍
@patoarvizu have you tested this for your sidecar plugin? After reviewing the code with @leoluz I'm not convinced this will work for sidecars.
@crenshaw-dev No, I haven't tested it. I was waiting for 2.3.4 to be formally released to give it a try.
Okay cool. Will test. We might want to make the longer-lived credentials opt-in, because it's a bit less secure, and many folks won't need git credentials during manifest generation. Will keep this thread up to date.
Reopening because I think this needs some more thought/work on our side.
I wanted to clarify @crenshaw-dev comment above a bit more. There is recent security related work done in CMP sidecar that detaches its file system from the one used by repo-server. This change is currently in master
and will be available in the next release. That being said, if there is a specific file required by a plugin, it needs to be available in the Application's source repo. Credentials will be passed as environment variables available in the process that executes the plugin's command. We have a e2e test to validate private repo access in CMP sidecar and it passes. Maybe jb update
isn't able to get the necessary credentials currently available. This needs further investigation.
@patoarvizu can you please provide the full Application manifest that you are applying to reproduce the issue? Is this easily reproducible locally?
I can create a reproduction repo for you, including the full Application
manifest, but it may need to be forked/edited to make it private to validate the credentials work as expected. I should be able to create it for you in the next day or two.
@leoluz @crenshaw-dev As promised, here's the reproduction repo: https://github.com/patoarvizu/argo-sidecar-plugin-creds-repro
It may need some tweaks to get it to work since it depends on access to private repos, but I think you should be able to take it from here. I added instructions for how to reproduce locally using a k3d cluster if it makes it easier, but I believe the Helm setup + the sample Application
manifest there should be enough for you.
Please let me know if there's anything else I can help you with!
Wow this is brilliant. Thanks, @patoarvizu! Will give it a try Monday.
Also a tanka user and just got bit by this, since argo overwrites GIT_ASKPASS in here. I worked around it by manually exporting GIT_ASKPASS to my mounted helper within the init command. E.g:
export GIT_ASKPASS=/usr/local/bin/git-ask-pass.sh &&
jb install
Confirming I ran into this issue too, thanks to @patoarvizu I knew what version to downgrade to. I'm just now trying out ArgoCD for the first time and wanted to use kustomize+envsubst so I added a simple custom tool via configmaps etc.
On the latest versions, it cannot authenticate with Git to pull from the remote Kustomize repo. On 2.2.5, it works fine.
For what it's worth, I was able to upgrade to 2.2.8 without issues.
I've used Git's AskPass to inject credentials in a similar case.
Git askpass script in a configmap:
https://github.com/HariSekhon/Kubernetes-configs/blob/master/git-askpass.configmap.yaml
patch the ArgoCD repo server with this script and environment variables to use the above script and whatever standard k8s secret credentials you want:
@HariSekhon can you please confirm the ArgoCD version you are running? Thank you.
@leoluz I did this on an ArgoCD 2.0 that I use in production to solve my Kustomize external bases not working.
I've used Git's AskPass to inject credentials in a similar case.
Git askpass script in a configmap:
https://github.com/HariSekhon/Kubernetes-configs/blob/master/git-askpass.configmap.yaml
patch the ArgoCD repo server with this script and environment variables to use the above script and whatever standard k8s secret credentials you want:
I'm doing something similar for the sidecar plugins. While it works, this methods only allows using a single credentials rather than reusing the pattern matching of the Credential Templates from argoCD (though I'm not sure if the older version works like that, I haven't tested it yet).
I'm doing something similar for the sidecar plugins. While it works, this methods only allows using a single credentials rather than reusing the pattern matching of the Credential Templates from argoCD (though I'm not sure if the older version works like that, I haven't tested it yet).
In the above simple example this is true, but actually the git askpass mechanism can pass in variables, which means you can add logic to the script to return different credentials based on different protocol, host and repo path:
Reading through this issue, I think this bug affects only plugins that use kind of git
integration themselves, right? E.g. those which need access to Git after Argo CD has provided the set of manifests it rendered to the plugin.
@jannfis that is my understanding as well. I do not believe that plugins which do not use git
(either directly or indirectly) are impacted.
Confirming I ran into this issue too, thanks to @patoarvizu I knew what version to downgrade to. I'm just now trying out ArgoCD for the first time and wanted to use kustomize+envsubst so I added a simple custom tool via configmaps etc.
On the latest versions, it cannot authenticate with Git to pull from the remote Kustomize repo. On 2.2.5, it works fine.
same with me. Works on 2.2.5 fine. @crenshaw-dev any chance we can get a fix in for the latest version?
This is an issue still and is blocking me from upgrading past 2.2.x version.
In 2.2 git-ask-pass.sh were trivial as the GIT_USERNAME and GIT_PASSWORD were exposed via the environment.
In 2.5 that isn't the case.
This is blocking our organization from being on the latest version and inturn not able to use new features (Argocd notifications)
This is definitely still top of my mind. I'll do what I can to get something ready before 2.6. The solution has to prioritize security and Argo CD admin control over credentials. I want to take time to get that right.
@crenshaw-dev, do you know if there's any progress on the above issue?
I've used Git's AskPass to inject credentials in a similar case. Git askpass script in a configmap: https://github.com/HariSekhon/Kubernetes-configs/blob/master/git-askpass.configmap.yaml patch the ArgoCD repo server with this script and environment variables to use the above script and whatever standard k8s secret credentials you want: https://github.com/HariSekhon/Kubernetes-configs/blob/master/argocd-git-askpass.repo-server.jsonpatch.yaml
I'm doing something similar for the sidecar plugins. While it works, this methods only allows using a single credentials rather than reusing the pattern matching of the Credential Templates from argoCD (though I'm not sure if the older version works like that, I haven't tested it yet).
Hello @zephyros-dev,
Could you please share what you've done for sidecar please? I'm facing the same issue and just can't get it working. Having it work for one single credentials would be enough for me waiting for the resolution (which does not seem to be coming pretty soon) Or if someone else from here has a workaround!
Thanks!
Note I've just had a repo re-org as it was getting too big and the ArgoCD patch has been moved to:
@patoarvizu argocd-git-ask-pass
feature is reverted via https://github.com/argoproj/applicationset/pull/533
@lukasmrtvy I think that was known from earlier in the thread. Is this just confirmation?
@pasha-codefresh @crenshaw-dev the PR https://github.com/argoproj/argo-cd/pull/15107, raised by @jmcshane is probably the closest thing we have to a resolution for this.
I think it's critical that ArgoCD plugins can access the ArgoCD credentials in some safe way, so I would appreciate your comments on this PRs approach.
Also, there is a related issue https://github.com/argoproj/argo-cd/issues/16091
@leoluz @crenshaw-dev any upated ETA on when this feature will be released? Looks like there are no comments to address on the PR #15107
For anyone who just wants to use a single set of creds for all their git requests, and wants an env var only solution, this way is working for me. These envs are added to my sidecar containers. In my example, the argo-cd access token is stored in a secret called github-secret. Credit to https://stackoverflow.com/a/78064753 for the idea.
- name: GITHUB_TOKEN
valueFrom:
secretKeyRef:
name: github-secret
key: password
- name: GIT_CONFIG_COUNT
value: "2"
- name: GIT_CONFIG_KEY_0
value: credential.https://github.com.username
- name: GIT_CONFIG_VALUE_0
value: x-access-token
- name: GIT_CONFIG_KEY_1
value: credential.https://github.com.helper
- name: GIT_CONFIG_VALUE_1
value: '!f() { test "$1" = get && echo "password=$GITHUB_TOKEN"; }; f'
Checklist:
argocd version
.Describe the bug
I have a sidecar plugin for Tanka, that uses jsonnet-bundler (
jb
) to pull dependencies from private repos. I have theConfigManagementPlugin
manifest configured to haveinit
runjb install
, which is how jsonnet-bundler pulls its dependencies, and it calls git under the covers. I assume that because these calls to git are unknown to/unmanaged by Argo, the pre-configured credentials aren't available, but up until 2.2.5, I was able to have a workaround this issue by mounting agit-ask-pass.sh
script via aConfigMap
, and settingGIT_USERNAME
andGIT_PASSWORD
as environment variables on the sidecar container. Once I tried to upgrade to 2.3.1, this stopped working. (Note: Argo was still able to clone the private repo that myApplication
was pointing to, but what stopped working was thejb update
command that runs in the sidecar as part of the initialization.)I see here that
argocd-git-ask-pass
was introduced, but I don't quite understand if this is something that should be available in custom sidecars, or how to use it. It would be good if I can understand why the upgrade to 2.3.1 broke my workaround, but I'd be satisfied with a path forward if that's all we can find.To Reproduce
I'm installing via the Helm chart, here are the relevant values:
The corresponding
ConfigMap
s are as follows:Expected behavior
GIT_ASKPASS
functionality.argocd-git-ask-pass
is available for custom sidecar plugins, I'd like to understand how to use it.Screenshots
Version
I'm not running the
argocd
command line, but I'm using Argo 2.3.1Logs