Open meyerpeter opened 3 weeks ago
vitaliyf
commented
3 weeks ago We also see this SSH agent requested but SSH_AUTH_SOCK not-specified after 2.11.7 to 2.12.0 (and 2.12.1) upgrade. Rollback of argo-cd helm chart 7.4.4 to 7.3.11 resolves the problem.
Worth mentioning that the repository is listed under Settings->Repositories with Connection Status: ✅ Successful, but syncing is failing with the above error message in repo-server logs.
gravufo
commented
3 weeks ago We seem to have the same issue, the only difference is we target our repos in HTTPS instead of SSH.
blakepettersson
commented
3 weeks ago @gravufo, @vitaliyf and @meyerpeter can you give the specs of the applications which are failing, and the (redacted) specs of the repo credentials that are expected to work with these applications? I suspect shenanigans from #18388. I'd like to specifically see what project the application belongs to, as well as the project the repository credential belongs to.
vitaliyf
commented
3 weeks ago You might be right, since we have many apps in many projects, all using same repo (which seem to be in "default" project)
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
finalizers:
- resources-finalizer.argocd.argoproj.io
labels:
app.kubernetes.io/instance: redacted-argocd-prod
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: argo-cd
app.kubernetes.io/version: 1.0.0
argocd.argoproj.io/instance: redacted-argocd-prod
helm.sh/chart: argo-cd-1.0.0
name: our-app-prod
namespace: argocd
spec:
destination:
namespace: our-prod
server: https://kubernetes.default.svc
project: our-prod
source:
path: helm/our-app-name/
plugin:
name: redacted-template
parameters:
- name: release-name
string: our-app-name
- array:
- prod.yaml
name: values-files
repoURL: git@github.com:redacted-org/redacted-repo-name.git
targetRevision: master
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- Validate=true
- ServerSideApplyand
apiVersion: v1
data:
name: redacted-base64("redacted-repo-name")
project: redacted-base64("default")
sshPrivateKey: redacted-base64("-----BEGIN OPENSSH PRIVATE KEY----- ...")
type: redacted-base64("git")
url: redacted-base64("git@github.com:redacted-org/redacted-repo-name.git")
kind: Secret
metadata:
annotations:
managed-by: argocd.argoproj.io
labels:
argocd.argoproj.io/secret-type: repository
name: repo-1111138409
namespace: argocd
type: OpaqueSome items worth mentioning:
blakepettersson
commented
3 weeks ago @vitaliyf that confirms my hypothesis - it's due to a change in how project-scoped repo creds are loaded. Previously cluster secrets previously matched by matching the url only, but now matches on url and project (if a project exists on the cluster secret). I'd argue that the way it is in 2.12 is in fact the correct behavior (a cluster-credential scoped to a certain project should not work for another project, and may pose a security issue), but sadly this is an inadvertently breaking change :disappointed:
To fix your issue you can remove the project key from your cluster secret.
cc @jessesuen
vitaliyf
commented
3 weeks ago Confirmed - removing project: default from repo-1111138409 Secret fixed the problem.
@blakepettersson conceptually that makes sense, though in some situations (like ours, where projects are simply used to logically group apps and there is no real security scoping necessary across them) it would be nice to keep ability to have cluster-wide repo configured, so we don't need to add a bunch of identical ones for each project.
blakepettersson
commented
3 weeks ago @vitaliyf if the repo cred is only used by our-prod applications, you can also set the project to be our-prod on the repo cred.
todaywasawesome
commented
3 weeks ago After discussion in the contributor's experience meeting the conclusion is that we need to add a note to release docs for 2.12.0 noting the breaking change and the scoped repositories documentation to describe the expected behavior.
gravufo
commented
3 weeks ago @vitaliyf that confirms my hypothesis - it's due to a change in how project-scoped repo creds are loaded. Previously cluster secrets previously matched by matching the url only, but now matches on url and project (if a
projectexists on the cluster secret). I'd argue that the way it is in 2.12 is in fact the correct behavior (a cluster-credential scoped to a certain project should not work for another project, and may pose a security issue), but sadly this is an inadvertently breaking change 😞To fix your issue you can remove the
projectkey from your cluster secret.cc @jessesuen
I also confirm it fixes the issue for me too. Thanks a lot for the workaround. I agree that the new behavior makes more sense and would consider the previous one as a bug, so I think it's fine to not consider this a breaking change, but the documentation/release note update is definitely welcome!
vitaliyf
commented
3 weeks ago @vitaliyf if the repo cred is only used by
our-prodapplications, you can also set theprojectto beour-prodon the repo cred.
No, we actually do have multiple apps, across multiple projects, all using the same repo.
We also have a use-case where we dynamically create projects + apps inside of them, all pointed at the same repository (this is done by GitHub PR generator, one project per open PR), so managing permission entry for each one would be slightly cumbersome.
Could repositories allow project: * scoping that would mean "all projects"?
blakepettersson
commented
3 weeks ago Could repositories allow project: * scoping that would mean "all projects"?
@vitaliy, not having a project set on a repository credential is the same as "all projects".
vitaliyf
commented
3 weeks ago @vitaliy, not having a
projectset on a repository credential is the same as "all projects".
Sounds good to me, as long as this functionality isn't removed. :)
In that case, I agree that it was "user's fault" that we had a repo configured for project "default" but expected it to work everywhere, and just documenting this as a change in behavior starting with 2.12 is sufficient.
crenshaw-dev
commented
3 weeks ago @blakepettersson I wonder if this is the underlying cause of https://github.com/argoproj/argo-cd/issues/19585 and https://github.com/argoproj/argo-cd/issues/18748. If so, the impact may be larger than I'd hoped. :-(
argo1984
commented
3 weeks ago Hi,
i double checked to give the requested informations and found out that we also had "project = default" from the example in the Secret. After removing it it worked.
meyerpeter
commented
3 weeks ago In my case, both secrets and application yamls are using the same non-default project.
blakepettersson
commented
2 weeks ago @meyerpeter perhaps this can be remedied by doing a hard-refresh of the app as in #19585?
meyerpeter
commented
2 weeks ago Sorry, I had a single repository indeed using project: default. Moved that to my applications project and everything is now synced :-)
Describe the bug
I am using the app of apps approach to deploy an application to a Kubernetes 1.30 cluster. The configuration for the app is applied by using multiple sources. Example given below. Currently I am running argocd 2.11.7. Everything works fine!
I was interested in the ability to use argocd's rollback feature for multiple-source approaches which was released in 2.12.
Two repositories, of which one describes the application, and the other provides the configuration values. Both are hosted on GitHub and are using different ssh keys to be fetched by argocd.
Once I upgrade to argocd 2.12, the app-of-apps approach seems to be failing and I can no longer synchronise the application. Downgrading makes everything working again.
To Reproduce
Expected behavior
I expect argocd, to sill be able to sync my application.
Screenshots
Version
v2.12.1+26b2039
Logs
Logs of repo-server: