argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
18.04k stars 5.51k forks source link

Upgrade from v2.11 to 2.12 Breaks Git Secret #19881

Open IdoOzeri opened 2 months ago

IdoOzeri commented 2 months ago

I've upgraded my Argocd installation via the official chart. From chart version 7.3.11 (app version v2.11.7) to 7.4.0 (app version v2.12.0).

I have a Secret resource in my cluster with the following label:

argocd.argoproj.io/secret-type: repository

Once I upgraded to v2.12.0, this repository connection still appears as "Successful" in the user interface, but my applications can no longer sync, complaining of the following error:

SSH agent requested but SSH_AUTH_SOCK not-specified

I've done some digging and found this issue, which suggested modifying the secret-type label to repo-creds. Once I did that, the problem was gone.

My question is, is the secret-type: repository label deprecated in > v2.12.0? I haven't seen any deprecation notice in the docs.

rickardk-se commented 2 months ago

Are you using the same secret across projects?

rhaex commented 2 months ago

I experienced the same issue when upgrading from v2.11.2 to v.2.12.3 using the manifest https://raw.githubusercontent.com/argoproj/argo-cd/v2.12.3/manifests/install.yaml.

Everything upgrades fine, but after the upgrade the SSH connection to the repo is broken and gives the errors as seen above and in the linked issue.

And yes, we use the same repo & secret for several projects.

Modifying the label worked: kubectl -n argocd label --overwrite secret repo-xxxxx argocd.argoproj.io/secret-type=repo-creds

So indeed it looks like the old value has been deprecated and was removed without mentioning it in the release notes or upgrade documentation.

IdoOzeri commented 2 months ago

Are you using the same secret across projects?

Yes, I only have 1 project and 1 secret. Nothing changed but the Argo version.

rickardk-se commented 2 months ago

Sharing secrets across projects stopped working in 2.12.

rhaex commented 2 months ago

Sharing secrets across projects stopped working in 2.12.

Yeah I know, but our secret is not bound to any project. So all projects use the same repo secret, but the secret itself is not bound to a project (so without project field as described here https://argo-cd.readthedocs.io/en/latest/operator-manual/upgrading/2.11-2.12/).

The upgrade breaks the repo connection, until you change secret-type: repository into secret-type: repo-creds, after everything works fine again. And that breaking change is not mentioned anywhere (was it even intentional?).

blakepettersson commented 2 months ago

Sharing secrets across projects stopped working in 2.12.

That is not the case.

And that breaking change is not mentioned anywhere (was it even intentional?).

secret-type: repository definitely works. What did change in 2.12 was that if a secret had a project value set, it can only be used by applications within that same project.

The upgrade breaks the repo connection, until you change secret-type: repository into secret-type: repo-creds, after everything works fine again. And that breaking change is not mentioned anywhere (was it even intentional?).

I haven't seen that one at least, nor have I heard of this elsewhere.

CubicFeat commented 2 months ago

We also ran into this issue.

Changing the label to repo-creds did solve the issue, but I noticed afterward that the k8s secret does contain a project field, and it was set to default, which is exactly what the upgrade notes say will be an issue. We may have created it this way years ago, I'm not sure, but from what I can see there is no way to see or set that via the UI.

I've since changed the label back to just repository and completely removed the project field from the k8s secret, and things still work.

@blakepettersson any chance there is a feature coming where a person can see what projects a repository is scoped to? There are no "view" or "edit" options under Settings->repositories, just a list.

blakepettersson commented 2 months ago

@CubicFeat do you mean in the UI? Since 2.12 there is a PROJECT column to see which project a repository can be scoped to. If it is empty it's globally scoped (it can be used by any project).

Screenshot 2024-09-16 at 19 42 26
Dutchy- commented 2 months ago

Which upgrade notes mention the default project? Because it is not on https://argo-cd.readthedocs.io/en/stable/operator-manual/upgrading/2.11-2.12/ and I think it should be.

blakepettersson commented 2 months ago

Hmm I guess the upgrade notes have not been propagated to readthedocs: https://github.com/argoproj/argo-cd/blob/master/docs/operator-manual/upgrading/2.11-2.12.md

andrii-korotkov-verkada commented 2 weeks ago

Can you share the secret yaml with masked data, please? Also, the code that references the secret.