Git Generator fails calling private repo which other ApplicationSets and Applications successfully connect to

cResults commented 1 month ago

Checklist:

[x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
[x] I've included steps to reproduce the bug.
[x] I've pasted the output of argocd version.

Describe the bug When adding an ApplicationSet that uses a Git Generator to a folder being watched by an app (ie app-of-apps pattern) and the Git Generator is configured with a repoURL that points to a private repo the ApplicationSet fails to deploy its apps throwing the following errors.
Mostly:

error generating params from git: error getting directories from repo: error retrieving Git Directories: rpc error: code = Internal desc = unable to checkout git repo git@github.com:krumIO/gitops_poc.git with revision b01d7cf2898f770e29d03ee1f38c41f991d369b9: failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed exit status 128: git@github.com: Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.

Sometimes:

error generating params from git: error getting directories from repo: error retrieving Git Directories: rpc error: code = Internal desc = unable to resolve git revision : error creating SSH agent: "SSH agent requested but SSH_AUTH_SOCK not-specified"

To Reproduce While this repo doesn't perform the issue out of the box, it has many of the necessary pieces.

1. Clone gitops_poc_public to your local machine
2. Make it private.
3. cd into the root of gitops_starter
4. helm repo add argo-cd https://argoproj.github.io/argo-helm
5. helm dep update ArgoCD/
6. ensure you have the correct context set:  kubectl config current-context
7. helm install argocd ArgoCD/ -n argo --create-namespace
8. kubectl get secret argocd-initial-admin-secret -n argo -o jsonpath="{.data.password}" | base64 -d
9. use port-forward to access the ui and log in
10. Configure ArgoCD to connect to your **private instance** of gitops_poc_public using ssh with -t ed25519
11. replace all instances of "git@github.com:krumIO/gitops_poc.git" with an ssh url for your **private instance** of gitops_poc_public.
12. push changes up to GitHub.
13. kubectl apply -f ./root.yaml  -n argo 
14. kubectl apply -f ./argocd.yaml  -n argo
15. kubectl delete secret -l owner=helm,name=argocd -n argo
16. you may be able to reproduce this issue without setting up ingress for argocd.  If not, you can set that up as you like.  We needed to used cert-manager, metallb and traefik.

Now you should have an instance of ArgoCD that is managing itself with GitOps and a root app-of-apps that points to root-apps directory.

As seen in the image below, we also had ArgoCD configured to manage a downstream cluster, including the single-app-multiple-clusters.yaml moved to the root-apps directory and committed into main successfully deploying apps from the gitops_poc_public repo suggesting there isn't anything wrong with ArgoCD's connection to that git repository.

Now moving the many-apps-single-cluster.yaml to the root-apps directory, I'd expect The error shown in the argocd-applicationset-controller-logs, many-apps-single-cluster with look as it does in the image below and when clicking it one of the errors listed above will be visible.

Expected behavior An ApplicationSet using a Git Generator should be able to load apps from a private git repository that has been properly configured in ArgoCD confirmed by other ArgoCD Applications and ApplicationSets correctly pulling data from the git repository.

Screen Shot

Version

argocd: v2.12.1+26b2039
  BuildDate: 2024-08-16T17:01:06Z
  GitCommit: 26b2039a55b9bdf807a70d344af8ade5171d3d39
  GitTreeState: clean
  GoVersion: go1.22.6
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.12.3+6b9cd82
  BuildDate: 2024-08-27T11:57:48Z
  GitCommit: 6b9cd828c6e9807398869ad5ac44efd2c28422d6
  GitTreeState: clean
  GoVersion: go1.22.4
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.4.2 2024-05-22T15:19:38Z
  Helm Version: v3.15.2+g1a500d5
  Kubectl Version: v0.29.6
  Jsonnet Version: v0.20.0

Logs argocd-applicationset-controller-logs This contains what appears to be an error stack Most important line is [controller-runtime] log.SetLogger(...) was never called; logs will not be displayed.

UPDATE: The logs below don't appear to be relevant to the issue. I performed another test starting with both ApplicationSets out of the root-apps directory (App-Of-Apps). When I committed the first ApplicationSet into the root-apps directory that just uses the cluster generator, similar logs to below were added in the argocd-applicationset-controller.

The relevant logs created when the ApplicationSet using git generator was added to root-app and committed into git for both the argocd-applicationset-controller and argocd-repo-server can be found here.

time="2024-09-24T20:14:41Z" level=info msg="ArgoCD ApplicationSet Controller is starting" built="2024-08-27T11:57:48Z" commit=6b9cd828c6e9807398869ad5ac44efd2c28422d6 namespace=argo version=v2.12.3+6b9cd82
time="2024-09-24T20:14:41Z" level=info msg="Starting configmap/secret informers"
time="2024-09-24T20:14:41Z" level=info msg="Configmap/secret informer synced"
time="2024-09-24T20:14:41Z" level=info msg="Starting webhook server"
time="2024-09-24T20:14:41Z" level=info msg="Starting manager"
time="2024-09-24T20:14:41Z" level=info msg="processing event for cluster secret" name=dcs-dev-secret namespace=argo type=createSecretEventHandler
time="2024-09-24T20:14:41Z" level=info msg="listed ApplicationSets" count=1 type=createSecretEventHandler
time="2024-09-24T20:14:41Z" level=info msg="matched local cluster" cluster="local cluster"
time="2024-09-24T20:14:41Z" level=info msg="matched cluster secret" cluster=dcs-dev-secret
time="2024-09-24T20:14:41Z" level=info msg="generated 2 applications" applicationset=argo/single-app-multiple-clusters generator="{nil &ClusterGenerator{Selector:{map[] []},Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:nil,Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},Values:map[string]string{},} nil nil nil nil nil nil nil nil}"
time="2024-09-24T20:14:41Z" level=info msg="end reconcile" applicationset=argo/single-app-multiple-clusters requeueAfter=0s
time="2024-09-24T20:14:41Z" level=info msg="matched local cluster" cluster="local cluster"
time="2024-09-24T20:14:41Z" level=info msg="matched cluster secret" cluster=dcs-dev-secret
time="2024-09-24T20:14:41Z" level=info msg="generated 2 applications" applicationset=argo/single-app-multiple-clusters generator="{nil &ClusterGenerator{Selector:{map[] []},Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:nil,Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},Values:map[string]string{},} nil nil nil nil nil nil nil nil}"
time="2024-09-24T20:14:41Z" level=info msg="end reconcile" applicationset=argo/single-app-multiple-clusters requeueAfter=0s
[controller-runtime] log.SetLogger(...) was never called; logs will not be displayed.
Detected at:
    >  goroutine 182 [running]:
    >  runtime/debug.Stack()
    >   /usr/local/go/src/runtime/debug/stack.go:24 +0x5e
    >  sigs.k8s.io/controller-runtime/pkg/log.eventuallyFulfillRoot()
    >   /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/log/log.go:60 +0xcd
    >  sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).WithValues(0xc000777cc0, {0xc000a61cc0, 0x2, 0x2})
    >   /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/log/deleg.go:168 +0x49
    >  github.com/go-logr/logr.Logger.WithValues(...)
    >   /go/pkg/mod/github.com/go-logr/logr@v1.4.1/logr.go:332
    >  sigs.k8s.io/controller-runtime/pkg/builder.(*Builder).doController.func1(0xc000a61ca0)
    >   /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/builder/controller.go:400 +0x193
    >  sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000b8af00, {0x558df00, 0xc00074c960}, {0x3a6d5a0, 0xc000a61c80})
    >   /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:306 +0x162
    >  sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000b8af00, {0x558df00, 0xc00074c960})
    >   /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266 +0x1be
    >  sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
    >   /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227 +0x79
    >  created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 127
    >   /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:223 +0x50c
time="2024-09-24T20:23:54Z" level=error msg="error generating params" error="error generating params from git: error getting directories from repo: error retrieving Git Directories: rpc error: code = Internal desc = unable to checkout git repo git@github.com:krumIO/gitops_poc.git with revision b01d7cf2898f770e29d03ee1f38c41f991d369b9: failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed exit status 128: git@github.com: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." generator="&{0xc0000722c0 argo}"
time="2024-09-24T20:23:54Z" level=error msg="error generating application from params" applicationset=argo/many-apps-single-cluster error="error generating params from git: error getting directories from repo: error retrieving Git Directories: rpc error: code = Internal desc = unable to checkout git repo git@github.com:krumIO/gitops_poc.git with revision b01d7cf2898f770e29d03ee1f38c41f991d369b9: failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed exit status 128: git@github.com: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." generator="{nil nil &GitGenerator{RepoURL:git@github.com:krumIO/gitops_poc.git,Directories:[]GitDirectoryGeneratorItem{GitDirectoryGeneratorItem{Path:services/platform/*,Exclude:false,},},Files:[]GitFileGeneratorItem{},Revision:HEAD,RequeueAfterSeconds:nil,Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:nil,Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},PathParamPrefix:,Values:map[string]string{},} nil nil nil nil nil nil nil}"
time="2024-09-24T20:23:54Z" level=error msg="error generating params" error="error generating params from git: error getting directories from repo: error retrieving Git Directories: rpc error: code = Internal desc = unable to checkout git repo git@github.com:krumIO/gitops_poc.git with revision b01d7cf2898f770e29d03ee1f38c41f991d369b9: failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed exit status 128: git@github.com: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." generator="&{0xc0000722c0 argo}"
time="2024-09-24T20:23:54Z" level=error msg="error generating application from params" applicationset=argo/many-apps-single-cluster error="error generating params from git: error getting directories from repo: error retrieving Git Directories: rpc error: code = Internal desc = unable to checkout git repo git@github.com:krumIO/gitops_poc.git with revision b01d7cf2898f770e29d03ee1f38c41f991d369b9: failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed exit status 128: git@github.com: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." generator="{nil nil &GitGenerator{RepoURL:git@github.com:krumIO/gitops_poc.git,Directories:[]GitDirectoryGeneratorItem{GitDirectoryGeneratorItem{Path:services/platform/*,Exclude:false,},},Files:[]GitFileGeneratorItem{},Revision:HEAD,RequeueAfterSeconds:nil,Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:nil,Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},PathParamPrefix:,Values:map[string]string{},} nil nil nil nil nil nil nil}"
time="2024-09-24T20:23:55Z" level=error msg="error generating params" error="error generating params from git: error getting directories from repo: error retrieving Git Directories: rpc error: code = Internal desc = unable to checkout git repo git@github.com:krumIO/gitops_poc.git with revision b01d7cf2898f770e29d03ee1f38c41f991d369b9: failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed exit status 128: git@github.com: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." generator="&{0xc0000722c0 argo}"

We also have argocd-repo-server logs if you think they would be helpful.

blakepettersson commented 1 month ago

What does the (redacted) secret look like?

cResults commented 1 month ago

@blakepettersson Thank you for the rapid response. Can you be more specific? What secret are you referring to and how is it associated with the Git Generator?

blakepettersson commented 1 month ago

@cResults well, if the Git repo is private you would need some credentials to be able to access it. Do you have such a secret and if so what does it look like?

cResults commented 1 month ago

@blakepettersson ArgoCD has in its UI the ability to connect with a private git repo via https, ssh, GitHub app and Google Cloud. The instance that is failing is configured with ssh as stated above. The repo that the Git Generator is failing to call is currently successfully being called by two ArgoCD Applications and an ApplicationSet using the Cluster Generator. Its only failing whenever the Git Generator is involved. Per the logs, it appears to be failing in the argocd-applicationset.controller and not in the argocd-repo-server.

blakepettersson commented 1 month ago

@cResults remove the project default so that it's unset. Applicationsets are admin-level objects which don't belong to a project.

cResults commented 1 month ago

I appreciate the dialog. We have an ApplicationSet using the cluster generator that is successfully communicating with the connection in the image above with default project. Additionally, the UI requires a project to be selected.

blakepettersson commented 1 month ago

I'm talking about the repository credential...

cResults commented 1 month ago

Ok. Where? In GitHub, ArgoCD UI, or the ApplicationSet git generator itself?

cResults commented 1 month ago

So and ran another test to find which logs are generated when the many-apps-single-cluster ApplicationSet (which as the git generator that is failing) was committed into main.

Minimal logs in argocd-applicationset-controller may be impacted by this log entry [controller-runtime] log.SetLogger(...) was never called; logs will not be displayed.

These are the only relevant logs from argocd-applicationset-controller:

time="2024-09-26T12:33:08Z" level=error msg="error generating params" error="error generating params from git: error getting directories from repo: error retrieving Git Directories: rpc error: code = Internal desc = unable to checkout git repo git@github.com:krumIO/gitops_poc.git with revision 350a00137816482192e621679d206b30d0a27907: failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed exit status 128: git@github.com: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." generator="&{0xc00077d280 argo}"
time="2024-09-26T12:33:08Z" level=error msg="error generating application from params" applicationset=argo/many-apps-single-cluster error="error generating params from git: error getting directories from repo: error retrieving Git Directories: rpc error: code = Internal desc = unable to checkout git repo git@github.com:krumIO/gitops_poc.git with revision 350a00137816482192e621679d206b30d0a27907: failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed exit status 128: git@github.com: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." generator="{nil nil &GitGenerator{RepoURL:git@github.com:krumIO/gitops_poc.git,Directories:[]GitDirectoryGeneratorItem{GitDirectoryGeneratorItem{Path:services/platform/*,Exclude:false,},},Files:[]GitFileGeneratorItem{},Revision:HEAD,RequeueAfterSeconds:nil,Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:nil,Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},PathParamPrefix:,Values:map[string]string{},} nil nil nil nil nil nil nil}"

The following appear to be relevant from the argocd-repo-server

time="2024-09-26T12:33:03Z" level=info msg="manifest cache hit: &ApplicationSource{RepoURL:git@github.com:krumIO/gitops_poc.git,Path:root-apps,TargetRevision:HEAD,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,}/350a00137816482192e621679d206b30d0a27907"
time="2024-09-26T12:33:03Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2024-09-26T12:33:03Z" grpc.time_ms=0.535 span.kind=server system=grpc
time="2024-09-26T12:33:06Z" level=info msg="manifest cache hit: &ApplicationSource{RepoURL:git@github.com:krumIO/gitops_poc.git,Path:root-apps,TargetRevision:HEAD,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,}/350a00137816482192e621679d206b30d0a27907"
time="2024-09-26T12:33:06Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2024-09-26T12:33:06Z" grpc.time_ms=0.388 span.kind=server system=grpc
time="2024-09-26T12:33:08Z" level=info msg="manifest cache hit: &ApplicationSource{RepoURL:git@github.com:krumIO/gitops_poc.git,Path:root-apps,TargetRevision:HEAD,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,}/350a00137816482192e621679d206b30d0a27907"
time="2024-09-26T12:33:08Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2024-09-26T12:33:08Z" grpc.time_ms=0.392 span.kind=server system=grpc
time="2024-09-26T12:33:08Z" level=info msg="manifest cache hit: &ApplicationSource{RepoURL:git@github.com:krumIO/gitops_poc.git,Path:root-apps,TargetRevision:HEAD,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,}/350a00137816482192e621679d206b30d0a27907"
time="2024-09-26T12:33:08Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2024-09-26T12:33:08Z" grpc.time_ms=1.234 span.kind=server system=grpc
time="2024-09-26T12:33:08Z" level=info msg="Initializing git@github.com:krumIO/gitops_poc.git to /tmp/_argocd-repo/5f186cfa-17dd-4b76-a597-fc7afe3adeef"
time="2024-09-26T12:33:08Z" level=info msg="git cat-file -t 350a00137816482192e621679d206b30d0a27907" dir=/tmp/_argocd-repo/5f186cfa-17dd-4b76-a597-fc7afe3adeef execID=10352
time="2024-09-26T12:33:08Z" level=info msg=Trace args="[git cat-file -t 350a00137816482192e621679d206b30d0a27907]" dir=/tmp/_argocd-repo/5f186cfa-17dd-4b76-a597-fc7afe3adeef operation_name="exec git" time_ms=2.75085
time="2024-09-26T12:33:08Z" level=info msg="git fetch origin --tags --force --prune" dir=/tmp/_argocd-repo/5f186cfa-17dd-4b76-a597-fc7afe3adeef execID=77373
time="2024-09-26T12:33:08Z" level=info msg="revision metadata cache miss: git@github.com:krumIO/gitops_poc.git/350a00137816482192e621679d206b30d0a27907"
time="2024-09-26T12:33:08Z" level=info msg="git cat-file -t 350a00137816482192e621679d206b30d0a27907" dir=/tmp/_argocd-repo/78ac7b89-d948-4208-a5e1-89252ce029dc execID=51550
time="2024-09-26T12:33:08Z" level=info msg=Trace args="[git cat-file -t 350a00137816482192e621679d206b30d0a27907]" dir=/tmp/_argocd-repo/78ac7b89-d948-4208-a5e1-89252ce029dc operation_name="exec git" time_ms=2.26162
time="2024-09-26T12:33:08Z" level=info msg="git checkout --force 350a00137816482192e621679d206b30d0a27907" dir=/tmp/_argocd-repo/78ac7b89-d948-4208-a5e1-89252ce029dc execID=c6c5d
time="2024-09-26T12:33:08Z" level=info msg=Trace args="[git checkout --force 350a00137816482192e621679d206b30d0a27907]" dir=/tmp/_argocd-repo/78ac7b89-d948-4208-a5e1-89252ce029dc operation_name="exec git" time_ms=2.567005
time="2024-09-26T12:33:08Z" level=info msg="git clean -ffdx" dir=/tmp/_argocd-repo/78ac7b89-d948-4208-a5e1-89252ce029dc execID=a6ab2
time="2024-09-26T12:33:08Z" level=info msg=Trace args="[git clean -ffdx]" dir=/tmp/_argocd-repo/78ac7b89-d948-4208-a5e1-89252ce029dc operation_name="exec git" time_ms=1.499759
time="2024-09-26T12:33:08Z" level=info msg="git show -s --format=%an <%ae>|%at|%B 350a00137816482192e621679d206b30d0a27907" dir=/tmp/_argocd-repo/78ac7b89-d948-4208-a5e1-89252ce029dc execID=df583
time="2024-09-26T12:33:08Z" level=info msg=Trace args="[git show -s --format=%an <%ae>|%at|%B 350a00137816482192e621679d206b30d0a27907]" dir=/tmp/_argocd-repo/78ac7b89-d948-4208-a5e1-89252ce029dc operation_name="exec git" time_ms=1.746332
time="2024-09-26T12:33:08Z" level=info msg="git tag --points-at 350a00137816482192e621679d206b30d0a27907" dir=/tmp/_argocd-repo/78ac7b89-d948-4208-a5e1-89252ce029dc execID=bd096
time="2024-09-26T12:33:08Z" level=info msg=Trace args="[git tag --points-at 350a00137816482192e621679d206b30d0a27907]" dir=/tmp/_argocd-repo/78ac7b89-d948-4208-a5e1-89252ce029dc operation_name="exec git" time_ms=2.685907
time="2024-09-26T12:33:08Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetRevisionMetadata grpc.service=repository.RepoServerService grpc.start_time="2024-09-26T12:33:08Z" grpc.time_ms=13.44 span.kind=server system=grpc
time="2024-09-26T12:33:08Z" level=error msg="`git fetch origin --tags --force --prune` failed exit status 128: git@github.com: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." execID=77373
time="2024-09-26T12:33:08Z" level=info msg=Trace args="[git fetch origin --tags --force --prune]" dir=/tmp/_argocd-repo/5f186cfa-17dd-4b76-a597-fc7afe3adeef operation_name="exec git" time_ms=214.741059
time="2024-09-26T12:33:08Z" level=error msg="finished unary call with code Internal" error="rpc error: code = Internal desc = unable to checkout git repo git@github.com:krumIO/gitops_poc.git with revision 350a00137816482192e621679d206b30d0a27907: failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed exit status 128: git@github.com: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." grpc.code=Internal grpc.method=GetGitDirectories grpc.service=repository.RepoServerService grpc.start_time="2024-09-26T12:33:08Z" grpc.time_ms=220.233 span.kind=server system=grpc
time="2024-09-26T12:33:08Z" level=info msg="git cat-file -t 350a00137816482192e621679d206b30d0a27907" dir=/tmp/_argocd-repo/5f186cfa-17dd-4b76-a597-fc7afe3adeef execID=81c52
time="2024-09-26T12:33:08Z" level=info msg=Trace args="[git cat-file -t 350a00137816482192e621679d206b30d0a27907]" dir=/tmp/_argocd-repo/5f186cfa-17dd-4b76-a597-fc7afe3adeef operation_name="exec git" time_ms=2.898126
time="2024-09-26T12:33:08Z" level=info msg="git fetch origin --tags --force --prune" dir=/tmp/_argocd-repo/5f186cfa-17dd-4b76-a597-fc7afe3adeef execID=41c1e
time="2024-09-26T12:33:08Z" level=error msg="`git fetch origin --tags --force --prune` failed exit status 128: git@github.com: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." execID=41c1e
time="2024-09-26T12:33:08Z" level=info msg=Trace args="[git fetch origin --tags --force --prune]" dir=/tmp/_argocd-repo/5f186cfa-17dd-4b76-a597-fc7afe3adeef operation_name="exec git" time_ms=200.75763899999998
time="2024-09-26T12:33:08Z" level=error msg="finished unary call with code Internal" error="rpc error: code = Internal desc = unable to checkout git repo git@github.com:krumIO/gitops_poc.git with revision 350a00137816482192e621679d206b30d0a27907: failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed exit status 128: git@github.com: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." grpc.code=Internal grpc.method=GetGitDirectories grpc.service=repository.RepoServerService grpc.start_time="2024-09-26T12:33:08Z" grpc.time_ms=204.353 span.kind=server system=grpc
time="2024-09-26T12:33:08Z" level=info msg="git cat-file -t 350a00137816482192e621679d206b30d0a27907" dir=/tmp/_argocd-repo/5f186cfa-17dd-4b76-a597-fc7afe3adeef execID=3e9d8
time="2024-09-26T12:33:08Z" level=info msg="manifest cache hit: &ApplicationSource{RepoURL:git@github.com:krumIO/gitops_poc.git,Path:root-apps,TargetRevision:HEAD,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,}/350a00137816482192e621679d206b30d0a27907"
time="2024-09-26T12:33:08Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2024-09-26T12:33:08Z" grpc.time_ms=364.012 span.kind=server system=grpc
time="2024-09-26T12:33:08Z" level=info msg=Trace args="[git cat-file -t 350a00137816482192e621679d206b30d0a27907]" dir=/tmp/_argocd-repo/5f186cfa-17dd-4b76-a597-fc7afe3adeef operation_name="exec git" time_ms=3.717657
time="2024-09-26T12:33:08Z" level=info msg="git fetch origin --tags --force --prune" dir=/tmp/_argocd-repo/5f186cfa-17dd-4b76-a597-fc7afe3adeef execID=4ceb9
time="2024-09-26T12:33:09Z" level=info msg="manifest cache hit: &ApplicationSource{RepoURL:git@github.com:krumIO/gitops_poc.git,Path:root-apps,TargetRevision:HEAD,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,}/350a00137816482192e621679d206b30d0a27907"
time="2024-09-26T12:33:09Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2024-09-26T12:33:09Z" grpc.time_ms=0.927 span.kind=server system=grpc
time="2024-09-26T12:33:09Z" level=error msg="`git fetch origin --tags --force --prune` failed exit status 128: git@github.com: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." execID=4ceb9
time="2024-09-26T12:33:09Z" level=info msg=Trace args="[git fetch origin --tags --force --prune]" dir=/tmp/_argocd-repo/5f186cfa-17dd-4b76-a597-fc7afe3adeef operation_name="exec git" time_ms=208.983739
time="2024-09-26T12:33:09Z" level=error msg="finished unary call with code Internal" error="rpc error: code = Internal desc = unable to checkout git repo git@github.com:krumIO/gitops_poc.git with revision 350a00137816482192e621679d206b30d0a27907: failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed exit status 128: git@github.com: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." grpc.code=Internal grpc.method=GetGitDirectories grpc.service=repository.RepoServerService grpc.start_time="2024-09-26T12:33:08Z" grpc.time_ms=213.325 span.kind=server system=grpc
time="2024-09-26T12:33:09Z" level=info msg="git cat-file -t 350a00137816482192e621679d206b30d0a27907" dir=/tmp/_argocd-repo/5f186cfa-17dd-4b76-a597-fc7afe3adeef execID=2a26d
time="2024-09-26T12:33:09Z" level=info msg=Trace args="[git cat-file -t 350a00137816482192e621679d206b30d0a27907]" dir=/tmp/_argocd-repo/5f186cfa-17dd-4b76-a597-fc7afe3adeef operation_name="exec git" time_ms=1.684326
time="2024-09-26T12:33:09Z" level=info msg="git fetch origin --tags --force --prune" dir=/tmp/_argocd-repo/5f186cfa-17dd-4b76-a597-fc7afe3adeef execID=f5a28
time="2024-09-26T12:33:09Z" level=error msg="`git fetch origin --tags --force --prune` failed exit status 128: git@github.com: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." execID=f5a28
time="2024-09-26T12:33:09Z" level=info msg=Trace args="[git fetch origin --tags --force --prune]" dir=/tmp/_argocd-repo/5f186cfa-17dd-4b76-a597-fc7afe3adeef operation_name="exec git" time_ms=199.080307
time="2024-09-26T12:33:09Z" level=error msg="finished unary call with code Internal" error="rpc error: code = Internal desc = unable to checkout git repo git@github.com:krumIO/gitops_poc.git with revision 350a00137816482192e621679d206b30d0a27907: failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed exit status 128: git@github.com: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." grpc.code=Internal grpc.method=GetGitDirectories grpc.service=repository.RepoServerService grpc.start_time="2024-09-26T12:33:09Z" grpc.time_ms=201.693 span.kind=server system=grpc
time="2024-09-26T12:33:09Z" level=info msg="git cat-file -t 350a00137816482192e621679d206b30d0a27907" dir=/tmp/_argocd-repo/5f186cfa-17dd-4b76-a597-fc7afe3adeef execID=7d753
time="2024-09-26T12:33:09Z" level=info msg=Trace args="[git cat-file -t 350a00137816482192e621679d206b30d0a27907]" dir=/tmp/_argocd-repo/5f186cfa-17dd-4b76-a597-fc7afe3adeef operation_name="exec git" time_ms=1.716956
time="2024-09-26T12:33:09Z" level=info msg="git fetch origin --tags --force --prune" dir=/tmp/_argocd-repo/5f186cfa-17dd-4b76-a597-fc7afe3adeef execID=4d88a
time="2024-09-26T12:33:09Z" level=error msg="`git fetch origin --tags --force --prune` failed exit status 128: git@github.com: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." execID=4d88a
time="2024-09-26T12:33:09Z" level=info msg=Trace args="[git fetch origin --tags --force --prune]" dir=/tmp/_argocd-repo/5f186cfa-17dd-4b76-a597-fc7afe3adeef operation_name="exec git" time_ms=199.84242899999998
time="2024-09-26T12:33:09Z" level=error msg="finished unary call with code Internal" error="rpc error: code = Internal desc = unable to checkout git repo git@github.com:krumIO/gitops_poc.git with revision 350a00137816482192e621679d206b30d0a27907: failed to initialize repository resources: rpc error: code = Internal desc = Failed to fetch default: `git fetch origin --tags --force --prune` failed exit status 128: git@github.com: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." grpc.code=Internal grpc.method=GetGitDirectories grpc.service=repository.RepoServerService grpc.start_time="2024-09-26T12:33:09Z" grpc.time_ms=202.125 span.kind=server system=grpc
time="2024-09-26T12:33:09Z" level=info msg="git cat-file -t 350a00137816482192e621679d206b30d0a27907" dir=/tmp/_argocd-repo/5f186cfa-17dd-4b76-a597-fc7afe3adeef execID=956a5
time="2024-09-26T12:33:09Z" level=info msg=Trace args="[git cat-file -t 350a00137816482192e621679d206b30d0a27907]" dir=/tmp/_argocd-repo/5f186cfa-17dd-4b76-a597-fc7afe3adeef operation_name="exec git" time_ms=1.8931079999999998
time="2024-09-26T12:33:09Z" level=info msg="git fetch origin --tags --force --prune" dir=/tmp/_argocd-repo/5f186cfa-17dd-4b76-a597-fc7afe3adeef execID=bf251

TrungKenbi commented 1 month ago

What does the (redacted) secret look like?

kubectl get secret repo-559988242 -n argocd -o json | jq -r '.data | to_entries[] | "\(.key): \(.value | @base64d)"'
githubAppID: 758838
githubAppInstallationID: 45888164
githubAppPrivateKey: -----BEGIN RSA PRIVATE KEY-----
MIIEpQIBAAKCAQEA3OpNV2YNroVwv+MhMSnNwhLYgtUcgp0FhYDpVG21PQsY6zu8
6aMs364Q9L2UH/yPNwTaWItnDpbzF10ljD2BxCYeuWxyj53QS4lGappd7J/vYum4
....
JTsgT926/8s713cZwDjiDGlDYoEtMqIKtb7yI7eoEB0/1One9bNoMEg=
-----END RSA PRIVATE KEY-----

project: default
type: git
url: https://github.com/org-name/gitops-repo

cResults commented 4 weeks ago

@agaudreault you re-labeled this from bug to question. Is there any feedback you can give me as to what might be the issue if its not a bug?

agaudreault commented 4 weeks ago

This seems to be a configuration error as the bug has not yet been identified. See https://github.com/argoproj/argo-cd/issues/20088#issuecomment-2374383045 for the suggestion on the fix and the release notes for a detailed explanation https://argo-cd.readthedocs.io/en/latest/operator-manual/upgrading/2.11-2.12/

cResults commented 3 weeks ago

Thank you @agaudreault for the link to the release notes. After reading that I found this at the bottom of the Git Generator page.

I didn't trust the prior responses as they were asking for the contents of my secret.

ivan-cai commented 3 weeks ago

Thank you @agaudreault for the link to the release notes. After reading that I found this at the bottom of the Git Generator page.

I didn't trust the prior responses as they were asking for the contents of my secret.

How can I unset the project field for git generator appset？ I removed project in repo secret，applicaitons can created，but I get the error： message: application repo git@git.xxx.cn:PEStudio/pegitops.git is not permitted in project ‘pecloud’

argoproj / argo-cd

Git Generator fails calling private repo which other ApplicationSets and Applications successfully connect to #20088