search

argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
17.94k stars 5.46k forks source link

verifyGnuPGSignature check triggers default case (Could not verify commit signature on revision) #14009

Closed Links2004 closed 1 year ago

Links2004 commented 1 year ago

Checklist:

Describe the bug

gitlab and argocd UI say GPG signature is valid but sync errors with:

ComparisonError - Could not verify commit signature on revision '910788d9d64dcb613ba1c1c2b21aeeb07121ca91', check logs for more information.

log does not provide more info.

looks like the code is ending up in this default case here: https://github.com/argoproj/argo-cd/blob/aa3b95561b0ad6902d7292819e2233257e6a2717/controller/state.go#L343-L345

To Reproduce

use git with GPG signature and enforce signature

Expected behavior

GPG check is working correct like in V2.5.18 or V2.6.8

Screenshots

image

image

image

Version

2.6.9 shows the same problem.

deployed via https://raw.githubusercontent.com/argoproj/argo-cd/v2.7.4/manifests/install.yaml

argocd: v2.7.4+a33baa3.dirty
  BuildDate: 2023-06-05T19:00:34Z
  GitCommit: a33baa301fe61b899dc8bbad9e554efbc77e0991
  GitTreeState: dirty
  GoVersion: go1.19.6
  Compiler: gc
  Platform: linux/amd64

Logs

argocd-repo-server-59c98499f8-wt2qx argocd-repo-server time="2023-06-12T19:00:40Z" level=info msg="manifest cache hit: &ApplicationSource{RepoURL:https://gitlab.xxxxxxxx/k8s/gitops_k8s_local.git,Path:kube-system/metallb,TargetRevision:master,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,}/910788d9d64dcb613ba1c1c2b21aeeb07121ca91"
argocd-repo-server-59c98499f8-wt2qx argocd-repo-server time="2023-06-12T19:00:40Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2023-06-12T19:00:40Z" grpc.time_ms=3.251 span.kind=server system=grpc
argocd-application-controller-0 argocd-application-controller time="2023-06-12T19:00:40Z" level=info msg="getRepoObjs stats" application=argocd/kube-system-metallb build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=24 unmarshal_ms=24 version_ms=0
argocd-repo-server-59c98499f8-wt2qx argocd-repo-server time="2023-06-12T19:00:40Z" level=info msg="manifest cache hit: &ApplicationSource{RepoURL:https://gitlab.xxxxxxxx/k8s/gitops_k8s_local.git,Path:kube-system,TargetRevision:master,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,}/910788d9d64dcb613ba1c1c2b21aeeb07121ca91"
argocd-repo-server-59c98499f8-wt2qx argocd-repo-server time="2023-06-12T19:00:40Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2023-06-12T19:00:40Z" grpc.time_ms=0.879 span.kind=server system=grpc
argocd-application-controller-0 argocd-application-controller time="2023-06-12T19:00:40Z" level=info msg="getRepoObjs stats" application=argocd/kube-system build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=10 unmarshal_ms=10 version_ms=0
argocd-server-65d6c845b6-bc688 argocd-server time="2023-06-12T19:00:40Z" level=info msg="received unary call /application.ApplicationService/GetApplicationSyncWindows" grpc.method=GetApplicationSyncWindows grpc.request.content="name:\"kube-system-metallb\" appNamespace:\"argocd\" " grpc.service=application.ApplicationService grpc.start_time="2023-06-12T19:00:40Z" span.kind=server system=grpc
argocd-server-65d6c845b6-bc688 argocd-server time="2023-06-12T19:00:40Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetApplicationSyncWindows grpc.service=application.ApplicationService grpc.start_time="2023-06-12T19:00:40Z" grpc.time_ms=17.355 span.kind=server system=grpc
argocd-application-controller-0 argocd-application-controller time="2023-06-12T19:00:40Z" level=info msg="No status changes. Skipping patch" application=argocd/kube-system
argocd-application-controller-0 argocd-application-controller time="2023-06-12T19:00:40Z" level=info msg="Reconciliation completed" application=argocd/kube-system dedup_ms=0 dest-name= dest-namespace=kube-system dest-server="https://kubernetes.default.svc" diff_ms=19 fields.level=1 git_ms=10 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=41
argocd-application-controller-0 argocd-application-controller time="2023-06-12T19:00:40Z" level=info msg="updated 'argocd/kube-system-metallb' operation (phase: Error)"
argocd-application-controller-0 argocd-application-controller time="2023-06-12T19:00:40Z" level=info msg="Sync operation to 910788d9d64dcb613ba1c1c2b21aeeb07121ca91 failed: ComparisonError: Could not verify commit signature on revision '910788d9d64dcb613ba1c1c2b21aeeb07121ca91', check logs for more information." application=kube-system-metallb dest-namespace=kube-system dest-server="https://kubernetes.default.svc" reason=OperationCompleted type=Warning
argocd-notifications-controller-65cddcf9d6-hw5wk argocd-notifications-controller time="2023-06-12T19:00:40Z" level=info msg="Start processing" resource=argocd/kube-system-metallb
argocd-notifications-controller-65cddcf9d6-hw5wk argocd-notifications-controller time="2023-06-12T19:00:40Z" level=info msg="Processing skipped: sync status out of date" resource=argocd/kube-system-metallb

example gitlog:

$ git log
commit 507df20b0167191986e6cfdd3f096557ab99b67c (HEAD -> master, origin/master, origin/HEAD)
Author: gitlab_ci_user <gitlab@XXXXXXXXXX>
Date:   Mon Jun 12 19:37:30 2023 +0000

    release argocd based on pipelines/583 - d0d786ff92449cd7c88216ddc90ff7983a802eec

commit 9ad88216630761dca48e19149713944165c8ea53
Author: gitlab_ci_user <gitlab@XXXXXXXXXX>
Date:   Mon Jun 12 19:29:00 2023 +0000

    release argocd based on pipelines/582 - 3e183275cac2914e41386cf3df8311379a160260

commit 0bd283decc59ddab5a6bdd1fe4a7f293fd7844ab
Author: gitlab_ci_user <gitlab@XXXXXXXXXX>
Date:   Mon Jun 12 19:23:33 2023 +0000

    release argocd based on pipelines/581 - 7e36a50f366a90bced4feb1ad1b3fd6af408b373

commit 474fbd25630f7ef184ac1a500434c1dea9e18360
Author: gitlab_ci_user <gitlab@XXXXXXXXXX>
Date:   Mon Jun 12 19:15:00 2023 +0000

    release argocd based on pipelines/580 - 896086e1694aa0633cec0afe23a52c91e2405808

commit 910788d9d64dcb613ba1c1c2b21aeeb07121ca91
Author: gitlab_ci_user <gitlab@XXXXXXXXXX>
Date:   Mon Jun 12 18:52:30 2023 +0000

    release argocd based on pipelines/579 - 3a351cfe25bbddb0e3d3997d5451631524d1eb54

commit da4fdd6f58267efb324600c438606aa336403620
Author: gitlab_ci_user <gitlab@XXXXXXXXXX>
Date:   Mon Jun 12 18:47:34 2023 +0000

    release argocd based on pipelines/578 - 88b2449b6a878f31676906870b9508ecce232a97
tvictor-paxos commented 1 year ago

Facing the exact same issue after an upgrade from 2.6.3 to 2.7.6

hapeho commented 1 year ago

Same here, after upgrade from v2.6.8 to v2.7.5 and still in v2.7.6. Only one Application is affected, it deploys a Helm chart from git and has a dependency with an oci:// URL. Others (mostly kustomize based) appear to work fine.

jannfis commented 1 year ago

Can you please paste output of git log --show-signature for the appropriate commit?

hapeho commented 1 year ago 
commit 976df8915507376005fa27fb1d90ca7be579125a (HEAD -> testing, origin/testing)
gpg: Signature made Thu Jun 22 14:02:04 2023 CEST
gpg:                using RSA key B0230D84D45111B6F873A33CF5B4B1B588A59946
gpg: Good signature from "{correct name/email redacted}" [ultimate]
Author: {correct name/email redacted}
Date:   Thu Jun 22 14:02:02 2023 +0200

    Whitespace change

All commits are signed by the same key.

As in the original report, the signatures are fine in Gitlab. ArgoCD itself reports Good signature from RSA key F5B4B1B588A59946, despite giving the message ComparisonError: Could not verify commit signature on revision '976df8915507376005fa27fb1d90ca7be579125a', check logs for more information. in the Sync Status.

jannfis commented 1 year ago

Is there any chance you can run this on the repository server on a clone of your repository?

For example, from within the repository server pod, run:

$ cd /tmp
$ git clone <yourrepository>
$ cd <yourrepository>
$ GNUPGHOME=/app/config/gpg/keys git verify-commit 976df8915507376005fa27fb1d90ca7be579125a

and then paste the output here?

That error message you see is weird. Potentially, some GnuPG breakage somewhere. It should only occur when the regexp define here doesn't match the output.

hapeho commented 1 year ago

It seems that the publickey in question configured in ArgoCD does not have the latest signatures, hence the signed commits appear to come from a key that expired in January. This did and does not bother ArgoCD, other Applications/Repositories) as are working with the same key.

Below is the non-working one as requested. I also used egrep to test against the regexp and it appears to match:

argocd@argocd-repo-server-5488979b5c-4hthf:/tmp/mastodon$ GNUPGHOME=/app/config/gpg/keys git verify-commit 976df8915507376005fa27fb1d90ca7be579125a
gpg: WARNING: unsafe ownership on homedir '/app/config/gpg/keys'
gpg: Signature made Thu Jun 22 12:02:04 2023 UTC
gpg:                using RSA key B0230D84D45111B6F873A33CF5B4B1B588A59946
gpg: Good signature from "Xxxxxxxxx Xxxxxxx <xxxxxxxxx.xxxxxxx@xxxx.xx.xx>" [expired]
gpg: Note: This key has expired!
Primary key fingerprint: 7AF1 4C48 84F6 20AE D8A8  DF35 AA6E 994F 6CC1 64FC
     Subkey fingerprint: B023 0D84 D451 11B6 F873  A33C F5B4 B1B5 88A5 9946

argocd@argocd-repo-server-5488979b5c-4hthf:/tmp/mastodon$ GNUPGHOME=/app/config/gpg/keys git verify-commit 976df8915507376005fa27fb1d90ca7be579125a 2>&1 | egrep '^gpg: ([a-zA-Z]+) signature from "([^"]+)" \[([a-zA-Z]+)\]$'
gpg: Good signature from "Xxxxxxxxx Xxxxxxx <xxxxxxxxx.xxxxxxx@xxxx.xx.xx>" [expired]

Additionaly, I have cloned a repository of a working Applikation (also Helm) and checked commit signatures by the key in question and then an older one by another key that has not yet expired. Output below, in all cases the regexp appears to match.

argocd@argocd-repo-server-5488979b5c-4hthf:/tmp/django-cms-test$ GNUPGHOME=/app/config/gpg/keys git verify-commit 6dc01abb8af39910de1ff964f9e9f80383a3fe37
gpg: WARNING: unsafe ownership on homedir '/app/config/gpg/keys'
gpg: Signature made Fri Jun 23 12:40:42 2023 UTC
gpg:                using RSA key B0230D84D45111B6F873A33CF5B4B1B588A59946
gpg: Good signature from "Xxxxxxxxx Xxxxxxx <xxxxxxxxx.xxxxxxx@xxxx.xx.xx>" [expired]
gpg: Note: This key has expired!
Primary key fingerprint: 7AF1 4C48 84F6 20AE D8A8  DF35 AA6E 994F 6CC1 64FC
     Subkey fingerprint: B023 0D84 D451 11B6 F873  A33C F5B4 B1B5 88A5 9946

argocd@argocd-repo-server-5488979b5c-4hthf:/tmp/django-cms-test$ GNUPGHOME=/app/config/gpg/keys git verify-commit 6dc01abb8af39910de1ff964f9e9f80383a3fe37 2>&1 | egrep '^gpg: ([a-zA-Z]+) signature from "([^"]+)" \[([a-zA-Z]+)\]$'
gpg: Good signature from "Xxxxxxxxx Xxxxxxx <xxxxxxxxx.xxxxxxx@xxxx.xx.xx>" [expired]
argocd@argocd-repo-server-5488979b5c-4hthf:/tmp/django-cms-test$ GNUPGHOME=/app/config/gpg/keys git verify-commit 80d49a2ca286ef648a5b965696f2fb75c8679766
gpg: WARNING: unsafe ownership on homedir '/app/config/gpg/keys'
gpg: Signature made Mon Jun 12 14:35:06 2023 UTC
gpg:                using RSA key EEF5E4CE822D4D88979B3E5BD7F67EBA7B6CC507
gpg: Good signature from "Xxxxx Xxxx (Xxx) <xxxxx.xxxx@xxxx.xx.xx>" [ultimate]

argocd@argocd-repo-server-5488979b5c-4hthf:/tmp/django-cms-test$ GNUPGHOME=/app/config/gpg/keys git verify-commit 80d49a2ca286ef648a5b965696f2fb75c8679766 2>&1 | egrep '^gpg: ([a-zA-Z]+) signature from "([^"]+)" \[([a-zA-Z]+)\]$'
gpg: Good signature from "Xxxxx Xxxx (Xxx) <xxxxx.xxxx@xxxx.xx.xx>" [ultimate]
jannfis commented 1 year ago

Thanks a lot for the additional information. I've taken the complete output of your git verify-commit as test data and the unit tests passes with it. This is weird.

Let me try to run the same tests in a container that resembles the 2.7.x image. Maybe there is a discrepancy in GnuPG or Git versions.

jannfis commented 1 year ago

OK, so even in the container the tests succeed.

One thing I forgot is that we run git verify-commit through a wrapper for determining whether we're operating against an annotated tag or a normal commit. So this might be a source of error, too.

Can you please run (again from within the repository server, in a local clone of your repository) the command:

$ GNUPGHOME=/app/config/gpg/keys /usr/local/bin/git-verify-wrapper.sh 976df8915507376005fa27fb1d90ca7be579125a

Also, it would be great to know if there's anything in the repository server's log file at the time the sync operation fails. The repo server pod should log the gpg commands it executes, along with any errors they may produce.

Links2004 commented 1 year ago

had a look by the changelog and this cached my eye: https://github.com/argoproj/argo-cd/pull/12797

I think there is a good change that it is related. since all versions tested by me which had the problem hat this change directly or via cherry-pick.

jannfis commented 1 year ago

Hm, yeah, that makes sense to further investigate.

What do you use as targetRevision for the failing app(s) in question? A branch name, a tag or a SHA revision?

Links2004 commented 1 year ago

in my case targetRevision is the master branch, no tags or other branches on the commit.

jannfis commented 1 year ago

So do you track HEAD or master in targetRevision then?

Links2004 commented 1 year ago

its master, here the CR:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kube-system-metallb
  namespace: argocd
spec:
  destination:
    namespace: kube-system
    server: https://kubernetes.default.svc
  project: kube-system
  source:
    path: kube-system/metallb
    repoURL: https://gitlab.XXXXXXXXX/k8s/gitops_k8s_local.git
    targetRevision: master
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
jannfis commented 1 year ago

Thanks a lot, I could reproduce the issue when the targetRevision is a specific branch name. Now that gives me some pointers to pursue further.

jannfis commented 1 year ago

So turns out we can't use the unresolved targetRevision (the change introduced by https://github.com/argoproj/argo-cd/pull/12797) when working with a branch name, because the repository will be in detached HEAD state after a checkout. The branch's name is unknown in that state.

I wrote a fix for this (and new e2e tests) which I'm currently testing. It has a dependency to new code in https://github.com/argoproj/pkg, though, so that has to go in first.

hapeho commented 1 year ago

Can you please run (again from within the repository server, in a local clone of your repository) the command:

$ GNUPGHOME=/app/config/gpg/keys /usr/local/bin/git-verify-wrapper.sh 976df8915507376005fa27fb1d90ca7be579125a
argocd@argocd-repo-server-5488979b5c-4hthf:/tmp/mastodon$ GNUPGHOME=/app/config/gpg/keys /usr/local/bin/git-verify-wrapper.sh 976df8915507376005fa27fb1d90ca7be579125a
gpg: WARNING: unsafe ownership on homedir '/app/config/gpg/keys'
gpg: Signature made Thu Jun 22 12:02:04 2023 UTC
gpg:                using RSA key B0230D84D45111B6F873A33CF5B4B1B588A59946
gpg: Good signature from "Xxxxxxxxx Xxxxxxx <xxxxxxxxx.xxxxxxx@xxxx.xx.xx>" [expired]
gpg: Note: This key has expired!
Primary key fingerprint: 7AF1 4C48 84F6 20AE D8A8  DF35 AA6E 994F 6CC1 64FC
Subkey fingerprint: B023 0D84 D451 11B6 F873  A33C F5B4 B1B5 88A5 9946
hapeho commented 1 year ago

Also, it would be great to know if there's anything in the repository server's log file at the time the sync operation fails. The repo server pod should log the gpg commands it executes, along with any errors they may produce.

These lines might be related:

time="2023-06-25T10:51:07Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2023-06-25T10:51:07Z" grpc.time_ms=15.639 span.kind=server system=grpc
time="2023-06-25T10:51:08Z" level=info msg="revision metadata cache miss: git@git.xxxxx.xx:k8s/mastodon/mastodon.git/976df8915507376005fa27fb1d90ca7be579125a"
time="2023-06-25T10:51:08Z" level=info msg="git fetch origin --tags --force --prune" dir=/tmp/_argocd-repo/44786264-1944-44ca-a3d4-9ddeac7bed17 execID=ccfe8
time="2023-06-25T10:51:08Z" level=info msg="revision metadata cache miss: git@git.xxxxx.xx:k8s/mastodon/mastodon.git/976df8915507376005fa27fb1d90ca7be579125a"
time="2023-06-25T10:51:08Z" level=info msg=Trace args="[git fetch origin --tags --force --prune]" dir=/tmp/_argocd-repo/44786264-1944-44ca-a3d4-9ddeac7bed17 operation_name="exec git" time_ms=418.207989
time="2023-06-25T10:51:08Z" level=info msg="git checkout --force 976df8915507376005fa27fb1d90ca7be579125a" dir=/tmp/_argocd-repo/44786264-1944-44ca-a3d4-9ddeac7bed17 execID=c5102
time="2023-06-25T10:51:08Z" level=info msg=Trace args="[git checkout --force 976df8915507376005fa27fb1d90ca7be579125a]" dir=/tmp/_argocd-repo/44786264-1944-44ca-a3d4-9ddeac7bed17 operation_name="exec git" time_ms=4.458752
time="2023-06-25T10:51:08Z" level=info msg="git clean -fdx" dir=/tmp/_argocd-repo/44786264-1944-44ca-a3d4-9ddeac7bed17 execID=c1b2e
time="2023-06-25T10:51:08Z" level=info msg=Trace args="[git clean -fdx]" dir=/tmp/_argocd-repo/44786264-1944-44ca-a3d4-9ddeac7bed17 operation_name="exec git" time_ms=3.421375
time="2023-06-25T10:51:08Z" level=info msg="git show -s --format=%an <%ae>|%at|%B 976df8915507376005fa27fb1d90ca7be579125a" dir=/tmp/_argocd-repo/44786264-1944-44ca-a3d4-9ddeac7bed17 execID=d6c96
time="2023-06-25T10:51:08Z" level=info msg="git show -s --format=%an <%ae>|%at|%B 976df8915507376005fa27fb1d90ca7be579125a" dir=/tmp/_argocd-repo/44786264-1944-44ca-a3d4-9ddeac7bed17 execID=9a0c5
time="2023-06-25T10:51:08Z" level=info msg=Trace args="[git show -s --format=%an <%ae>|%at|%B 976df8915507376005fa27fb1d90ca7be579125a]" dir=/tmp/_argocd-repo/44786264-1944-44ca-a3d4-9ddeac7bed17 operation_name="exec git" time_ms=4.70432
time="2023-06-25T10:51:08Z" level=info msg="git tag --points-at 976df8915507376005fa27fb1d90ca7be579125a" dir=/tmp/_argocd-repo/44786264-1944-44ca-a3d4-9ddeac7bed17 execID=f1640
time="2023-06-25T10:51:08Z" level=info msg=Trace args="[git show -s --format=%an <%ae>|%at|%B 976df8915507376005fa27fb1d90ca7be579125a]" dir=/tmp/_argocd-repo/44786264-1944-44ca-a3d4-9ddeac7bed17 operation_name="exec git" time_ms=4.864464
time="2023-06-25T10:51:08Z" level=info msg="git tag --points-at 976df8915507376005fa27fb1d90ca7be579125a" dir=/tmp/_argocd-repo/44786264-1944-44ca-a3d4-9ddeac7bed17 execID=2eaeb
time="2023-06-25T10:51:08Z" level=info msg=Trace args="[git tag --points-at 976df8915507376005fa27fb1d90ca7be579125a]" dir=/tmp/_argocd-repo/44786264-1944-44ca-a3d4-9ddeac7bed17 operation_name="exec git" time_ms=3.0119830000000003
time="2023-06-25T10:51:08Z" level=info msg="git-verify-wrapper.sh 976df8915507376005fa27fb1d90ca7be579125a" dir=/tmp/_argocd-repo/44786264-1944-44ca-a3d4-9ddeac7bed17 execID=24979
time="2023-06-25T10:51:08Z" level=info msg=Trace args="[git tag --points-at 976df8915507376005fa27fb1d90ca7be579125a]" dir=/tmp/_argocd-repo/44786264-1944-44ca-a3d4-9ddeac7bed17 operation_name="exec git" time_ms=3.8011310000000003
time="2023-06-25T10:51:08Z" level=info msg="git-verify-wrapper.sh 976df8915507376005fa27fb1d90ca7be579125a" dir=/tmp/_argocd-repo/44786264-1944-44ca-a3d4-9ddeac7bed17 execID=a33a4
time="2023-06-25T10:51:08Z" level=info msg=Trace args="[git-verify-wrapper.sh 976df8915507376005fa27fb1d90ca7be579125a]" dir=/tmp/_argocd-repo/44786264-1944-44ca-a3d4-9ddeac7bed17 operation_name="exec git-verify-wrapper.sh" time_ms=17.412307

time="2023-06-25T10:51:15Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ResolveRevision grpc.service=repository.RepoServerService grpc.start_time="2023-06-25T10:51:14Z" grpc.time_ms=350.932 span.kind=server system=grpc
time="2023-06-25T10:51:15Z" level=info msg="manifest cache hit: &ApplicationSource{RepoURL:git@git.xxxxx.xx:k8s/cluster/suture.git,Path:apps,TargetRevision:HEAD,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,}/8dd65d35adf6c9764b9bb06987a2794b6c73c893"
time="2023-06-25T10:51:15Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2023-06-25T10:51:15Z" grpc.time_ms=1.56 span.kind=server system=grp
time="2023-06-25T10:51:15Z" level=info msg="manifest cache hit: &ApplicationSource{RepoURL:git@git.xxxxx.xx:k8s/mastodon/mastodon.git,Path:.,TargetRevision:testing,Helm:&ApplicationSourceHelm{ValueFiles:[values.yaml],Parameters:[]HelmParameter{},ReleaseName:,Values:,FileParameters:[]HelmFileParameter{},Version:,PassCredentials:false,IgnoreMissingValueFiles:false,SkipCrds:false,},Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:,}/976df8915507376005fa27fb1d90ca7be579125a"