I would like to install the AVP by the ArgoCD Helm chart on my Amazon EKS cluster and make it working with AWS Secret Manager.
As somebody already mentioned in this issue https://github.com/argoproj-labs/argocd-vault-plugin/issues/474, the installation of AVP by ArgoCD Helm chart is not described properly in the official AVP documentation so here is the way how I did it, but at the end I was not able to make it working:
Create Secret in AWS Secret Manager, Policy, IRSA, ... - I'll skip it and put here just output of the kubectl showing the pre created ServiceAccount argocd-repo-server with role:
Unfortunately the <path:... was not replaced by the password from the AWS Secret Manager as you can see it by checking the k8s objects:
❯ kubectl get pod -n podinfo -o yaml | grep -B2 '<path:' ⎈ k01/argocd
env:
- name: PODINFO_UI_MESSAGE
value: <path:k01.k8s.mylabs.dev-PodinfoSecret#podinfo_secret_message>
❯ kubectl get pod -n podinfo2 -o yaml | grep '<path:' ⎈ k01/argocd
secret-test: <path:k01.k8s.mylabs.dev-PodinfoSecret#podinfo_secret_message>
secret-test: <path:k01.k8s.mylabs.dev-PodinfoSecret#podinfo_secret_message>
I'm adding here the ArgoCD logs and few more details which may be handy.
I was not able to find anything in the logs related to AVP :-( - it looks like AVP was not "properly" installed by the Helm Chart...
❯ kubectl stern -n argocd . ✘ INT 4s ⎈ k01/argocd
+ argo-cd-argocd-server-fb568f96d-4h4ng › server
+ argo-cd-argocd-applicationset-controller-588b76448-k2pdf › applicationset-controller
+ argo-cd-argocd-repo-server-54d8954bfd-mctbl › avp-helm
+ argo-cd-argocd-repo-server-54d8954bfd-mctbl › avp
+ argo-cd-argocd-repo-server-54d8954bfd-mctbl › avp-kustomize
+ argo-cd-argocd-repo-server-54d8954bfd-mctbl › repo-server
+ argo-cd-argocd-dex-server-56ffbf9fc8-ktz8n › dex-server
+ argo-cd-argocd-notifications-controller-7749b5c5b4-gb5wf › notifications-controller
+ argo-cd-argocd-application-controller-0 › application-controller
argo-cd-argocd-server-fb568f96d-4h4ng server time="2023-06-03T11:26:51Z" level=info msg="ArgoCD API Server is starting" built="2023-05-12T13:43:25Z" commit=cbee7e6011407ed2d1066c482db74e97e0cc6bdb namespace=argocd port=8080 version=v2.7.2+cbee7e6.dirty
argo-cd-argocd-server-fb568f96d-4h4ng server time="2023-06-03T11:26:51Z" level=info msg="Starting configmap/secret informers"
+ argo-cd-argocd-redis-86b6c546db-hrcfv › redis
argo-cd-argocd-server-fb568f96d-4h4ng server time="2023-06-03T11:26:51Z" level=info msg="Configmap/secret informer synced"
argo-cd-argocd-applicationset-controller-588b76448-k2pdf applicationset-controller time="2023-06-03T11:26:52Z" level=info msg="ArgoCD ApplicationSet Controller is starting" built="2023-05-12T13:43:25Z" commit=cbee7e6011407ed2d1066c482db74e97e0cc6bdb namespace=argocd version=v2.7.2+cbee7e6.dirty
argo-cd-argocd-applicationset-controller-588b76448-k2pdf applicationset-controller time="2023-06-03T11:26:53Z" level=info msg="Starting configmap/secret informers"
argo-cd-argocd-applicationset-controller-588b76448-k2pdf applicationset-controller time="2023-06-03T11:26:53Z" level=info msg="Configmap/secret informer synced"
argo-cd-argocd-server-fb568f96d-4h4ng server time="2023-06-03T11:26:51Z" level=info msg="Initialized server signature"
argo-cd-argocd-server-fb568f96d-4h4ng server time="2023-06-03T11:26:51Z" level=info msg="Initialized admin password"
argo-cd-argocd-applicationset-controller-588b76448-k2pdf applicationset-controller time="2023-06-03T11:26:53Z" level=info msg="Starting webhook server"
argo-cd-argocd-server-fb568f96d-4h4ng server time="2023-06-03T11:26:51Z" level=info msg="Starting configmap/secret informers"
argo-cd-argocd-server-fb568f96d-4h4ng server time="2023-06-03T11:26:51Z" level=info msg="configmap informer cancelled"
argo-cd-argocd-applicationset-controller-588b76448-k2pdf applicationset-controller time="2023-06-03T11:26:53Z" level=info msg="Starting manager"
argo-cd-argocd-applicationset-controller-588b76448-k2pdf applicationset-controller time="2023-06-03T11:27:52Z" level=debug msg="received create event from owning an application"
argo-cd-argocd-server-fb568f96d-4h4ng server time="2023-06-03T11:26:51Z" level=warning msg="Unable to parse updated settings: server.secretkey is missing"
argo-cd-argocd-server-fb568f96d-4h4ng server time="2023-06-03T11:26:51Z" level=info msg="Configmap/secret informer synced"
argo-cd-argocd-server-fb568f96d-4h4ng server time="2023-06-03T11:26:51Z" level=info msg="secrets informer cancelled"
argo-cd-argocd-server-fb568f96d-4h4ng server time="2023-06-03T11:26:51Z" level=info msg="Starting configmap/secret informers"
argo-cd-argocd-server-fb568f96d-4h4ng server time="2023-06-03T11:26:51Z" level=info msg="configmap informer cancelled"
argo-cd-argocd-server-fb568f96d-4h4ng server time="2023-06-03T11:26:51Z" level=info msg="Configmap/secret informer synced"
argo-cd-argocd-server-fb568f96d-4h4ng server time="2023-06-03T11:26:51Z" level=info msg="secrets informer cancelled"
argo-cd-argocd-applicationset-controller-588b76448-k2pdf applicationset-controller time="2023-06-03T11:27:55Z" level=debug msg="received update event from owning an application"
argo-cd-argocd-server-fb568f96d-4h4ng server time="2023-06-03T11:26:51Z" level=info msg="argocd v2.7.2+cbee7e6.dirty serving on port 8080 (url: , tls: false, namespace: argocd, sso: false)"
argo-cd-argocd-server-fb568f96d-4h4ng server time="2023-06-03T11:26:51Z" level=info msg="Enabled application namespace patterns: argocd"
argo-cd-argocd-server-fb568f96d-4h4ng server time="2023-06-03T11:26:51Z" level=info msg="0x4000c29f80 subscribed to settings updates"
argo-cd-argocd-applicationset-controller-588b76448-k2pdf applicationset-controller time="2023-06-03T11:27:55Z" level=debug msg="requeue: false caused by application podinfo\n"
argo-cd-argocd-server-fb568f96d-4h4ng server time="2023-06-03T11:26:51Z" level=info msg="Starting rbac config informer"
argo-cd-argocd-server-fb568f96d-4h4ng server time="2023-06-03T11:26:51Z" level=info msg="RBAC ConfigMap 'argocd-rbac-cm' added"
argo-cd-argocd-server-fb568f96d-4h4ng server time="2023-06-03T11:27:18Z" level=info msg="Notifying 1 settings subscribers: [0x4000c29f80]"
argo-cd-argocd-applicationset-controller-588b76448-k2pdf applicationset-controller time="2023-06-03T11:27:55Z" level=debug msg="received update event from owning an application"
argo-cd-argocd-applicationset-controller-588b76448-k2pdf applicationset-controller time="2023-06-03T11:27:55Z" level=debug msg="requeue: false caused by application podinfo\n"
...
...
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:30Z" level=debug msg="Refreshing app argocd/podinfo2 for change in cluster of object podinfo2/podinfo-5986769856-pnz46 of type v1/Pod"
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:30Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:30Z" level=info msg="No status changes. Skipping patch" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:30Z" level=info msg="Reconciliation completed" application=argocd/podinfo2 dest-name= dest-namespace=podinfo2 dest-server="https://kubernetes.default.svc" fields.level=0 time_ms=3
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:30Z" level=debug msg="Refreshing app argocd/podinfo2 for change in cluster of object podinfo2/podinfo-5986769856 of type apps/v1/ReplicaSet"
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:30Z" level=debug msg="Refreshing app argocd/podinfo2 for change in cluster of object podinfo2/podinfo-zknc6 of type discovery.k8s.io/v1/EndpointSlice"
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:30Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:30Z" level=info msg="No status changes. Skipping patch" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:30Z" level=info msg="Reconciliation completed" application=argocd/podinfo2 dest-name= dest-namespace=podinfo2 dest-server="https://kubernetes.default.svc" fields.level=0 time_ms=2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:30Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:30Z" level=info msg="No status changes. Skipping patch" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:30Z" level=info msg="Reconciliation completed" application=argocd/podinfo2 dest-name= dest-namespace=podinfo2 dest-server="https://kubernetes.default.svc" fields.level=0 time_ms=2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:30Z" level=debug msg="Refreshing app argocd/podinfo2 for change in cluster of object podinfo2/podinfo of type apps/v1/Deployment"
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:30Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:30Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: podinfo2)" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:30Z" level=debug msg="Generating Manifest for source {https://github.com/stefanprodan/podinfo.git kustomize nil &ApplicationSourceKustomize{NamePrefix:,NameSuffix:,Images:[],CommonLabels:map[string]string{},Version:,CommonAnnotations:map[string]string{secret-test: <path:k01.k8s.mylabs.dev-PodinfoSecret#podinfo_secret_message>,},ForceCommonLabels:false,ForceCommonAnnotations:false,Namespace:,CommonAnnotationsEnvsubst:false,Replicas:[]KustomizeReplica{},} nil nil } revision 073f1ec5aff930bd3411d33534e91cbe23302324"
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:30Z" level=info msg="getRepoObjs stats" application=argocd/podinfo2 build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=12 unmarshal_ms=12 version_ms=0
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:30Z" level=debug msg="Retrieved live manifests" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:30Z" level=info msg="Skipping auto-sync: application status is Synced" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:30Z" level=info msg="No status changes. Skipping patch" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:30Z" level=info msg="Reconciliation completed" application=argocd/podinfo2 dedup_ms=0 dest-name= dest-namespace=podinfo2 dest-server="https://kubernetes.default.svc" diff_ms=12 fields.level=1 git_ms=12 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=31
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:32Z" level=debug msg="Successfully saved info of 1 clusters"
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:33Z" level=debug msg="Refreshing app argocd/podinfo2 for change in cluster of object podinfo2/podinfo-5986769856 of type apps/v1/ReplicaSet"
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:33Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:33Z" level=info msg="No status changes. Skipping patch" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:33Z" level=info msg="Reconciliation completed" application=argocd/podinfo2 dest-name= dest-namespace=podinfo2 dest-server="https://kubernetes.default.svc" fields.level=0 time_ms=2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:33Z" level=debug msg="Refreshing app argocd/podinfo2 for change in cluster of object podinfo2/podinfo of type apps/v1/Deployment"
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:33Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:33Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: podinfo2)" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:33Z" level=debug msg="Generating Manifest for source {https://github.com/stefanprodan/podinfo.git kustomize nil &ApplicationSourceKustomize{NamePrefix:,NameSuffix:,Images:[],CommonLabels:map[string]string{},Version:,CommonAnnotations:map[string]string{secret-test: <path:k01.k8s.mylabs.dev-PodinfoSecret#podinfo_secret_message>,},ForceCommonLabels:false,ForceCommonAnnotations:false,Namespace:,CommonAnnotationsEnvsubst:false,Replicas:[]KustomizeReplica{},} nil nil } revision 073f1ec5aff930bd3411d33534e91cbe23302324"
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:33Z" level=info msg="getRepoObjs stats" application=argocd/podinfo2 build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=9 unmarshal_ms=9 version_ms=0
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:33Z" level=debug msg="Retrieved live manifests" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:33Z" level=info msg="Skipping auto-sync: application status is Synced" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:33Z" level=info msg="Updated health status: Progressing -> Healthy" application=podinfo2 dest-namespace=podinfo2 dest-server="https://kubernetes.default.svc" reason=ResourceUpdated type=Normal
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:33Z" level=info msg="Update successful" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:33Z" level=info msg="Reconciliation completed" application=argocd/podinfo2 dedup_ms=0 dest-name= dest-namespace=podinfo2 dest-server="https://kubernetes.default.svc" diff_ms=14 fields.level=1 git_ms=9 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=65
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:33Z" level=debug msg="Refreshing app argocd/podinfo2 for change in cluster of object podinfo2/podinfo of type autoscaling/v2/HorizontalPodAutoscaler"
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:33Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:33Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: podinfo2)" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:33Z" level=debug msg="Generating Manifest for source {https://github.com/stefanprodan/podinfo.git kustomize nil &ApplicationSourceKustomize{NamePrefix:,NameSuffix:,Images:[],CommonLabels:map[string]string{},Version:,CommonAnnotations:map[string]string{secret-test: <path:k01.k8s.mylabs.dev-PodinfoSecret#podinfo_secret_message>,},ForceCommonLabels:false,ForceCommonAnnotations:false,Namespace:,CommonAnnotationsEnvsubst:false,Replicas:[]KustomizeReplica{},} nil nil } revision 073f1ec5aff930bd3411d33534e91cbe23302324"
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:33Z" level=info msg="getRepoObjs stats" application=argocd/podinfo2 build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=9 unmarshal_ms=9 version_ms=0
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:33Z" level=debug msg="Retrieved live manifests" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:33Z" level=info msg="Skipping auto-sync: application status is Synced" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:33Z" level=info msg="Update successful" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:33Z" level=info msg="Reconciliation completed" application=argocd/podinfo2 dedup_ms=0 dest-name= dest-namespace=podinfo2 dest-server="https://kubernetes.default.svc" diff_ms=11 fields.level=1 git_ms=9 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=36
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:42Z" level=debug msg="Successfully saved info of 1 clusters"
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:48Z" level=debug msg="Refreshing app argocd/podinfo2 for change in cluster of object podinfo2/podinfo of type autoscaling/v2/HorizontalPodAutoscaler"
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:48Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:48Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: podinfo2)" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:48Z" level=debug msg="Generating Manifest for source {https://github.com/stefanprodan/podinfo.git kustomize nil &ApplicationSourceKustomize{NamePrefix:,NameSuffix:,Images:[],CommonLabels:map[string]string{},Version:,CommonAnnotations:map[string]string{secret-test: <path:k01.k8s.mylabs.dev-PodinfoSecret#podinfo_secret_message>,},ForceCommonLabels:false,ForceCommonAnnotations:false,Namespace:,CommonAnnotationsEnvsubst:false,Replicas:[]KustomizeReplica{},} nil nil } revision 073f1ec5aff930bd3411d33534e91cbe23302324"
argo-cd-argocd-repo-server-54d8954bfd-mctbl repo-server time="2023-06-03T11:28:48Z" level=debug msg="getting manifests cache" appName=podinfo2 appSrc="{\"appSrc\":{\"repoURL\":\"\",\"path\":\"kustomize\",\"kustomize\":{\"commonAnnotations\":{\"secret-test\":\"\\u003cpath:k01.k8s.mylabs.dev-PodinfoSecret#podinfo_secret_message\\u003e\"}}},\"srcRefs\":{}}" clusterInfo="1.25+|acme.cert-manager.io/v1,acme.cert-manager.io/v1/Challenge,acme.cert-manager.io/v1/Order,admissionregistration.k8s.io/v1,admissionregistration.k8s.io/v1/MutatingWebhookConfiguration,admissionregistration.k8s.io/v1/ValidatingWebhookConfiguration,apiextensions.k8s.io/v1,apiextensions.k8s.io/v1/CustomResourceDefinition,apiregistration.k8s.io/v1,apiregistration.k8s.io/v1/APIService,apps/v1,apps/v1/ControllerRevision,apps/v1/DaemonSet,apps/v1/Deployment,apps/v1/ReplicaSet,apps/v1/StatefulSet,argoproj.io/v1alpha1,argoproj.io/v1alpha1/AppProject,argoproj.io/v1alpha1/Application,argoproj.io/v1alpha1/ApplicationSet,autoscaling/v1,autoscaling/v1/HorizontalPodAutoscaler,autoscaling/v2,autoscaling/v2/HorizontalPodAutoscaler,autoscaling/v2beta2,autoscaling/v2beta2/HorizontalPodAutoscaler,batch/v1,batch/v1/CronJob,batch/v1/Job,cert-manager.io/v1,cert-manager.io/v1/Certificate,cert-manager.io/v1/CertificateRequest,cert-manager.io/v1/ClusterIssuer,cert-manager.io/v1/Issuer,certificates.k8s.io/v1,certificates.k8s.io/v1/CertificateSigningRequest,coordination.k8s.io/v1,coordination.k8s.io/v1/Lease,crd.k8s.amazonaws.com/v1alpha1,crd.k8s.amazonaws.com/v1alpha1/ENIConfig,discovery.k8s.io/v1,discovery.k8s.io/v1/EndpointSlice,events.k8s.io/v1,events.k8s.io/v1/Event,flowcontrol.apiserver.k8s.io/v1beta1,flowcontrol.apiserver.k8s.io/v1beta1/FlowSchema,flowcontrol.apiserver.k8s.io/v1beta1/PriorityLevelConfiguration,flowcontrol.apiserver.k8s.io/v1beta2,flowcontrol.apiserver.k8s.io/v1beta2/FlowSchema,flowcontrol.apiserver.k8s.io/v1beta2/PriorityLevelConfiguration,forecastle.stakater.com/v1alpha1,forecastle.stakater.com/v1alpha1/ForecastleApp,karpenter.k8s.aws/v1alpha1,karpenter.k8s.aws/v1alpha1/AWSNodeTemplate,karpenter.sh/v1alpha5,karpenter.sh/v1alpha5/Provisioner,monitoring.coreos.com/v1,monitoring.coreos.com/v1/Alertmanager,monitoring.coreos.com/v1/PodMonitor,monitoring.coreos.com/v1/Probe,monitoring.coreos.com/v1/Prometheus,monitoring.coreos.com/v1/PrometheusRule,monitoring.coreos.com/v1/ServiceMonitor,monitoring.coreos.com/v1/ThanosRuler,monitoring.coreos.com/v1alpha1,monitoring.coreos.com/v1alpha1/AlertmanagerConfig,networking.k8s.io/v1,networking.k8s.io/v1/Ingress,networking.k8s.io/v1/IngressClass,networking.k8s.io/v1/NetworkPolicy,node.k8s.io/v1,node.k8s.io/v1/RuntimeClass,policy/v1,policy/v1/PodDisruptionBudget,rbac.authorization.k8s.io/v1,rbac.authorization.k8s.io/v1/ClusterRole,rbac.authorization.k8s.io/v1/ClusterRoleBinding,rbac.authorization.k8s.io/v1/Role,rbac.authorization.k8s.io/v1/RoleBinding,scheduling.k8s.io/v1,scheduling.k8s.io/v1/PriorityClass,storage.k8s.io/v1,storage.k8s.io/v1/CSIDriver,storage.k8s.io/v1/CSINode,storage.k8s.io/v1/CSIStorageCapacity,storage.k8s.io/v1/StorageClass,storage.k8s.io/v1/VolumeAttachment,storage.k8s.io/v1beta1,storage.k8s.io/v1beta1/CSIStorageCapacity,v1,v1/ConfigMap,v1/Endpoints,v1/Event,v1/LimitRange,v1/Namespace,v1/Node,v1/PersistentVolume,v1/PersistentVolumeClaim,v1/Pod,v1/PodTemplate,v1/ReplicationController,v1/ResourceQuota,v1/Secret,v1/Service,v1/ServiceAccount,vpcresources.k8s.aws/v1beta1,vpcresources.k8s.aws/v1beta1/SecurityGroupPolicy" namespace=podinfo2 reason="GenerateManifest API call" revision=073f1ec5aff930bd3411d33534e91cbe23302324 trackingKey=argocd.argoproj.io/instance
argo-cd-argocd-repo-server-54d8954bfd-mctbl repo-server time="2023-06-03T11:28:48Z" level=info msg="manifest cache hit: &ApplicationSource{RepoURL:https://github.com/stefanprodan/podinfo.git,Path:kustomize,TargetRevision:,Helm:nil,Kustomize:&ApplicationSourceKustomize{NamePrefix:,NameSuffix:,Images:[],CommonLabels:map[string]string{},Version:,CommonAnnotations:map[string]string{secret-test: <path:k01.k8s.mylabs.dev-PodinfoSecret#podinfo_secret_message>,},ForceCommonLabels:false,ForceCommonAnnotations:false,Namespace:,CommonAnnotationsEnvsubst:false,Replicas:[]KustomizeReplica{},},Directory:nil,Plugin:nil,Chart:,Ref:,}/073f1ec5aff930bd3411d33534e91cbe23302324"
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:48Z" level=info msg="getRepoObjs stats" application=argocd/podinfo2 build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=11 unmarshal_ms=10 version_ms=0
argo-cd-argocd-repo-server-54d8954bfd-mctbl repo-server time="2023-06-03T11:28:48Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2023-06-03T11:28:48Z" grpc.time_ms=1.513 span.kind=server system=grpc
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:48Z" level=debug msg="Retrieved live manifests" application=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:48Z" level=info msg="Skipping auto-sync: application status is Synced" application=argocd/podinfo2
argo-cd-argocd-notifications-controller-7749b5c5b4-gb5wf notifications-controller time="2023-06-03T11:28:48Z" level=info msg="Start processing" resource=argocd/podinfo2
argo-cd-argocd-applicationset-controller-588b76448-k2pdf applicationset-controller time="2023-06-03T11:28:48Z" level=debug msg="received update event from owning an application"
argo-cd-argocd-applicationset-controller-588b76448-k2pdf applicationset-controller time="2023-06-03T11:28:48Z" level=debug msg="requeue: false caused by application podinfo2\n"
argo-cd-argocd-notifications-controller-7749b5c5b4-gb5wf notifications-controller time="2023-06-03T11:28:48Z" level=debug msg="SyncStatus up-to-date (FinishedAt=2023-06-03 11:28:03 +0000 UTC, ReconciledAt=2023-06-03 11:28:03 +0000 UTC, Observed=0001-01-01 00:00:00 +0000 UTC" app=podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:48Z" level=info msg="Update successful" application=argocd/podinfo2
argo-cd-argocd-notifications-controller-7749b5c5b4-gb5wf notifications-controller time="2023-06-03T11:28:48Z" level=info msg="Processing completed" resource=argocd/podinfo2
argo-cd-argocd-application-controller-0 application-controller time="2023-06-03T11:28:48Z" level=info msg="Reconciliation completed" application=argocd/podinfo2 dedup_ms=0 dest-name= dest-namespace=podinfo2 dest-server="https://kubernetes.default.svc" diff_ms=11 fields.level=1 git_ms=11 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=41
Describe the bug
I would like to install the AVP by the ArgoCD Helm chart on my Amazon EKS cluster and make it working with AWS Secret Manager.
As somebody already mentioned in this issue https://github.com/argoproj-labs/argocd-vault-plugin/issues/474, the installation of AVP by ArgoCD Helm chart is not described properly in the official AVP documentation so here is the way how I did it, but at the end I was not able to make it working:
Create Secret in AWS Secret Manager, Policy, IRSA, ... - I'll skip it and put here just output of the
kubectl
showing the pre created ServiceAccountargocd-repo-server
with role:AVP + ArgoCD Helm Installation (there is probably something "wrong"):
The I installed two apps where I used
<path:
with secret which should be taken from the AWS Secret Manager:First is using the secret as a helm parameter - the secret should be injected into the installed helm chart.
Second is using
kustomize
and should inject the secret into the annotation.Unfortunately the
<path:...
was not replaced by the password from the AWS Secret Manager as you can see it by checking the k8s objects:I'm adding here the ArgoCD logs and few more details which may be handy. I was not able to find anything in the logs related to AVP :-( - it looks like AVP was not "properly" installed by the Helm Chart...
Thank you...