search

jkroepke / helm-secrets

A helm plugin that help manage secrets with Git workflow and store them anywhere
https://github.com/jkroepke/helm-secrets/wiki
Apache License 2.0
1.51k stars 128 forks source link

Problem with ArgoCD Multi-Source Application Support #349

Closed jeroen-s closed 1 year ago

jeroen-s commented 1 year ago

Current Behavior

I am using your latest instructions here to the letter: https://github.com/jkroepke/helm-secrets/wiki/ArgoCD-Integration#multi-source-application-support-beta

I am seeing encrypted values in my Kubernetes secrets and from the repo-server log:

download-tools sh: can't create /usr/local/sbin/helm: nonexistent directory

Might there be an error on this line?

printf '#!/usr/bin/env sh\nif [ "${HELM_SECRETS_WRAPPER_ENABLED}" = "true" ]; then exec %1$s secrets "$@"; fi\nexec %1$s "$@"' "helm" "${HELM_SECRETS_HELM_PATH}" >"/usr/local/sbin/helm" && chmod +x "/custom-tools/helm"

Expected Behavior

No response

Steps To Reproduce

No response

Environment

Anything else?

No response

jkroepke commented 1 year ago

Yes, you are correct: Could you try this line?

printf '#!/usr/bin/env sh\nif [ "${HELM_SECRETS_WRAPPER_ENABLED}" = "true" ]; then exec %1$s secrets "$@"; fi\nexec %1$s "$@"' "helm" "${HELM_SECRETS_HELM_PATH}" >"/custom-tools/helm" && chmod +x "/custom-tools/helm"
jeroen-s commented 1 year ago

Will try this out tomorrow morning and report back. Thanks for the quick response!

tunguyen9889 commented 1 year ago

Yes, you are correct: Could you try this line?

printf '#!/usr/bin/env sh\nif [ "${HELM_SECRETS_WRAPPER_ENABLED}" = "true" ]; then exec %1$s secrets "$@"; fi\nexec %1$s "$@"' "helm" "${HELM_SECRETS_HELM_PATH}" >"/custom-tools/helm" && chmod +x "/custom-tools/helm"

I tried this and got the error:

sh: %1$s secrets "$@"; fi\nexec %1$s "$@": invalid format

The helm template in repo-server also broken and failed to generate the manifests:

time="2023-03-01T18:24:31Z" level=error msg="`helm template . --name-template infra-root --namespace argo --kube-version 1.25 --api-versions admissionregistration.k8s.io/v1 --api-versions admissionregistration.k8s.io/v1/MutatingWebhookConfiguration --api-versions admissionregistration.k8s.io/v1/ValidatingWebhookConfiguration --api-versions apiextensions.k8s.io/v1 --api-versions apiextensions.k8s.io/v1/CustomResourceDefinition --api-versions apiregistration.k8s.io/v1 --api-versions apiregistration.k8s.io/v1/APIService --api-versions apps/v1 --api-versions apps/v1/ControllerRevision --api-versions apps/v1/DaemonSet --api-versions apps/v1/Deployment --api-versions apps/v1/ReplicaSet --api-versions apps/v1/StatefulSet --api-versions argoproj.io/v1alpha1 --api-versions argoproj.io/v1alpha1/AppProject --api-versions argoproj.io/v1alpha1/Application --api-versions argoproj.io/v1alpha1/ApplicationSet --api-versions auto.gke.io/v1 --api-versions auto.gke.io/v1/AllowlistedV2Workload --api-versions auto.gke.io/v1/AllowlistedWorkload --api-versions auto.gke.io/v1alpha1 --api-versions auto.gke.io/v1alpha1/AllowlistedWorkload --api-versions autoscaling/v1 --api-versions autoscaling/v1/HorizontalPodAutoscaler --api-versions autoscaling/v2 --api-versions autoscaling/v2/HorizontalPodAutoscaler --api-versions autoscaling/v2beta2 --api-versions autoscaling/v2beta2/HorizontalPodAutoscaler --api-versions batch/v1 --api-versions batch/v1/CronJob --api-versions batch/v1/Job --api-versions certificates.k8s.io/v1 --api-versions certificates.k8s.io/v1/CertificateSigningRequest --api-versions cloud.google.com/v1 --api-versions cloud.google.com/v1/BackendConfig --api-versions cloud.google.com/v1beta1 --api-versions cloud.google.com/v1beta1/BackendConfig --api-versions coordination.k8s.io/v1 --api-versions coordination.k8s.io/v1/Lease --api-versions discovery.k8s.io/v1 --api-versions discovery.k8s.io/v1/EndpointSlice --api-versions events.k8s.io/v1 --api-versions events.k8s.io/v1/Event --api-versions external-secrets.io/v1alpha1 --api-versions external-secrets.io/v1alpha1/ClusterSecretStore --api-versions external-secrets.io/v1alpha1/ExternalSecret --api-versions external-secrets.io/v1alpha1/PushSecret --api-versions external-secrets.io/v1alpha1/SecretStore --api-versions external-secrets.io/v1beta1 --api-versions external-secrets.io/v1beta1/ClusterExternalSecret --api-versions external-secrets.io/v1beta1/ClusterSecretStore --api-versions external-secrets.io/v1beta1/ExternalSecret --api-versions external-secrets.io/v1beta1/SecretStore --api-versions flowcontrol.apiserver.k8s.io/v1beta1 --api-versions flowcontrol.apiserver.k8s.io/v1beta1/FlowSchema --api-versions flowcontrol.apiserver.k8s.io/v1beta1/PriorityLevelConfiguration --api-versions flowcontrol.apiserver.k8s.io/v1beta2 --api-versions flowcontrol.apiserver.k8s.io/v1beta2/FlowSchema --api-versions flowcontrol.apiserver.k8s.io/v1beta2/PriorityLevelConfiguration --api-versions generators.external-secrets.io/v1alpha1 --api-versions generators.external-secrets.io/v1alpha1/ACRAccessToken --api-versions generators.external-secrets.io/v1alpha1/ECRAuthorizationToken --api-versions generators.external-secrets.io/v1alpha1/Fake --api-versions generators.external-secrets.io/v1alpha1/GCRAccessToken --api-versions generators.external-secrets.io/v1alpha1/Password --api-versions hub.gke.io/v1 --api-versions hub.gke.io/v1/Membership --api-versions internal.autoscaling.gke.io/v1alpha1 --api-versions internal.autoscaling.gke.io/v1alpha1/CapacityRequest --api-versions networking.gke.io/v1 --api-versions networking.gke.io/v1/ManagedCertificate --api-versions networking.gke.io/v1/ServiceAttachment --api-versions networking.gke.io/v1beta1 --api-versions networking.gke.io/v1beta1/FrontendConfig --api-versions networking.gke.io/v1beta1/ManagedCertificate --api-versions networking.gke.io/v1beta1/ServiceAttachment --api-versions networking.gke.io/v1beta1/ServiceNetworkEndpointGroup --api-versions networking.gke.io/v1beta2 --api-versions networking.gke.io/v1beta2/ManagedCertificate --api-versions networking.k8s.io/v1 --api-versions networking.k8s.io/v1/Ingress --api-versions networking.k8s.io/v1/IngressClass --api-versions networking.k8s.io/v1/NetworkPolicy --api-versions node.k8s.io/v1 --api-versions node.k8s.io/v1/RuntimeClass --api-versions nodemanagement.gke.io/v1alpha1 --api-versions nodemanagement.gke.io/v1alpha1/UpdateInfo --api-versions policy/v1 --api-versions policy/v1/PodDisruptionBudget --api-versions rbac.authorization.k8s.io/v1 --api-versions rbac.authorization.k8s.io/v1/ClusterRole --api-versions rbac.authorization.k8s.io/v1/ClusterRoleBinding --api-versions rbac.authorization.k8s.io/v1/Role --api-versions rbac.authorization.k8s.io/v1/RoleBinding --api-versions scheduling.k8s.io/v1 --api-versions scheduling.k8s.io/v1/PriorityClass --api-versions snapshot.storage.k8s.io/v1 --api-versions snapshot.storage.k8s.io/v1/VolumeSnapshot --api-versions snapshot.storage.k8s.io/v1/VolumeSnapshotClass --api-versions snapshot.storage.k8s.io/v1/VolumeSnapshotContent --api-versions snapshot.storage.k8s.io/v1beta1 --api-versions snapshot.storage.k8s.io/v1beta1/VolumeSnapshot --api-versions snapshot.storage.k8s.io/v1beta1/VolumeSnapshotClass --api-versions snapshot.storage.k8s.io/v1beta1/VolumeSnapshotContent --api-versions storage.k8s.io/v1 --api-versions storage.k8s.io/v1/CSIDriver --api-versions storage.k8s.io/v1/CSINode --api-versions storage.k8s.io/v1/CSIStorageCapacity --api-versions storage.k8s.io/v1/StorageClass --api-versions storage.k8s.io/v1/VolumeAttachment --api-versions storage.k8s.io/v1beta1 --api-versions storage.k8s.io/v1beta1/CSIStorageCapacity --api-versions v1 --api-versions v1/ConfigMap --api-versions v1/Endpoints --api-versions v1/Event --api-versions v1/LimitRange --api-versions v1/Namespace --api-versions v1/Node --api-versions v1/PersistentVolume --api-versions v1/PersistentVolumeClaim --api-versions v1/Pod --api-versions v1/PodTemplate --api-versions v1/ReplicationController --api-versions v1/ResourceQuota --api-versions v1/Secret --api-versions v1/Service --api-versions v1/ServiceAccount --api-versions warden.gke.io/v1 --api-versions warden.gke.io/v1/Audit --include-crds` failed exit status 2: /usr/local/sbin/helm: 2: Syntax error: end of file unexpected (expecting \"fi\")" execID=087fc
2
jkroepke commented 1 year ago

It seems like printf have an different behaivor inside the argocd container.

I have to check this... this should be the correct one.

jeroen-s commented 1 year ago

I have replaced the last two lines of the script with this:

cat > /custom-tools/helm <<'EOF'
if [ "${HELM_SECRETS_WRAPPER_ENABLED}" = "true" ]; then
    exec "${HELM_SECRETS_HELM_PATH:-${HELM_BIN:-"helm"}}" secrets "$@"
else
    exec "${HELM_SECRETS_HELM_PATH:-${HELM_BIN:-"helm"}}" "$@"
fi
EOF

chmod +x /custom-tools/*

I verified in the repo-server container that /custom-tools/helm is executable and looks like the file you referenced.

No errors in logs but still encrypted values in my Kubernetes secrets.

jkroepke commented 1 year ago

No errors in logs but still encrypted values in my Kubernetes secrets.

Can you confirm, that HELM_SECRETS_WRAPPER_ENABLED is set to true?

jeroen-s commented 1 year ago

from inside the container:

argocd@argocd-repo-server-75bb99-wrj9f:~$ echo $HELM_SECRETS_WRAPPER_ENABLED
true
jeroen-s commented 1 year ago

So there's regular helm at /usr/local/bin/helm and the above wrapper script at /usr/local/sbin/helm. How does repo-server know to run the wrapper script instead of the regular helm executable?

jkroepke commented 1 year ago

if you run which helm, it should result /usr/local/sbin/helm, because in PATH variable, /usr/local/sbin/ is before /usr/local/bin/

jeroen-s commented 1 year ago

after a hard refresh in ArgoCD I'm getting this:

ComparisonError rpc error: code = Unknown desc = fork/exec /usr/local/sbin/helm: exec format error

Everything seems fine from inside the container:

argocd@argocd-repo-server-5d4fc6dfbf-gcpt5:~$ helm --help

helm-secrets is a helm plugin for decrypt encrypted helm value files on the fly.
...
jkroepke commented 1 year ago

after a hard refresh in ArgoCD I'm getting this.

Ah, yeah.. ArgoCD caching manifest and using the git revision as cache key. Any modifications outside the git repository requires a "hard refresh".

The shebang line (#!) is missing.

cat > /custom-tools/helm <<'EOF'
#!/usr/bin/env sh
if [ "${HELM_SECRETS_WRAPPER_ENABLED}" = "true" ]; then
    exec "${HELM_SECRETS_HELM_PATH:-${HELM_BIN:-"helm"}}" secrets "$@"
else
    exec "${HELM_SECRETS_HELM_PATH:-${HELM_BIN:-"helm"}}" "$@"
fi
EOF

chmod +x /custom-tools/*
jeroen-s commented 1 year ago

Please ignore my last comment (I forgot the shebang in the wrapper script).

I am now seeing the following. my app:

...
      helm:
        valueFiles:
          - $values/configuration/actions-runner-deployment-io/values.tst.yaml
          - $values/configuration/actions-runner-deployment-io/secrets.tst.enc.yaml

Error:

[helm-secrets] Values filepath '.configuration/actions-runner-deployment-io/values.tst.yaml' is an absolute path. Absolute paths are not allowed. [helm-secrets] File does not exist: .configuration/actions-runner-deployment-io/values.tst.yaml Error: plugin "secrets"

jkroepke commented 1 year ago

is an absolute path. Absolute paths are not allowed.

Context: https://github.com/jkroepke/helm-secrets/wiki/Security-in-shared-environments

The ArgoCD documentation here recommends to set HELM_SECRETS_VALUES_ALLOW_ABSOLUTE_PATH=false.

Set the env var to true should resolve the issue.

Did you strip the error message? is the Values filepath an absolute path?

jeroen-s commented 1 year ago

not sure what you mean, the path is relative to the repo root. the files are in my-repo/configuration/actions-runner-deployment-io/

jeroen-s commented 1 year ago

I should add that this setup was working fine with an older helm-secrets config, the only difference was the location of secrets.tst.enc.yaml

...
      helm:
        valueFiles:
          - $values/configuration/actions-runner-deployment-io/values.tst.yaml
          - secrets://config/secrets.tst.enc.yaml
repoServer:
  serviceAccount:
    create: true
    name: "argocd-repo-server"
    automountServiceAccountToken: true
  env:
    - name: HELM_PLUGINS
      value: /custom-tools/helm-plugins/
    - name: HELM_SECRETS_SOPS_PATH
      value: /custom-tools/sops
    - name: HELM_SECRETS_VALS_PATH
      value: /custom-tools/vals
    - name: HELM_SECRETS_KUBECTL_PATH
      value: /custom-tools/kubectl
    - name: HELM_SECRETS_CURL_PATH
      value: /custom-tools/curl
    # https://github.com/jkroepke/helm-secrets/wiki/Security-in-shared-environments
    - name: HELM_SECRETS_VALUES_ALLOW_SYMLINKS
      value: "false"
    - name: HELM_SECRETS_VALUES_ALLOW_ABSOLUTE_PATH
      value: "false"
    - name: HELM_SECRETS_VALUES_ALLOW_PATH_TRAVERSAL
      value: "false"
  volumes:
    - name: custom-tools
      emptyDir: {}
  volumeMounts:
    - mountPath: /custom-tools
      name: custom-tools
  initContainers:
    - name: download-tools
      image: alpine:latest
      command: [sh, -ec]
      env:
        - name: HELM_SECRETS_VERSION
          value: "3.12.0"
        - name: KUBECTL_VERSION
          value: "1.24.3"
        - name: VALS_VERSION
          value: "0.18.0"
        - name: SOPS_VERSION
          value: "3.7.3"
      args:
        - |
          mkdir -p /custom-tools/helm-plugins
          wget -qO- https://github.com/jkroepke/helm-secrets/releases/download/v${HELM_SECRETS_VERSION}/helm-secrets.tar.gz | tar -C /custom-tools/helm-plugins -xzf-;

          wget -qO /custom-tools/sops https://github.com/mozilla/sops/releases/download/v${SOPS_VERSION}/sops-v${SOPS_VERSION}.linux
          wget -qO /custom-tools/kubectl https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/amd64/kubectl

          wget -qO- https://github.com/variantdev/vals/releases/download/v${VALS_VERSION}/vals_${VALS_VERSION}_linux_amd64.tar.gz | tar -xzf- -C /custom-tools/ vals;

          # helm secrets wrapper mode installation (optional)
          # RUN printf '#!/usr/bin/env sh\nexec %s secrets "$@"' "${HELM_SECRETS_HELM_PATH}" >"/usr/local/sbin/helm" && chmod +x "/custom-tools/helm"

          chmod +x /custom-tools/*
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools
jkroepke commented 1 year ago

[helm-secrets] Values filepath '.configuration/actions-runner-deployment-io/values.tst.yaml' is an absolute path. Absolute paths are not allowed. [helm-secrets] File does not exist: .configuration/actions-runner-deployment-io/values.tst.yaml Error: plugin "secrets"

this is the complete error messages?

The error Values filepath '...' is an absolute path.

Appears only, if the path begins with an slash or contains a colon.

https://github.com/jkroepke/helm-secrets/blob/e6e2a10b152b2d98b5cb7cc2d093df8811a2bb19/scripts/lib/file.sh#L48-L50

jeroen-s commented 1 year ago

Here's the complete, unedited error message:

ComparisonError rpc error: code = Unknown desc = `helm template . --name-template actions-runner-deployment-io --namespace actions-runner --kube-version 1.23 --values .configuration/actions-runner-deployment-io/values.tst.yaml --values .configuration/actions-runner-deployment-io/secrets.tst.enc.yaml --api-versions acme.cert-manager.io/v1 --api-versions acme.cert-manager.io/v1/Challenge --api-versions acme.cert-manager.io/v1/Order --api-versions actions.summerwind.dev/v1alpha1 --api-versions actions.summerwind.dev/v1alpha1/HorizontalRunnerAutoscaler --api-versions actions.summerwind.dev/v1alpha1/Runner --api-versions actions.summerwind.dev/v1alpha1/RunnerDeployment --api-versions actions.summerwind.dev/v1alpha1/RunnerReplicaSet --api-versions actions.summerwind.dev/v1alpha1/RunnerSet --api-versions admissionregistration.k8s.io/v1 --api-versions admissionregistration.k8s.io/v1/MutatingWebhookConfiguration --api-versions admissionregistration.k8s.io/v1/ValidatingWebhookConfiguration --api-versions apiextensions.k8s.io/v1 --api-versions apiextensions.k8s.io/v1/CustomResourceDefinition --api-versions apiregistration.k8s.io/v1 --api-versions apiregistration.k8s.io/v1/APIService --api-versions apps/v1 --api-versions apps/v1/ControllerRevision --api-versions apps/v1/DaemonSet --api-versions apps/v1/Deployment --api-versions apps/v1/ReplicaSet --api-versions apps/v1/StatefulSet --api-versions argoproj.io/v1alpha1 --api-versions argoproj.io/v1alpha1/AppProject --api-versions argoproj.io/v1alpha1/Application --api-versions argoproj.io/v1alpha1/ApplicationSet --api-versions argoproj.io/v1alpha1/ClusterWorkflowTemplate --api-versions argoproj.io/v1alpha1/CronWorkflow --api-versions argoproj.io/v1alpha1/Workflow --api-versions argoproj.io/v1alpha1/WorkflowArtifactGCTask --api-versions argoproj.io/v1alpha1/WorkflowEventBinding --api-versions argoproj.io/v1alpha1/WorkflowTaskResult --api-versions argoproj.io/v1alpha1/WorkflowTaskSet --api-versions argoproj.io/v1alpha1/WorkflowTemplate --api-versions autoscaling/v1 --api-versions autoscaling/v1/HorizontalPodAutoscaler --api-versions autoscaling/v2 --api-versions autoscaling/v2/HorizontalPodAutoscaler --api-versions autoscaling/v2beta1 --api-versions autoscaling/v2beta1/HorizontalPodAutoscaler --api-versions autoscaling/v2beta2 --api-versions autoscaling/v2beta2/HorizontalPodAutoscaler --api-versions batch/v1 --api-versions batch/v1/CronJob --api-versions batch/v1/Job --api-versions batch/v1beta1 --api-versions batch/v1beta1/CronJob --api-versions cert-manager.io/v1 --api-versions cert-manager.io/v1/Certificate --api-versions cert-manager.io/v1/CertificateRequest --api-versions cert-manager.io/v1/ClusterIssuer --api-versions cert-manager.io/v1/Issuer --api-versions certificates.k8s.io/v1 --api-versions certificates.k8s.io/v1/CertificateSigningRequest --api-versions configuration.konghq.com/v1 --api-versions configuration.konghq.com/v1/KongClusterPlugin --api-versions configuration.konghq.com/v1/KongConsumer --api-versions configuration.konghq.com/v1/KongIngress --api-versions configuration.konghq.com/v1/KongPlugin --api-versions configuration.konghq.com/v1alpha1 --api-versions configuration.konghq.com/v1alpha1/IngressClassParameters --api-versions configuration.konghq.com/v1beta1 --api-versions configuration.konghq.com/v1beta1/TCPIngress --api-versions configuration.konghq.com/v1beta1/UDPIngress --api-versions coordination.k8s.io/v1 --api-versions coordination.k8s.io/v1/Lease --api-versions crd.k8s.amazonaws.com/v1alpha1 --api-versions crd.k8s.amazonaws.com/v1alpha1/ENIConfig --api-versions discovery.k8s.io/v1 --api-versions discovery.k8s.io/v1/EndpointSlice --api-versions discovery.k8s.io/v1beta1 --api-versions discovery.k8s.io/v1beta1/EndpointSlice --api-versions elbv2.k8s.aws/v1alpha1 --api-versions elbv2.k8s.aws/v1alpha1/TargetGroupBinding --api-versions elbv2.k8s.aws/v1beta1 --api-versions elbv2.k8s.aws/v1beta1/IngressClassParams --api-versions elbv2.k8s.aws/v1beta1/TargetGroupBinding --api-versions events.k8s.io/v1 --api-versions events.k8s.io/v1/Event --api-versions events.k8s.io/v1beta1 --api-versions events.k8s.io/v1beta1/Event --api-versions flowcontrol.apiserver.k8s.io/v1beta1 --api-versions flowcontrol.apiserver.k8s.io/v1beta1/FlowSchema --api-versions flowcontrol.apiserver.k8s.io/v1beta1/PriorityLevelConfiguration --api-versions flowcontrol.apiserver.k8s.io/v1beta2 --api-versions flowcontrol.apiserver.k8s.io/v1beta2/FlowSchema --api-versions flowcontrol.apiserver.k8s.io/v1beta2/PriorityLevelConfiguration --api-versions jaegertracing.io/v1 --api-versions jaegertracing.io/v1/Jaeger --api-versions k6.io/v1alpha1 --api-versions k6.io/v1alpha1/K6 --api-versions logging.opsgy.com/v1beta1 --api-versions logging.opsgy.com/v1beta1/GlobalLokiRule --api-versions logging.opsgy.com/v1beta1/LokiRule --api-versions monitoring.coreos.com/v1 --api-versions monitoring.coreos.com/v1/Alertmanager --api-versions monitoring.coreos.com/v1/PodMonitor --api-versions monitoring.coreos.com/v1/Probe --api-versions monitoring.coreos.com/v1/Prometheus --api-versions monitoring.coreos.com/v1/PrometheusRule --api-versions monitoring.coreos.com/v1/ServiceMonitor --api-versions monitoring.coreos.com/v1/ThanosRuler --api-versions monitoring.coreos.com/v1alpha1 --api-versions monitoring.coreos.com/v1alpha1/AlertmanagerConfig --api-versions networking.k8s.io/v1 --api-versions networking.k8s.io/v1/Ingress --api-versions networking.k8s.io/v1/IngressClass --api-versions networking.k8s.io/v1/NetworkPolicy --api-versions node.k8s.io/v1 --api-versions node.k8s.io/v1/RuntimeClass --api-versions node.k8s.io/v1beta1 --api-versions node.k8s.io/v1beta1/RuntimeClass --api-versions policy/v1 --api-versions policy/v1/PodDisruptionBudget --api-versions policy/v1beta1 --api-versions policy/v1beta1/PodDisruptionBudget --api-versions policy/v1beta1/PodSecurityPolicy --api-versions rbac.authorization.k8s.io/v1 --api-versions rbac.authorization.k8s.io/v1/ClusterRole --api-versions rbac.authorization.k8s.io/v1/ClusterRoleBinding --api-versions rbac.authorization.k8s.io/v1/Role --api-versions rbac.authorization.k8s.io/v1/RoleBinding --api-versions scheduling.k8s.io/v1 --api-versions scheduling.k8s.io/v1/PriorityClass --api-versions storage.k8s.io/v1 --api-versions storage.k8s.io/v1/CSIDriver --api-versions storage.k8s.io/v1/CSINode --api-versions storage.k8s.io/v1/StorageClass --api-versions storage.k8s.io/v1/VolumeAttachment --api-versions storage.k8s.io/v1beta1 --api-versions storage.k8s.io/v1beta1/CSIStorageCapacity --api-versions v1 --api-versions v1/ConfigMap --api-versions v1/Endpoints --api-versions v1/Event --api-versions v1/LimitRange --api-versions v1/Namespace --api-versions v1/Node --api-versions v1/PersistentVolume --api-versions v1/PersistentVolumeClaim --api-versions v1/Pod --api-versions v1/PodTemplate --api-versions v1/ReplicationController --api-versions v1/ResourceQuota --api-versions v1/Secret --api-versions v1/Service --api-versions v1/ServiceAccount --api-versions vpcresources.k8s.aws/v1beta1 --api-versions vpcresources.k8s.aws/v1beta1/SecurityGroupPolicy --include-crds` failed exit status 1: [helm-secrets] Values filepath '.configuration/actions-runner-deployment-io/values.tst.yaml' is an absolute path. Absolute paths are not allowed. [helm-secrets] File does not exist: .configuration/actions-runner-deployment-io/values.tst.yaml Error: plugin "secrets" exited with error
jkroepke commented 1 year ago

Thats a strange behavior...

    - name: HELM_SECRETS_VALUES_ALLOW_ABSOLUTE_PATH
      value: "true"

cloud be a possible workaround. Setting

    - name: HELM_SECRETS_DEBUG
      value: "true"

may helps here, but can result into a huge amount of data.

jkroepke commented 1 year ago

Hi,

I'm able to reproduce the error which this application yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: example
spec:
  project: default
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  sources:
    - chart: example
      repoURL: https://jkroepke.github.io/helm-charts/
      targetRevision: 0.1.0
      helm:
        valueFiles:
          - $values/examples/sops/secrets.yaml
    - repoURL: 'https://github.com/jkroepke/helm-charts.git'
      targetRevision: main
      ref: values

will look into it.

Edit: Thats sad. ArgoCD is the imposter here...

ArgoCD strips the error message nativly.

On the ArgoCD UI, I had the same error message like

helm template . --name-template example --namespace default --kube-version 1.26 --values .examples/sops/secrets.yaml --api-versions admissionregistration.k8s.io/v1 ....

But looking at the argocd-repo-server logs, the command is executed differently:

helm template . --name-template example --namespace default --kube-version 1.26 --values /tmp/_argocd-repo/2917ddcd-1446-4d6c-a85b-d1f8f54f19ab/examples/sops/secrets.yaml --api-versions admissionregistration.k8s.io/v1 ....

Thats the reason, why helm-secrets denies the execution, since ArgoCD will pass an absolute path where is not allowed.

jkroepke commented 1 year ago

I update the documentations for ArgoCD. Check #351 for diff, otherwise here is full doc: https://github.com/jkroepke/helm-secrets/blob/fix-argocd-multi-app/docs/ArgoCD%20Integration.md

jeroen-s commented 1 year ago

Still seeing encrypted values in my Kubernetes secrets after hard refresh using this config:

repoServer:
  env:
    - name: HELM_PLUGINS
      value: /custom-tools/helm-plugins/
    - name: HELM_SECRETS_SOPS_PATH
      value: /custom-tools/sops
    - name: HELM_SECRETS_VALS_PATH
      value: /custom-tools/vals
    - name: HELM_SECRETS_KUBECTL_PATH
      value: /custom-tools/kubectl
    - name: HELM_SECRETS_CURL_PATH
      value: /custom-tools/curl
    - name: HELM_SECRETS_HELM_PATH
      value: /usr/local/bin/helm
    - name: HELM_SECRETS_BACKEND
      value: "vals" # or sops
    # https://github.com/jkroepke/helm-secrets/wiki/Security-in-shared-environments
    - name: HELM_SECRETS_VALUES_ALLOW_SYMLINKS
      value: "false"
    - name: HELM_SECRETS_VALUES_ALLOW_ABSOLUTE_PATH
      value: "true" # see https://github.com/jkroepke/helm-secrets/wiki/ArgoCD-Integration#multi-source-application-support-beta
    - name: HELM_SECRETS_VALUES_ALLOW_PATH_TRAVERSAL
      value: "false"
    - name: HELM_SECRETS_WRAPPER_ENABLED
      value: "true" # see https://github.com/jkroepke/helm-secrets/wiki/ArgoCD-Integration#multi-source-application-support-beta
  volumes:
    - name: custom-tools
      emptyDir: {}
  volumeMounts:
    - mountPath: /custom-tools
      name: custom-tools
    - mountPath: /usr/local/sbin/helm
      subPath: helm
      name: custom-tools

  initContainers:
    - name: download-tools
      image: alpine:latest
      command: [sh, -ec]
      env:
        - name: HELM_SECRETS_VERSION
          value: "4.4.0"
        - name: KUBECTL_VERSION
          value: "1.26.1"
        - name: VALS_VERSION
          value: "0.22.0"
        - name: SOPS_VERSION
          value: "3.7.3"
      args:
        - |
          mkdir -p /custom-tools/helm-plugins
          wget -qO- https://github.com/jkroepke/helm-secrets/releases/download/v${HELM_SECRETS_VERSION}/helm-secrets.tar.gz | tar -C /custom-tools/helm-plugins -xzf-;

          wget -qO /custom-tools/sops https://github.com/mozilla/sops/releases/download/v${SOPS_VERSION}/sops-v${SOPS_VERSION}.linux
          wget -qO /custom-tools/kubectl https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/amd64/kubectl

          wget -qO- https://github.com/helmfile/vals/releases/download/v${VALS_VERSION}/vals_${VALS_VERSION}_linux_amd64.tar.gz | tar -xzf- -C /custom-tools/ vals;

          cp /custom-tools/helm-plugins/helm-secrets/scripts/wrapper/helm.sh /custom-tools/helm

          chmod +x /custom-tools/*
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools

From the container:

argocd@argocd-repo-server-55cf65f8f4-j2vj9:~$ which helm
/usr/local/sbin/helm
argocd@argocd-repo-server-55cf65f8f4-j2vj9:~$ cat /usr/local/sbin/helm
#!/usr/bin/env sh
if [ "${HELM_SECRETS_WRAPPER_ENABLED}" = "true" ]; then
    exec "${HELM_SECRETS_HELM_PATH:-${HELM_BIN:-"helm"}}" secrets "$@"
else
    exec "${HELM_SECRETS_HELM_PATH:-${HELM_BIN:-"helm"}}" "$@"
fi
argocd@argocd-repo-server-55cf65f8f4-j2vj9:~$ echo $HELM_SECRETS_WRAPPER_ENABLED
true
argocd@argocd-repo-server-55cf65f8f4-j2vj9:~$ echo $HELM_SECRETS_VALUES_ALLOW_ABSOLUTE_PATH
true
jkroepke commented 1 year ago

Are you using sops encrypted secrets?

jeroen-s commented 1 year ago

Yes, this app with secrets.tst.enc.yaml being SOPS-encrypted:

...
      helm:
        valueFiles:
          - $values/configuration/actions-runner-deployment-io/values.tst.yaml
          - $values/configuration/actions-runner-deployment-io/secrets.tst.enc.yaml

I am not using ref+sops:// anywhere, could that be the problem?

jkroepke commented 1 year ago

I am not using ref+sops:// anywhere, could that be the problem?

Normally not, and I guess to would not work in this situation...

You have to configure sops here.

and

    - name: HELM_SECRETS_BACKEND
      value: "vals" # or sops

and

you have to configure the path of the gpg keys

    - name: HELM_SECRETS_LOAD_GPG_KEYS
      value: "/mount/keys/key.asc"

I'm aware thats a really bad user experience here, working with a lot of Workarounds here. But not able a solve this unless ArgoCD support the source references in urls

jeroen-s commented 1 year ago

sops backend works. The reason I was using vals was because the "Multi-Source Application Support" section is poorly phrased in my opinion. This put me on the wrong foot:

"On ArgoCD 2.6.x, sops isn't supported in Multi-Source application"

jkroepke commented 1 year ago

"On ArgoCD 2.6.x, sops isn't supported in Multi-Source application"

thats was true, but later than I got an idea with HELM_SECRETS_LOAD_GPG_KEYS which make the line in the docs obsolete ...

jkroepke commented 1 year ago

On my local setup, I had success. But I had an error with HELM_SECRETS_LOAD_GPG_KEYS inside ArgoCD. https://github.com/jkroepke/helm-secrets/pull/351 contains the fix.

Here is what I do

Create a Secret ```bash kubectl create secret generic helm-secrets-private-keys --from-file=key.asc=tests/assets/gpg/private2.gpg ```
values.yaml ```yaml repoServer: env: - name: HELM_PLUGINS value: /custom-tools/helm-plugins/ - name: HELM_SECRETS_CURL_PATH value: /custom-tools/curl - name: HELM_SECRETS_SOPS_PATH value: /custom-tools/sops - name: HELM_SECRETS_VALS_PATH value: /custom-tools/vals - name: HELM_SECRETS_KUBECTL_PATH value: /custom-tools/kubectl - name: HELM_SECRETS_BACKEND value: sops # https://github.com/jkroepke/helm-secrets/wiki/Security-in-shared-environments - name: HELM_SECRETS_VALUES_ALLOW_SYMLINKS value: "false" - name: HELM_SECRETS_VALUES_ALLOW_ABSOLUTE_PATH value: "true" - name: HELM_SECRETS_VALUES_ALLOW_PATH_TRAVERSAL value: "false" - name: HELM_SECRETS_WRAPPER_ENABLED value: "true" - name: HELM_SECRETS_HELM_PATH value: /usr/local/bin/helm - name: HELM_SECRETS_LOAD_GPG_KEYS # Multiple keys can be separated by space value: /helm-secrets-private-keys/key.asc volumes: - name: custom-tools emptyDir: {} # kubectl create secret generic helm-secrets-private-keys --from-file=key.asc=assets/gpg/private2.gpg - name: helm-secrets-private-keys secret: secretName: helm-secrets-private-keys volumeMounts: - mountPath: /custom-tools name: custom-tools - mountPath: /usr/local/sbin/helm subPath: helm name: custom-tools - mountPath: /helm-secrets-private-keys/ name: helm-secrets-private-keys initContainers: - name: download-tools image: alpine:latest imagePullPolicy: IfNotPresent command: [sh, -ec] env: - name: HELM_SECRETS_VERSION value: "4.4.0" - name: KUBECTL_VERSION value: "1.26.1" - name: VALS_VERSION value: "0.22.0" - name: SOPS_VERSION value: "3.7.3" args: - | mkdir -p /custom-tools/helm-plugins # wget -qO- https://github.com/jkroepke/helm-secrets/releases/download/v${HELM_SECRETS_VERSION}/helm-secrets.tar.gz | tar -C /custom-tools/helm-plugins -xzf-; wget -qO- https://github.com/jkroepke/helm-secrets/archive/refs/heads/fix-argocd-multi-app.tar.gz | tar -C /custom-tools/helm-plugins -xzf - helm-secrets-fix-argocd-multi-app/; wget -qO /custom-tools/curl https://github.com/moparisthebest/static-curl/releases/latest/download/curl-amd64 wget -qO /custom-tools/sops https://github.com/mozilla/sops/releases/download/v${SOPS_VERSION}/sops-v${SOPS_VERSION}.linux wget -qO /custom-tools/kubectl https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/amd64/kubectl wget -qO- https://github.com/helmfile/vals/releases/download/v${VALS_VERSION}/vals_${VALS_VERSION}_linux_amd64.tar.gz | tar -xzf- -C /custom-tools/ vals; # cp /custom-tools/helm-plugins/helm-secrets/scripts/wrapper/helm.sh /custom-tools/helm cp /custom-tools/helm-plugins/*/scripts/wrapper/helm.sh /custom-tools/helm chmod +x /custom-tools/* volumeMounts: - mountPath: /custom-tools name: custom-tools ```
application.yaml ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: example spec: project: default destination: server: https://kubernetes.default.svc namespace: default sources: - chart: example repoURL: https://jkroepke.github.io/helm-charts/ targetRevision: 0.1.0 helm: valueFiles: - $values/tests/assets/values/sops/secrets.gpg_key.yaml - repoURL: 'https://github.com/jkroepke/helm-secrets.git' targetRevision: main ref: values ```
jete-vian commented 1 year ago

The following line produces a 404 for me. I'm guessing we can't use version 4.4.0, that this fix is necessary?

 args:
        - |
          mkdir -p /custom-tools/helm-plugins
          # wget -qO- https://github.com/jkroepke/helm-secrets/releases/download/v${HELM_SECRETS_VERSION}/helm-secrets.tar.gz | tar -C /custom-tools/helm-plugins -xzf-;
          wget -qO- https://github.com/jkroepke/helm-secrets/archive/refs/heads/fix-argocd-multi-app.tar.gz | tar -C /custom-tools/helm-plugins -xzf - helm-secrets-fix-argocd-multi-app/;
wget -O- https://github.com/jkroepke/helm-secrets/archive/refs/heads/
fix-argocd-multi-app.tar.gz | tar -C /custom-tools/helm-plugins -xzf - helm-secrets-fix-argocd-multi-app/
--2023-03-06 19:19:22--  https://github.com/jkroepke/helm-secrets/archive/refs/heads/fix-argocd-multi-app.tar.gz
Resolving github.com (github.com)... 140.82.113.3
Connecting to github.com (github.com)|140.82.113.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/jkroepke/helm-secrets/tar.gz/refs/heads/fix-argocd-multi-app [following]
--2023-03-06 19:19:22--  https://codeload.github.com/jkroepke/helm-secrets/tar.gz/refs/heads/fix-argocd-multi-app
Resolving codeload.github.com (codeload.github.com)... 140.82.114.9
Connecting to codeload.github.com (codeload.github.com)|140.82.114.9|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-03-06 19:19:22 ERROR 404: Not Found.

Also, when I set HELM_SECRETS_VALUES_ALLOW_ABSOLUTE_PATH and HELM_SECRETS_WRAPPER_ENABLED to true see the following errors in my ArgoCD applications

mkdir: cannot create directory '/home/argocd/.secrets': Read-only file system Error: plugin "secrets" exited with error
jkroepke commented 1 year ago

@jete-vian Bad timing. I merged the branch 30 minutes ago and preparing an official release in the next minutes.

mkdir: cannot create directory '/home/argocd/.secrets': Read-only file system Error: plugin "secrets" exited with error

4.4.1 is coming soon. that contains the fix. That the necessary fix from the merged branch (#351).