search

travisghansen / argo-cd-helmfile

Integration between argo-cd and helmfile
MIT License
213 stars 55 forks source link

Passing args to helm #36

Open moreiramarti opened 1 year ago

moreiramarti commented 1 year ago

Hello,

I'm experiencing a strange behavior when I'm using helmfile.d tree structure. Some args are passed to helm pull command but they shouldn't (cf output below)

It seems passing args from helmfile to helm is not a good practice (doc)

Do you think it could be improved or make it optional ? For now, I removed the --args parameter from the script like I'm not using it.

moreiramarti commented 1 year ago 
argocd app get ephemeral-pull-helmfile --hard-refresh
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web. 
WARN[0003] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web. 
Name:               argocd/ephemeral-pull-helmfile
Project:            infra
Server:             testbed
Namespace:          ephemeral-pull-helmfile
URL:                https://argocd.voodoo-infra-sandbox.io/applications/ephemeral-pull-helmfile
Repo:               https://github.com/VoodooTeam/argocd-sandbox-infra.git
Target:             master
Path:               ephemeral/
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Unknown
Health Status:      Healthy

CONDITION        MESSAGE  LAST TRANSITION
ComparisonError  rpc error: code = Unknown desc = plugin sidecar failed. error generating manifests in cmp: rpc error: code = Unknown desc = error generating manifests: `bash -c /usr/local/bin/argo-cd-helmfile.sh generate` failed exit status 1: v3.10.3+g835b733
helmfile version 0.151.0
starting generate
Templating release=db, chart=bitnami/postgresql
Templating release=db-migration, chart=../charts/db-migration
Pulling 456072703506.dkr.ecr.eu-west-1.amazonaws.com/helm/stateless-service:6.0.1
in helmfile.d/01_application.yaml: [release "ephemeral": command "/usr/local/bin/helm" exited with non-zero status:

PATH:
  /usr/local/bin/helm

ARGS:
  0: /usr/local/bin/helm (19 bytes)
  1: pull (4 bytes)
  2: oci://456072703506.dkr.ecr.eu-west-1.amazonaws.com/helm/stateless-service (73 bytes)
  3: --version (9 bytes)
  4: 6.0.1 (5 bytes)
  5: --destination (13 bytes)
  6: /tmp/helmfile2452714502/ephemeral-pull-helmfile/ephemeral/stateless-service/6.0.1 (81 bytes)
  7: --untar (7 bytes)
  8: --kube-version=1.23 (19 bytes)
  9: --api-versions=acme.cert-manager.io/v1 (38 bytes)
  10: --api-versions=acme.cert-manager.io/v1/Challenge (48 bytes)
  11: --api-versions=acme.cert-manager.io/v1/Order (44 bytes)
  12: --api-versions=admissionregistration.k8s.io/v1 (46 bytes)
  13: --api-versions=admissionregistration.k8s.io/v1/MutatingWebhookConfiguration (75 bytes)
  14: --api-versions=admissionregistration.k8s.io/v1/ValidatingWebhookConfiguration (77 bytes)
  15: --api-versions=apiextensions.k8s.io/v1 (38 bytes)
  16: --api-versions=apiextensions.k8s.io/v1/CustomResourceDefinition (63 bytes)
  17: --api-versions=apiregistration.k8s.io/v1 (40 bytes)
  18: --api-versions=apiregistration.k8s.io/v1/APIService (51 bytes)
  19: --api-versions=apps/v1 (22 bytes)
  20: --api-versions=apps/v1/ControllerRevision (41 bytes)
  21: --api-versions=apps/v1/DaemonSet (32 bytes)
  22: --api-versions=apps/v1/Deployment (33 bytes)
  23: --api-versions=apps/v1/ReplicaSet (33 bytes)
  24: --api-versions=apps/v1/StatefulSet (34 bytes)
  25: --api-versions=argoproj.io/v1alpha1 (35 bytes)
  26: --api-versions=argoproj.io/v1alpha1/AppProject (46 bytes)
  27: --api-versions=argoproj.io/v1alpha1/Application (47 bytes)
  28: --api-versions=argoproj.io/v1alpha1/ApplicationSet (50 bytes)
  29: --api-versions=argoproj.io/v1alpha1/ArgoCDExtension (51 bytes)
  30: --api-versions=autoscaling/v1 (29 bytes)
  31: --api-versions=autoscaling/v1/HorizontalPodAutoscaler (53 bytes)
  32: --api-versions=autoscaling/v2 (29 bytes)
  33: --api-versions=autoscaling/v2/HorizontalPodAutoscaler (53 bytes)
  34: --api-versions=autoscaling/v2beta1 (34 bytes)
  35: --api-versions=autoscaling/v2beta1/HorizontalPodAutoscaler (58 bytes)
  36: --api-versions=autoscaling/v2beta2 (34 bytes)
  37: --api-versions=autoscaling/v2beta2/HorizontalPodAutoscaler (58 bytes)
  38: --api-versions=batch/v1 (23 bytes)
  39: --api-versions=batch/v1/CronJob (31 bytes)
  40: --api-versions=batch/v1/Job (27 bytes)
  41: --api-versions=batch/v1beta1 (28 bytes)
  42: --api-versions=batch/v1beta1/CronJob (36 bytes)
  43: --api-versions=bitnami.com/v1alpha1 (35 bytes)
  44: --api-versions=bitnami.com/v1alpha1/SealedSecret (48 bytes)
  45: --api-versions=cert-manager.io/v1 (33 bytes)
  46: --api-versions=cert-manager.io/v1/Certificate (45 bytes)
  47: --api-versions=cert-manager.io/v1/CertificateRequest (52 bytes)
  48: --api-versions=cert-manager.io/v1/ClusterIssuer (47 bytes)
  49: --api-versions=cert-manager.io/v1/Issuer (40 bytes)
  50: --api-versions=certificates.k8s.io/v1 (37 bytes)
  51: --api-versions=certificates.k8s.io/v1/CertificateSigningRequest (63 bytes)
  52: --api-versions=coordination.k8s.io/v1 (37 bytes)
  53: --api-versions=coordination.k8s.io/v1/Lease (43 bytes)
  54: --api-versions=crd.k8s.amazonaws.com/v1alpha1 (45 bytes)
  55: --api-versions=crd.k8s.amazonaws.com/v1alpha1/ENIConfig (55 bytes)
  56: --api-versions=discovery.k8s.io/v1 (34 bytes)
  57: --api-versions=discovery.k8s.io/v1/EndpointSlice (48 bytes)
  58: --api-versions=discovery.k8s.io/v1beta1 (39 bytes)
  59: --api-versions=discovery.k8s.io/v1beta1/EndpointSlice (53 bytes)
  60: --api-versions=elbv2.k8s.aws/v1alpha1 (37 bytes)
  61: --api-versions=elbv2.k8s.aws/v1alpha1/TargetGroupBinding (56 bytes)
  62: --api-versions=elbv2.k8s.aws/v1beta1 (36 bytes)
  63: --api-versions=elbv2.k8s.aws/v1beta1/IngressClassParams (55 bytes)
  64: --api-versions=elbv2.k8s.aws/v1beta1/TargetGroupBinding (55 bytes)
  65: --api-versions=events.k8s.io/v1 (31 bytes)
  66: --api-versions=events.k8s.io/v1/Event (37 bytes)
  67: --api-versions=events.k8s.io/v1beta1 (36 bytes)
  68: --api-versions=events.k8s.io/v1beta1/Event (42 bytes)
  69: --api-versions=flowcontrol.apiserver.k8s.io/v1beta1 (51 bytes)
  70: --api-versions=flowcontrol.apiserver.k8s.io/v1beta1/FlowSchema (62 bytes)
  71: --api-versions=flowcontrol.apiserver.k8s.io/v1beta1/PriorityLevelConfiguration (78 bytes)
  72: --api-versions=flowcontrol.apiserver.k8s.io/v1beta2 (51 bytes)
  73: --api-versions=flowcontrol.apiserver.k8s.io/v1beta2/FlowSchema (62 bytes)
  74: --api-versions=flowcontrol.apiserver.k8s.io/v1beta2/PriorityLevelConfiguration (78 bytes)
  75: --api-versions=irsa.voodoo.io/v1alpha1 (38 bytes)
  76: --api-versions=irsa.voodoo.io/v1alpha1/IamRoleServiceAccount (60 bytes)
  77: --api-versions=irsa.voodoo.io/v1alpha1/Policy (45 bytes)
  78: --api-versions=irsa.voodoo.io/v1alpha1/Role (43 bytes)
  79: --api-versions=k6.io/v1alpha1 (29 bytes)
  80: --api-versions=k6.io/v1alpha1/K6 (32 bytes)
  81: --api-versions=monitoring.coreos.com/v1 (39 bytes)
  82: --api-versions=monitoring.coreos.com/v1/Alertmanager (52 bytes)
  83: --api-versions=monitoring.coreos.com/v1/PodMonitor (50 bytes)
  84: --api-versions=monitoring.coreos.com/v1/Probe (45 bytes)
  85: --api-versions=monitoring.coreos.com/v1/Prometheus (50 bytes)
  86: --api-versions=monitoring.coreos.com/v1/PrometheusRule (54 bytes)
  87: --api-versions=monitoring.coreos.com/v1/ServiceMonitor (54 bytes)
  88: --api-versions=monitoring.coreos.com/v1/ThanosRuler (51 bytes)
  89: --api-versions=monitoring.coreos.com/v1alpha1 (45 bytes)
  90: --api-versions=monitoring.coreos.com/v1alpha1/AlertmanagerConfig (64 bytes)
  91: --api-versions=monitoring.grafana.com/v1alpha1 (46 bytes)
  92: --api-versions=monitoring.grafana.com/v1alpha1/GrafanaAgent (59 bytes)
  93: --api-versions=monitoring.grafana.com/v1alpha1/Integration (58 bytes)
  94: --api-versions=monitoring.grafana.com/v1alpha1/LogsInstance (59 bytes)
  95: --api-versions=monitoring.grafana.com/v1alpha1/MetricsInstance (62 bytes)
  96: --api-versions=monitoring.grafana.com/v1alpha1/PodLogs (54 bytes)
  97: --api-versions=networking.k8s.io/v1 (35 bytes)
  98: --api-versions=networking.k8s.io/v1/Ingress (43 bytes)
  99: --api-versions=networking.k8s.io/v1/IngressClass (48 bytes)
  100: --api-versions=networking.k8s.io/v1/NetworkPolicy (49 bytes)
  101: --api-versions=node.k8s.io/v1 (29 bytes)
  102: --api-versions=node.k8s.io/v1/RuntimeClass (42 bytes)
  103: --api-versions=node.k8s.io/v1beta1 (34 bytes)
  104: --api-versions=node.k8s.io/v1beta1/RuntimeClass (47 bytes)
  105: --api-versions=policy/v1 (24 bytes)
  106: --api-versions=policy/v1/PodDisruptionBudget (44 bytes)
  107: --api-versions=policy/v1beta1 (29 bytes)
  108: --api-versions=policy/v1beta1/PodDisruptionBudget (49 bytes)
  109: --api-versions=policy/v1beta1/PodSecurityPolicy (47 bytes)
  110: --api-versions=rbac.authorization.k8s.io/v1 (43 bytes)
  111: --api-versions=rbac.authorization.k8s.io/v1/ClusterRole (55 bytes)
  112: --api-versions=rbac.authorization.k8s.io/v1/ClusterRoleBinding (62 bytes)
  113: --api-versions=rbac.authorization.k8s.io/v1/Role (48 bytes)
  114: --api-versions=rbac.authorization.k8s.io/v1/RoleBinding (55 bytes)
  115: --api-versions=scheduling.k8s.io/v1 (35 bytes)
  116: --api-versions=scheduling.k8s.io/v1/PriorityClass (49 bytes)
  117: --api-versions=storage.k8s.io/v1 (32 bytes)
  118: --api-versions=storage.k8s.io/v1/CSIDriver (42 bytes)
  119: --api-versions=storage.k8s.io/v1/CSINode (40 bytes)
  120: --api-versions=storage.k8s.io/v1/StorageClass (45 bytes)
  121: --api-versions=storage.k8s.io/v1/VolumeAttachment (49 bytes)
  122: --api-versions=storage.k8s.io/v1beta1 (37 bytes)
  123: --api-versions=storage.k8s.io/v1beta1/CSIStorageCapacity (56 bytes)
  124: --api-versions=v1 (17 bytes)
  125: --api-versions=v1/ConfigMap (27 bytes)
  126: --api-versions=v1/Endpoints (27 bytes)
  127: --api-versions=v1/Event (23 bytes)
  128: --api-versions=v1/LimitRange (28 bytes)
  129: --api-versions=v1/Namespace (27 bytes)
  130: --api-versions=v1/Node (22 bytes)
  131: --api-versions=v1/PersistentVolume (34 bytes)
  132: --api-versions=v1/PersistentVolumeClaim (39 bytes)
  133: --api-versions=v1/Pod (21 bytes)
  134: --api-versions=v1/PodTemplate (29 bytes)
  135: --api-versions=v1/ReplicationController (39 bytes)
  136: --api-versions=v1/ResourceQuota (31 bytes)
  137: --api-versions=v1/Secret (24 bytes)
  138: --api-versions=v1/Service (25 bytes)
  139: --api-versions=v1/ServiceAccount (32 bytes)
  140: --api-versions=velero.io/v1 (27 bytes)
  141: --api-versions=velero.io/v1/Backup (34 bytes)
  142: --api-versions=velero.io/v1/BackupStorageLocation (49 bytes)
  143: --api-versions=velero.io/v1/DeleteBackupRequest (47 bytes)
  144: --api-versions=velero.io/v1/DownloadRequest (43 bytes)
  145: --api-versions=velero.io/v1/PodVolumeBackup (43 bytes)
  146: --api-versions=velero.io/v1/PodVolumeRestore (44 bytes)
  147: --api-versions=velero.io/v1/ResticRepository (44 bytes)
  148: --api-versions=velero.io/v1/Restore (35 bytes)
  149: --api-versions=velero.io/v1/Schedule (36 bytes)
  150: --api-versions=velero.io/v1/ServerStatusRequest (47 bytes)
  151: --api-versions=velero.io/v1/VolumeSnapshotLocation (50 bytes)
  152: --api-versions=vpcresources.k8s.aws/v1beta1 (43 bytes)
  153: --api-versions=vpcresources.k8s.aws/v1beta1/SecurityGroupPolicy (63 bytes)

ERROR:
  exit status 1

EXIT STATUS
  1

STDERR:
  Error: unknown flag: --kube-version

COMBINED OUTPUT:
  Error: unknown flag: --kube-version]  2023-03-03 10:31:41 +0100 CET

GROUP              KIND                 NAMESPACE                NAME                   STATUS   HEALTH   HOOK  MESSAGE
                   Secret               ephemeral-pull-helmfile  ephemeral-postgres     Unknown                 
                   Service              ephemeral-pull-helmfile  ephemeral              Unknown  Healthy        
                   Service              ephemeral-pull-helmfile  ephemeral-postgres     Unknown  Healthy        
                   Service              ephemeral-pull-helmfile  ephemeral-postgres-hl  Unknown  Healthy        
                   ServiceAccount       ephemeral-pull-helmfile  ephemeral              Unknown                 
apps               Deployment           ephemeral-pull-helmfile  ephemeral              Unknown  Healthy        
apps               StatefulSet          ephemeral-pull-helmfile  ephemeral-postgres     Unknown  Healthy        
batch              Job                  ephemeral-pull-helmfile  ephemeral-db-migrate   Unknown  Healthy        
networking.k8s.io  Ingress              ephemeral-pull-helmfile  ephemeral              Unknown  Healthy        
policy             PodDisruptionBudget  ephemeral-pull-helmfile  ephemeral              Unknown
travisghansen commented 1 year ago

Can you send over the app definition so I can see what you have set?

travisghansen commented 1 year ago

Or the helmfile content if you have the args defined there.

moreiramarti commented 1 year ago

Like I'm using multiple yaml in helmfile.d directory, I don't have a helmfile.yaml. This is my helmfile yaml :

releases:

releases:

And my app definition :

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: ephemeral-as
spec:
  generators:
  - pullRequest:
      github:
        owner: VoodooTeam
        repo: ephemeral-env
        tokenRef:
          secretName: argocd-github-secret
          key: pat
      requeueAfterSeconds: 1800
      filters:
      - branchMatch: pull-.*
  template:
    metadata:
      name: ephemeral-{{branch}}
    spec:
      source:
        repoURL: https://github.com/xxx/argocd-sandbox.git
        targetRevision: master
        path: ephemeral/
        plugin:
          parameters:
          - name: ephemeral_values
            map:
              image.tag: "{{head_sha}}"
              ingress.host: "{{branch_slug}}.xxx.xx"
              environment.CSRF_TRUSTED_ORIGINS: "https://{{branch_slug}}.xxx.xx"
      project: infra
      destination:
        name: testbed
        namespace: ephemeral-{{branch}}
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
        - CreateNamespace=true
travisghansen commented 1 year ago

I suspect this has some behavior difference due to the repository being oci. Conceptually the goal is to make sure all repos are added and necessary charts downloads during the init phase (which occurs before generate phase). The error above is occurring during the generate phase which leads me to believe something fundamentally different with how helmfile handles oci vs non-oci registries. Can you send the (cleansed) content of the repositories.yaml file?

travisghansen commented 1 year ago

I've opened this: https://github.com/helmfile/helmfile/discussions/727

moreiramarti commented 1 year ago

Sorry for the delay The fiie : 

repositories:
- name: bitnami
  url: https://charts.bitnami.com/bitnami
- name: localstack
  url: https://localstack.github.io/helm-charts
- name: ecr
  url: xxx.dkr.ecr.eu-west-1.amazonaws.com/helm
  oci: true
welderpb commented 1 year ago

@travisghansen I have the same issue with oci repositories. I did some investigation and can say: helmfile before pass template command passing pull command for oci repositories to helm. helm 'pull' command doesn't support --kube-version and --api-versions arguments.. also I checked 'fetch' command before 'template' command. it is doesn't help. Template still pulling charts.

you are always passing this arguments to 'template' command:

 # TODO: support post process pipeline here
    ${helmfile} \
      template \
      --skip-deps ${INTERNAL_HELMFILE_TEMPLATE_OPTIONS} \
      --args "${INTERNAL_HELM_TEMPLATE_OPTIONS} ${HELM_TEMPLATE_OPTIONS}" \
      ${HELMFILE_TEMPLATE_OPTIONS}
    ;;

So, my suggestion is make this arguments are optional, or add a new ENV to possibility disable this args.

travisghansen commented 1 year ago

@welderpb thanks! I'm unaware of another option to ensure proper kubeapi versions etc can be passed down to helm, am I missing something?

welderpb commented 1 year ago

@travisghansen it is not necessary to pass this arguments to helm templating, it is should be optional.

travisghansen commented 1 year ago

In many situations they absolutely are required unfortunately :( we’ll work closely with helmfile to make sure we sanely cover the necessary use cases and the issue will go away.