search

argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
17.71k stars 5.4k forks source link

ENV variables from "params" #1705

Open rosscdh opened 5 years ago

rosscdh commented 5 years ago

Is your feature request related to a problem? Please describe. Hi there, firstly thanks for a really well composed system (a few rough edges but all together great!)

Describe the solution you'd like Id like to know the possibility of allowing argocd app set guestbook -p MY_VAR=MY_VALUE to be used as environment vars with kustomize .. as there are certain workflows that would require this.

Have you thought about contributing yourself?

Would love to.. need to powerup the go skills tho

alexec commented 5 years ago

This looks like a neat feature. As I mentioned in Slack:

Anyone wanting to contribute but worried you don't know Go, spend 2h on the tutorial https://tour.golang.org, I read this before lunch and was commit changes before my first afternoon coffee.

This is where you'd need to add this:

rosscdh commented 5 years ago

Ok so, using Cobera for cli, its then a case of

  1. find the Params hash
  2. decide on creating a new proj subcommand probably argocd proj upsert-envvar|delete-envvar NAME=VALUE
  3. write a generic env method to add/remove env vars to the cmd.Env
  4. find the kustomize command caller and use the updated cmd.Env when calling kustomize and use envsubst or similar to pipe to content

question

  1. using kustomize build or the new kubctl -k may be worth investigating the new kustomze integration

Sound about right?

alexec commented 5 years ago

@rosscdh you're bang-on right.

A thought I have is that with kubectl, you have --env="DNS_DOMAIN=cluster" (note we can't use --env, but it's already used, but we could use -e. Environment variables be a nice feature for all the tools (Helm, Ksonnet, plain YAML), but just do Kustomize today.

This would be ... more complex:

  1. In types.go to ApplicationSource a Envmap[string]string.
  2. Generate the code using make codegen.

Optionally, any of the following depending on how you personally want to set/see the env, change the following:

And maybe display in the UI somewhere.

Don't implement something you won't use mind you!

@jessesuen thoughts?

rosscdh commented 5 years ago

Unfortunately I am on holiday for 2 weeks in FR if i get a chance ill hack otherwise will do when i get back (if someone else hasn't claimed it)

rosscdh commented 5 years ago

on codegen it seems redis-ha has been removed from : https://kubernetes-charts.storage.googleapis.com

+ helm dependency update ./chart --skip-refresh
Error: Can't get a valid version for repositories redis-ha. Try changing the version constraint in requirements.yaml
rosscdh commented 5 years ago

Manually downloaded and now proceeding (strange internet pipes here)

rosscdh commented 5 years ago

https://github.com/argoproj/argo-cd/pull/1718 still [WIP]

alexec commented 5 years ago

@rosscdh you mention that there are examples when you'd use this for Kustomize?

alexec commented 5 years ago

I think you could achieve this now using a custom plugin + envsubst.

rosscdh commented 5 years ago

Yes thats what i thought initially too; in fact i have one build that still makes use of it for secrets atm.

Coudl oyu point me at the custom plugin please?

On Wed, Jul 3, 2019 at 12:09 AM Alex Collins notifications@github.com wrote:

I think you could achieve this now using a custom plugin + envsubst.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/argoproj/argo-cd/issues/1705?email_source=notifications&email_token=AADA6MVSAWQVEY537WLBRFLP5PGZTA5CNFSM4HVCU4BKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZCWCGY#issuecomment-507863323, or mute the thread https://github.com/notifications/unsubscribe-auth/AADA6MWP3WD4A3MKBQPIGZDP5PGZTANCNFSM4HVCU4BA .

rosscdh commented 5 years ago

Ah https://argoproj.github.io/argo-cd/user-guide/config-management-plugins/ yes that would def work, tho would be better integrated with vaule or similar for secrets. https://argoproj.github.io/argo-cd/user-guide/config-management-plugins/ tho hacking things into the pod is a little anti-patterny.

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

ptux commented 5 years ago

I want set env vars into argocd which kustomize could use. Something like circle ci does.

rosscdh commented 5 years ago

@ptux I had the same thought, however you should not be using dynamic variables in kustomize (there is a specific reason why they opted against it)

alexec commented 5 years ago

2415 supports some special env vars for Helm/Jsonnet. I don't think we can do this for Kustomize because we invoke kustomize build . and this does not have parameters nor take the env into account. Am I missing something?

ptux commented 4 years ago 
$ kustomize version
Version: {KustomizeVersion:3.1.0 GitCommit:95f3303493fdea243ae83b767978092396169baf BuildDate:2019-07-26T19:21:45+01:00 GoOs:darwin GoArch:amd64}

Here is what I can I do on local .

 $ tree
.
├── kustomization.yaml
└── secret.env

0 directories, 2 files
$ cat kustomization.yaml 
namespace: hello-cloudbuild-dev

secretGenerator:
- name: mysecret
  envs:
  - secret.env
  type: Opaque

generatorOptions:
  disableNameSuffixHash: true
$ cat secret.env 
SECRET_USERNAME
SECRET_PASSWORD
$ env | grep SECRET
SECRET_PASSWORD=password2
SECRET_USERNAME=admin2
$ kustomize build .
apiVersion: v1
data:
  SECRET_PASSWORD: cGFzc3dvcmQy
  SECRET_USERNAME: YWRtaW4y
kind: Secret
metadata:
  name: mysecret
  namespace: hello-cloudbuild-dev
type: Opaque

So, I want to add env to argo CD server. Somthing like this.

argocd app creat appname --env SECRET_USERNAME=admin2 --env SECRET_PASSWORD=password2
alexec commented 4 years ago

So you want Argo CD to create the secret.env file?

ptux commented 4 years ago

Well, technically speaking, I want Argo CD to create values for env vars.

secret.env file just only holds the keys of env vars but not the values.

SECRET_USERNAME
SECRET_PASSWORD

So we could push secret.env to the git repo.

alexec commented 4 years ago

OK. So you need a file containing a list of the available envvars so you can use them in your secrets? Where do the values of those vars come from?

https://github.com/kubernetes-sigs/kustomize/blob/master/examples/secretGeneratorPlugin.md#secret-values-from-local-files

anarcher commented 4 years ago

I would like to set env vars into argocd which kustomize could use. IMHO, kustomize custom plugin can use the envs

olegstepura commented 2 years ago

I've created a simple docker container image containing envsubst which can be used as config management plugin to post-process manifests replacing ENV vars. Usage example in the readme: olegstepura/glibc-envsubst

Idea is the following:

Now when your app will be synced by ArgoCD it will run kustomize build --enable-helm . | envsubst piping resulting multifile yaml to envsubst which will substitute ENV vars mounted from cluster-global-vars secret (and ARGOCD_* own ones as well) and resulting yaml with vars replaced will be applied to your cluster.

pearj commented 2 years ago

@olegstepura kustomize already supports environment variables, so you don't need envsubst, as long as the environment variables are available when you run kustomize. It is worth noting that environment variables are slightly fiddly to use with kustomize, but it does work, just not out of the box with ArgoCD.

olegstepura commented 2 years ago

Hi, @pearj !

Thanks! You must be talking about this feature, right?

For some reason I could not make it work. In this example:

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

helmCharts:
- name: metallb
  version: 0.12.1
  repo: https://metallb.github.io/metallb
  releaseName: metallb
  valuesFile: values.yaml
# values.yaml
configInline:
  address-pools:
    - name: default
      protocol: layer2
      addresses:
        - $(CLUSTER_IP_RANGE)

It keeps var intact after ArgoCD syncs it.

image

While in argocd-repo-server ENV var is defined:

$ kubectl exec --stdin --tty -n argocd argocd-repo-server-69cc9f789f-758cs  -- env | grep CLUSTER_IP_RANGE
Defaulted container "argocd-repo-server" out of: argocd-repo-server, download-tools (init), copyutil (init)
CLUSTER_IP_RANGE=10.10.10.30-10.10.10.44

I would be glad to get rid of a complex plugin setup if it would work. Any idea what am I missing here?

Edit: I tried both ArgoCD own detection of repo type and using a plugin that simply runs kustomize build --enable-helm .. Both do not replace $(CLUSTER_IP_RANGE)

Edit2: Does not work this way either:

# kustomization.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: metallb
spec:
  destination:
    namespace: metallb-system
  source:
    repoURL: https://metallb.github.io/metallb
    chart: metallb
    targetRevision: 0.12.1
    helm:
      values: |
        configInline:
          address-pools:
            - name: default
              protocol: layer2
              addresses:
                - $(CLUSTER_IP_RANGE)
pearj commented 2 years ago

Hi @olegstepura,

It's only a recently documented feature of being able to use environment variables to populate vars in kustomize.

However, the main catch is that ArgoCD doesn’t let you specify environment variables in the application definition. I’m pretty sure that is what this issue is actually about. It doesn’t look like it’d actually be that hard a change in go to implement and I’m pondering doing it myself.

Until argocd supports it you’d still need a plugin setup that calls kustomize but in your use case you might be able to remove the envsubst step, and use kustomize var substitution.

I did notice that the plugin section in argocd applications do support specifying environment variables. So you could pass in the values you want there and then when you call kustomize it should use your environment variables a kustomize vars if you configure it right.

Here are my notes on using environment variables:

Using Environment Variables

Kustomize can use environment variables and substitute those variables.

It needs the following:

config.env:

AADPODIDBINDING

kustomization.yaml:

patchesStrategicMerge:
- |-
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: secret-agent-controller-manager
    namespace: secret-agent-system
  spec:
    template:
      metadata:
        labels:
          aadpodidbinding: $(AADPODIDBINDING)

configMapGenerator:
- name: secret-agent-config
  envs:
    - config.env

# Not strictly necessary, but allows for a config map with a stable name
generatorOptions:
  disableNameSuffixHash: true

vars:
- name: AADPODIDBINDING
  fieldref:
    fieldpath: data.AADPODIDBINDING
  objref:
    apiVersion: v1
    kind: ConfigMap
    name: secret-agent-config

configurations:
  - kustomizeConfig.yaml

kustomizeConfig.yaml:

varReference:
- kind: Deployment
  path: spec/template/metadata/labels/aadpodidbinding

Then if you call kustomize like this:

AADPODIDBINDING=abc1234 kustomize build .

Reference:

tornike commented 2 years ago

I am using custom plugin just to pass custom environment variables and run kustomize build with them. I don't get it why kustomize plugin does not have env support like CMP. It's like argocd is pushing you towards using helm, but I don't want to use helm.

In addition to all of these, image-updater only supports kustomize and helm plugins, so not having possibility to pass some parameters is a big disadvantage.

bburky commented 2 years ago

A while ago, I put together a repo demonstrating KV → ConfigMap → vars → string replacement in resources. https://github.com/bburky/kustomize-transformer-environment-variables

The intended use case was Argo CD, but I never deployed it. I understand that Argo CD's native Kustomize support doesn't pass it any environment variables, and a Config Management Plugin (CMP) would be needed just to pick up the environment variables.

It's really not very nice Kustomize code, and notably leaves around a CM in your deployed environment. The vars documentation mentions "WARNING: There are plans to deprecate vars" and recommends using replacements instead (but it's string replacement is less powerful).

qixiaobo commented 2 years ago

Hi, @pearj !

Thanks! You must be talking about this feature, right?

For some reason I could not make it work. In this example:

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

helmCharts:
- name: metallb
  version: 0.12.1
  repo: https://metallb.github.io/metallb
  releaseName: metallb
  valuesFile: values.yaml
# values.yaml
configInline:
  address-pools:
    - name: default
      protocol: layer2
      addresses:
        - $(CLUSTER_IP_RANGE)

It keeps var intact after ArgoCD syncs it. image

While in argocd-repo-server ENV var is defined:

$ kubectl exec --stdin --tty -n argocd argocd-repo-server-69cc9f789f-758cs  -- env | grep CLUSTER_IP_RANGE
Defaulted container "argocd-repo-server" out of: argocd-repo-server, download-tools (init), copyutil (init)
CLUSTER_IP_RANGE=10.10.10.30-10.10.10.44

I would be glad to get rid of a complex plugin setup if it would work. Any idea what am I missing here?

Edit: I tried both ArgoCD own detection of repo type and using a plugin that simply runs kustomize build --enable-helm .. Both do not replace $(CLUSTER_IP_RANGE)

Edit2: Does not work this way either:

# kustomization.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: metallb
spec:
  destination:
    namespace: metallb-system
  source:
    repoURL: https://metallb.github.io/metallb
    chart: metallb
    targetRevision: 0.12.1
    helm:
      values: |
        configInline:
          address-pools:
            - name: default
              protocol: layer2
              addresses:
                - $(CLUSTER_IP_RANGE)

Really appreciate

zs-dima commented 10 months ago

@olegstepura @pearj vars was deprecated in v5.0.0

joebowbeer commented 10 months ago

kustomize removed the apparent misfeature that allowed naked env assignments to inherit from the shell environment.

However, the requested feature does look like a job for a preprocessor running in a configuration management plugin (CMP).

The preprocessor would inject values into a ConfigMap resource.

zs-dima commented 10 months ago

FluxCD has substitution in this way currently:

https://fluxcd.io/flux/components/kustomize/kustomizations/#post-build-variable-substitution

postBuild:
  substituteFrom:
    - kind: ConfigMap
      name: variables-config
patches:
    - target:
        kind: Kustomization
        group: kustomize.toolkit.fluxcd.io
      patch: |-
        - op: add
          path: /spec/postBuild
          value:
            substituteFrom:
              - kind: ConfigMap
                name: variables-config
kubectl create configmap variables-config \
  --from-literal=DOMAIN=$DOMAIN \
  --from-literal=EMAIL=$EMAIL \
  -n flux-system

Then variables can be used in this way:

    external-dns.kubernetes.io/target: app.${DOMAIN}

Maybe Argo CD can it do the same or find some better solution

zs-dima commented 10 months ago

variables specially helpful to specify children apps ingress for the App of Apps pattern, could be nice to substitude them using root app Helm:

Root app:

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  source:
    repoURL: https://github.com/user/repository
    path: apps
    env:
      - name: DOMAIN
        value: {{ .Values.app.domain }}

Children apps, as you can see helm values have different structure:

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  source:
    helm:
      valuesObject:
        ingressRoute:
          dashboard:
            matchRule: Host(`traefik.${DOMAIN}`) && (PathPrefix(`/dashboard`) || PathPrefix(`/api`))
            annotations:
              external-dns.kubernetes.io/target: traefik.${DOMAIN}

. . .

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  source:
    helm:
      valuesObject:
        ingress:
          annotations:
            external-dns.kubernetes.io/target: longhorn.${DOMAIN}
          host: longhorn.${DOMAIN}

The workaround is to place all repository content under the root app templates folder, but this looks weird.

mrkhachaturov commented 4 months ago

variables specially helpful to specify children apps ingress for the App of Apps pattern, could be nice to substitude them using root app Helm:

Root app:

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  source:
    repoURL: https://github.com/user/repository
    path: apps
    env:
      - name: DOMAIN
        value: {{ .Values.app.domain }}

Children apps, as you can see helm values have different structure:

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  source:
    helm:
      valuesObject:
        ingressRoute:
          dashboard:
            matchRule: Host(`traefik.${DOMAIN}`) && (PathPrefix(`/dashboard`) || PathPrefix(`/api`))
            annotations:
              external-dns.kubernetes.io/target: traefik.${DOMAIN}

. . .

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  source:
    helm:
      valuesObject:
        ingress:
          annotations:
            external-dns.kubernetes.io/target: longhorn.${DOMAIN}
          host: longhorn.${DOMAIN}

The workaround is to place all repository content under the root app templates folder, but this looks weird.

Today I installed ArgoCD and trying to use variables for apps ingresses, that was a BIG surprise that ArgoCD dosn`t support such function.

joebowbeer commented 4 months ago

I suggest either closing this issue (as dup of #9165) or removing kustomize from the description.

kustomize does not by design provide a way to use build-time side effects from env vars or CLI args:

https://kubectl.docs.kubernetes.io/faq/kustomize/eschewedfeatures/#build-time-side-effects-from-cli-args-or-env-variables

A misfeature was even removed: https://github.com/kubernetes/website/issues/35669

The devs would like to remove all vars from kustomize once their capability can be achieved through other means: https://github.com/kubernetes-sigs/kustomize/issues/4049 (More discussion: https://github.com/kubernetes-sigs/kustomize/discussions/5046)

With respect to ArgoCD, I think a lot of this capabily can be achieved using templated ApplicationSet generators.

zs-dima commented 4 months ago

@joebowbeer the point is that capability can NOT achieved through other means. therefore FluxCD provides kustomizations variables substitution: https://fluxcd.io/flux/components/kustomize/kustomizations/#post-build-variable-substitution

joebowbeer commented 4 months ago

@zs-dima The description mentions using env vars with kustomize, which confused me.

What Flux supports is post-build substitution. This is an extra stage after kustomize build. (Not something kustomize build can/will do.)

kustomize build ./apps/ | flux envsubst --strict

Flux also supports post-render processing for helm. (See #3698 for ArgoCD RFE.)

I think a better title for this issue would mention post-build substitution.

What ArgoCD does provide at this time is CMP; the lovely plugin can perform post-build substitution:

https://github.com/crumbhole/argocd-lovely-plugin