feature request - combine generators

[1337andre]

Hi folks,

first, thanks for this great project!

Would it be possible to combine generators?

in this example combine directory discover with clusters secret

example:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cluster-addons
spec:
  generators:
  - git:
      repoURL: https://github.com/infra-team/cluster-deployments.git
      directories:
      - path: add-ons/*
  - clusters:
     selector:
       matchLabels:
         argocd.argoproj.io/secret-type: cluster
       values:
         project: default 
  template:
    metadata:
      name: '{{name}}-{{path.basename}}'
    spec:
      project: '{{values.project}}'
      source:
        repoURL: https://github.com/infra-team/cluster-deployments.git
        targetRevision: HEAD
        path: '{{path}}'
      destination:
        server: {{server}}
        namespace: '{{path.basename}}'

When I try, run into

error converting YAML to JSON: yaml: invalid map key: map[interface {}]interface {}{"server":interface {}(nil)}

[OmerKahani]

Hi @1337andre what would be the expected behavior this case? which application would you expected to be created?

[1337andre]

given:

NAME                                                                                     TYPE     DATA   AGE
cluster-XXX.gr7.eu-central-1.eks.amazonaws.com-3203871447   Opaque   3      232d
cluster-kubernetes.default.svc-3396314289                                                Opaque   3      138d
cluster-pro.XXX.de-1583813391                                               Opaque   3      267d
cluster-stg.XXX.de-786650180

and

# cluster-deployments
# └── add-ons
#     ├── argo-rollouts
#     │   ├── all.yaml
#     │   └── kustomization.yaml
#     ├── argo-workflows
#     │   └── install.yaml
#     ├── grafana
#     │   ├── Chart.yaml
#     │   └── values.yaml

expected behaviour would be the creation of 4(clusters)*3(directories) = 12 applications

appname= {{name}}-{{path.basename}}

[OmerKahani]

Cool use case.

I think this is a special case for the cluster generators. List and git or two git generators will not behave the same way.

Can we define it like this: If there are multiple generators and one of them is the cluster generator. Then the input for the cluster generator should be the output from the other generator.

Does this make sense?

[jgwest]

The original design of the generators field appears to be that generators are additive, eg that you can list multiple generators in the generators field an ApplicationSet, and each individual generator will add parameters to be templated. (eg 2 git generators, a and b, would produce a+b parameter sets)

The change discussed here would need to be more sophisticated, where now if multiple generators are listed they are multiplicative (2 git generators, a and b, would produce a * b parameter sets). Since I presume we want to support both additive (current behaviour) and multiplicative generator usage (proposed behaviour), we would probably need to carefully consider how to change the ApplicationSet spec.

[OmerKahani]

I experimented with two generators:

1. Two lists:

   spec:
   generators:
   - list:
    elements:
      - cluster: engineering-dev1
        url: https://kubernetes.default.svc
        values:
           foo: bar
   - list:
    elements:
       - cluster: engineering-dev1
         url: https://kubernetes.default.svc
   template:
   metadata:
    name: '{{cluster}}-guestbook'
   spec:
    project: '{{foo}}'

   gives an error: ApplicationSet guestbook contains applications with duplicate name: engineering-dev1-guestbook

I think that the in the current state two generators produce two lists of separate applciations

[OmerKahani]

Drafting a couple of examples https://github.com/argoproj-labs/applicationset/pull/165 @1337andre what do you think?

[jgwest]

@OmerKahani @1337andre

How about this, let me know what you think: lets implement a new 'merge' generator which combines the parameters of child generators... this has a few advantages:

• Allows both additive and multiplicative parameter generation, without needing to change the existing spec
• Maintains a tree like structure (which differs from #165 examples)
• Powerful, as it allows introduces the concept of a generator tree, eg allowing generators to contain generators (to contain generators), which should allow for flexibility in creating complex applicationsets

Here's an example:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cluster-git
spec:
  generators:
  - merge: # new combine/merge generator (need a good name for this), which combines values
    - git:
        name: cluster-deployments
        repoURL: https://github.com/infra-team/cluster-deployments.git
        directories:
        - path: add-ons/*
    - clusters:
        dependsOn: cluster-deployments
        selector:
          matchLabels:
            argocd.argoproj.io/secret-type: cluster

This new 'merge' generator would take as input an array of generators, and mutiplicatively combine the values of them. So if the merge generator had 2 generators, A and B... A produced 3 parameter sets (abc) and B products 3 parameters (def), the merge generator would produce: { ad, ae, af, bd, be, bf, cd, ce, cf}

I'm not sure what we would call the new generator, but here are my thoughts:

• merge
• combine
• dot product
• product
• multiply
• multiplier
• permutation
• other ideas?

[joshm91]

How about "matrix"? I've seen this term used in various test runners before and Github Actions have a similar concept to what you're proposing here.

[WillooWisp]

I'm looking for a way to combine git directory and file generators together, so they merge. I want applications to be generated based on directories in a folder, but I want to specify additional properties for each application in a config file, e.g. namespace overrides per application and other specific settings. Is that possible today or something that this merge generator would solve?

The use case is a mono repo for different environments / clusters (different AKS resources) that should all have the same applications but different properties depending on the environment.

[jgwest]

Hi @WillooWisp , for the current release (0.1.0) the closest you can get for this scenario is the Git file generator, which would get you most of the way there (but you would need to define config files for each directory, unlike the Git directory generator which will automatically identify directories).

When the Matrix generator is merged (and released), you would be able to use it to combine Git file generator and Git directory generator: in this scenario you would use the define config file for each different environment (cluster) for the Git file generator, and rely on Git directory generator to locate directories in your monorepo.

The end result would be a templated Application for each (repo directory matching Git directory filter, parsed configuration file values) pair.

[annabarnes1138]

I am also looking to solve a similar scenario as @WillooWisp. Any estimate on when this might be available?

To be clear, I am trying to avoid the app-of-apps pattern without having to apply each app separately:

Here is what I am thinking for my structure (which is very similar to the git-generator-files-discovery example)

├── apps
│   ├── app1
│   │   ├── Chart.yaml
│   │   └── values.yaml
│   └── app2
│       ├── Chart.yaml
│       └── values.yaml
└── clusters
    ├── dev
    │   ├── global.yaml
    │   ├── app1.yaml
    │   └── app2.yaml
    └── prod
        ├── global.yaml
        ├── app1.yaml
        └── app2.yaml

global.yaml

global:
  environment: dev
  clusteraddress: http://1.2.3.4
  domain: dev.mycompany.com
  lang: en

app1.yaml

name: dev-app1
values:
  dbHost: somehostname
  dbPassword: pass1234

app2.yaml

name: some-custom-name
values:
  extApiUrl: http://some-external-api.com
  extApiKey: my-key

and my ApplicationSet might look something like this

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: guestbook
spec:
  generators:
  - git:
      repoURL: https://github.com/test/applicationset.git
      revision: HEAD
      directories:
      - path: apps/*
  - git:
      repoURL: https://github.com/test/applicationset.git
      revision: HEAD
      files:
      - path: clusters/**/{{path.basename}}.yaml
  template:
    metadata:
      name: '{{name}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/argoproj-labs/applicationset.git
        targetRevision: HEAD
        path: '{{path}}'
        helm:
          releaseName: '{{path.basename}}'
          values: |
            {{global}}
            {{values}}
      destination:
        server: '{{global.clusteraddress}}'
        namespace: '{{name}}'

[WillooWisp]

Any plans for when this matrix generator will be released?

[jgwest]

@WillooWisp see https://github.com/argoproj-labs/applicationset/discussions/225 for discussion of release cadence.