Matrix generator should accept more than two child generators

[huynhsontung]

Summary

When working with ApplicationSet, I encountered a situation where I wish the matrix generator had supported more than two child generators. Like the current matrix generator, it should generate all possible parameter combinations for all child generators. This would allow the matrix generator to handle more complex combinations without having to use nested matrix generators.

Motivation

My current app structure requires me to generate one Application per app config per cluster. An app config is an entry in an array. That means combining the outputs of Git generator, List generator, and Cluster generator. Having a matrix generator that supports more than two children would make this easier to set up.

Proposal

For example, consider we have two clusters:

• cluster1
• cluster2

Given this Git folder structure:

• argocd/app1/config.yaml
• argocd/app2/config.yaml

The content of each config.yaml is:

configs:
  - configName: config1
     custom: value1
  - configName: config2
     custom: value2

Now we want to combine all the above parameters using Git generator, List generator, and Cluster generator:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cluster-git
spec:
  goTemplate: true
  generators:
    # matrix 'parent' generator
    - matrix:
        generators:
          # git generator, 'child' #1
          - git:
              repoURL: <<URL>>
              revision: HEAD
              files:
                - path: argocd/*/config.yaml
          # list generator, 'child' #2
          - list:
              elements: []
              elementsYaml: "{{ .configs | toJson }}"
          # cluster generator, 'child' #3
          - clusters:
              selector:
                matchLabels:
                  argocd.argoproj.io/secret-type: cluster

The Matrix generator should output:

# Some params are omitted for brevity
- name: cluster1
  path: argocd/app1/config.yaml
  configName: config1
  custom: value1
- name: cluster2
  path: argocd/app1/config.yaml
  configName: config1
  custom: value1
- name: cluster1
  path: argocd/app1/config.yaml
  configName: config2
  custom: value2
- name: cluster2
  path: argocd/app1/config.yaml
  configName: config2
  custom: value2
- name: cluster1
  path: argocd/app2/config.yaml
  configName: config1
  custom: value1
- name: cluster2
  path: argocd/app2/config.yaml
  configName: config1
  custom: value1
- name: cluster1
  path: argocd/app2/config.yaml
  configName: config2
  custom: value2
- name: cluster2
  path: argocd/app2/config.yaml
  configName: config2
  custom: value2

For template handling, when going through the child generators, the current generator should have access to the combined params of all the generators before it. Using the above example, List generator should have access to all Git params (like the current implementation), and the Cluster generator should have access to both Git and List params.

I can raise a PR later.

[crenshaw-dev]

I'm not opposed! I think the limit of 2 was for the sake of implementation simplicity. I don't know of any technical reason it can't be arbitrarily large.

[ptr1120]

Would be really great to allow matrix generator to have more than 2 children. I regularly hit this limitation and have to build nested matrix generators, which also causes other problems. Is there anything missing in the existing pr #14189?

[mihaigalos]

Hi, any news here? The linked PR's comments seem addressed.

[twanbeeren]

What is the preferred way to tackle this? Since it's been build by design for "simplicity"

In my case I would like to use two git generators and one scm generator in the same matrix

[juanmancebo]

Would be really good to have this. In my case I need something like this:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cluster-apps
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - matrix:
        generators:
             - scmProvider: {}
             - matrix:
                 generators: 
                     - clusters: {}
                     - git:
                         repoURL: '{{ .url }}'
                         revision: '{{ .branch }}'
                         files:
                           - path: '.deployment/argocd.yaml'
                     - list:
                         elements:
                           - appEnabled: '{{ dig "clusters" .name "enabled" false . }}'
      selector:
        matchLabels:
          appEnabled: "true"
  template: {}
  templatePatch: {}