Agent: Generator functionality

[juliev0]

Summary

Implement the list generator and file generator functionality for Agent, which can be used to evaluate a helm chart and apply it.

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[juliev0]

FYI @xdevxy my plan is that we can have a single file that the Agent ConfigMap generator and the multicluster GitSyncSet can both read from, stored in the Control Repo.

My thought was that the generator could actually be a map of cluster name to the keys/values pertaining to that cluster. So, essentially both the Agent and the Numaplane Controller would find their own cluster name, and then get the keys/values associated with it. One of those keys would be the "wave" for the cluster, which would serve as the RepositoryPath.

Also, ultimately this file would need to be automatically created by the "cluster creation" team, so all fields/values would be automatically derivable from IKSM. It would have to use simple logic to determine "wave" based on things like ppd vs prod, etc. Derek would also like an "override file" where we could put clusters into certain waves that they wouldn't automatically be designated to. I'm thinking that if we allow multiple generators in a list, then maybe generators in the list can override previous generators. Any thoughts or objections?

[xdevxy]

Can you present a concrete example for that file to walk through the flow? Thanks!

[juliev0]

Can you present a concrete example for that file to walk through the flow? Thanks!

So, each of the files (the auto-generated one plus the override file) would be formatted something like this:

{
    "staging-usw2-k8s":
        [
            "cluster": "staging-usw2-k8s",
            "numaflow_ui_arn": "arn:aws:acm:us-west-2:432219235508:certificate/d3875b2c-79da-4c9c-b33a-59b023bbac74",
            "numaflow_ui_host": "numaflow.stagingusw2.iks2.a.intuit.com",
            "numaplane_ui_arn": "arn:aws:acm:us-west-2:432219235508:certificate/d3875b2c-79da-4c9c-b33a-343534234324",
            "numaplane_ui_host": "numaplane.stagingusw2.iks2.a.intuit.com",
        ],
    "staging-use2-k8s":
        [
            "cluster": "staging-use2-k8s",
            "numaflow_ui_arn": "arn:aws:acm:us-west-2:432219235508:certificate/d3875b2c-79da-4c9c-b33a-950k5904i44",
            "numaflow_ui_host": "numaflow.staginguse2.iks2.a.intuit.com",
            "numaplane_ui_arn": "arn:aws:acm:us-west-2:432219235508:certificate/d3875b2c-79da-4c9c-b33a-8j88845233",
            "numaplane_ui_host": "numaplane.staginguse2.iks2.a.intuit.com",
        ],

}

Basically, the map allows us to look up our own cluster. Then the key/value pairs that are associated with it are what populate either the GitSyncSet spec or the Agent source, each of them replacing any instances of {{......}} with the corresponding value.

If there is more than one file, then the latter files always override the previous files.