Graduating ApplicationSets, and shipping it with ArgoCD

[sbose78]

Greetings @OmerKahani @dgoodwin, all contributors & users of argoproj-labs/applicationset .

Thank you for the amazing work on Application Sets. We love it! I'm Shoubhik, an ArgoCD ( and ApplicationSets) user and a member of the Argo steering committee. I, as a member of the ArgoCD community would like to propose graduating this to https://github.com/argoproj and eventually ship it with ArgoCD.

I wanted to provide you a heads-up of the conversations that are happening on this subject in the Argo Contributor Meetings :)

• We've started working on a proposal led by @jgwest ( who's recently ramped up his contributions )

• We brainstormed different ways to initiate the integration into the ArgoCD stack( same process, multiple processes, etc )

• TODO: Decide on the initial integration approach.

• TODO: Finalize workitems items based on the integration approach chosen.

Would you be able to join the "Contributor office hours" (https://calendar.google.com/calendar/embed?src=argoproj@gmail.com) which happen at 1 PM US East Coast time every Thursday ? If not, no worries, I will ensure you have the meeting notes in this Github thread so that we can track the transition.

[dgoodwin]

I apologize I can't make it today, but I can start attending in the new year if that is helpful.

[sbose78]

No worries, thank you for the note. See you soon!

[jgwest]

As discussed at today's meeting, Thursday December 17th:

ApplicationSet Integration

Project Logistics

• We'll begin the process to graduate ApplicationSet repository from argoproj-labs/ to argoproj/, likely in the new year (will just be an 'Argo CD ecosystem' project, eg not a top-level Argo project)

Focus on repository QoL improvements, automated testing (unit/E2E), and documentation

• I've opened a set of new issues that aim to get AppSet ready for closer Argo CD integration
  • Adding new E2E tests (tests that target live k8s/argo cd) for supported generators
  • Investigate unit test coverage data, to identify if there are any unit test improvement areas
  • Writing additional end-user focused documentation, based on (and adding to) the great design proposal docs that folks have already written
  • Some additional GitHub commit/PR checks to improve repo QoL for devs/reviewers working in the codebase
  • For those reading this, if one of the issues jumps out at you, feel free to take a look!

Initial integration strategy

• Complete the code quality and testing, as above

• Complete the items required on the Argo CD 1.9 side, for ApplicationSet

  • Add needed API to reposerver in Argo CD, on behalf of ApplicationSet
  • Create a new kustomize configuration in Argo CD repo, that generates a new install.yaml-style yaml, containing a combined install of ArgoCD+ApplicationSet controller (see argo-cd/hack/update-manifests.sh). Allows end users to quickly kubectl apply themselves an install that contains both.

• At this stage:

  • The ApplicationSet codebase will live in argoproj/applicationset, remaining separate from the Argo CD codebase
  • ApplicationSet will ship as part of Argo CD via an optional kustomize-generated install YAML (see argo-cd/manifests) that allows users to easily install ArgoCD+AppSet together
  • It will also be possible to install ApplicationSet against an existing Argo CD install, after the fact
  • ApplicationSet to live in its own k8s Deployment, with a single replica (no need to support cluster sharding akin to application controller, at this time)

Post Release

• Once we've completed initial integration as described, and we've released a release of ArgoCD+AppSet, we will ask the community for feedback to help us validate ApplicationSet functionality with a wider user base
• We've completed initial discussions and gathered everyone's thoughts on whether merge the AppSet codebase into Argo CD, and whether to merge the Argo CD and ApplicationSet k8s artifacts.
  • As above, we are keeping both separate for the time being, and future decisions here will rely heavily on community feedback/usage on AppSet controller, and via shared agreement over the technical merits of integration.

Improving user self-service scenario with ApplicationSet controller

Also discussed at the meeting was @alexmt and @jessesuen's suggestions regarding an additional use case that it would be beneficial for ApplicationSet controller to handle.

There are a couple of user roles that are important here:

• Cluster administrators: Privileged administrators with full cluster access and full git org access. They are responsible for both cluster admin and Argo CD admin (creating Argo CD Applications and Projects for devs)
• Developers (cluster end users): Developers wishing to deploy applications to the cluster, via Argo CD, but not having full cluster access. For example, they might be limited to a particular namespace (and repo) and likely not able to install CRD, cluster roles, etc. Thus they need (in most cases) Argo CD Apps and Projects created for them.

The 'self-service' use case seeks to allow developers (cluster end users) greater flexibility to:

• Deploy multiple applications to a cluster
• Deploy to multiple clusters But, in both cases, empowering developers to do this on their own (by committing changes JSON/YAML manifests, containing AppSet template data, to a Git repo), without needing to involve a cluster administrator to create the Argo CD Applications/Projects on their behalf (as doing so requires privileged access, which devs lack).

Alex raised the question of whether end users should be able to create/update Argo CD Projects using application set controller (a 'project set'), especially in the case where projects didn't exist. For instance, in this case, would ApplicationSet throw an error if a targeted Argo CD Project didn't exist, or it would it create and manage it? Jesse noted that he didn't believe that application sets should span projects, as a general rule.

At present, the project field of an ApplicationSet can be parameterized with values from the generator, allowing one to target multiple projects from a single ApplicationSet.

@jopit noted allowing the cluster/list generator to target multiple applications would be beneficial, as presently these generators are more limited in their scope. Jesse noted that the ability to add labels to cluster secrets would be beneficial here (but that it is not currently supported in Argo CD).

Ultimately, the generators currently implemented should improve this use case to some degree, but the Git file generator (a WIP PR) should provide the greatest power/flexibility for end-users, with additional improvements described above to be considered further in the future.

[rajivml]

@sbose78 @jgwest since, app-of-apps will be eventually replaced with ApplicationSet, will ApplicationSet support application ordering as well i.e. the order in which the applications can be deployed, today the same can be achieved using app-of-apps sync-wave ordering but I didn't find anything about deployment ordering under the ApplicationSet design doc

[jgwest]

Hi @rajivml, the short answer is no, this is out of scope for AppliationSet functionality at this time, but another proposed solution is being tracked in this Argo CD issue.

For a longer answer, see the above link, plus this discussion of relationship between ApplicationSet and sync order in Argo community slack: https://argoproj.slack.com/archives/C014ZPM32LU/p1600274721058300

[rajivml]

thanks @jgwest as long as there is an alternative it's fine to move over to ApplicationSet because for us, Application deployment ordering is a P1 requirement and that's we have onboarded to ArgoCD

[OmerKahani]

Hi, can we have a project / milestone with all the issues from the doc? It will make it easier to follow the status

[jgwest]

I've created a new milestone to track our progress towards Argo CD integration. Within the milestone I've added all the items discussed on this tracking issue, the google doc, and related meetings, plus a few issues that have cropped up since then. If you had an open PR that is in review, I have added it to the milestone as well.

Note that this milestone is focused on tracking Argo CD integration, and should not be seen as gating the merge of items that are not on the list: PRs for other items continue to be welcome.

Finally, the list is fluid (more to be added as needed), so feel free to discuss on GitHub or in Argo community slack if there are additional items that you feel could/should be included.