Allow users to template and render Application's resources

[noaabarki]

Summary

As a user, I'd like to be able to inspect my resources before applying them. 
One use case would be to debug, lint and verify Helm Charts before deploying them. Another use-case would be to validate resources(plain or via Helm/Kustomize) for misconfigurations and ensure they comply with pre-defined policies. 
To do so, I need to be able to template and inspect the applied resources right before they are being applied without affecting cluster.

Motivation

When working with ArgoCD it can be very difficult to template and output the objects that will be applied for inspection. One example is when using App-Of-App and ApplicationSet, these patterns delegate the majority of the templating work to Argo and therefore, make it harder to properly inspect the objects that eventually will be applied to the cluster. Another example would be using Helm Charts and overriding values.yaml in an Application manifest. As a user, I can't create templates for my resources since Argo is responsible for templating them. 



This feature will allow any user keep production-clusters stable and secured while shifting-left validation processes to an earlier stages. 

 

 

Additionally, in Datree, we see more and more companies that uses ArgoCD and look for ways to prevent misconfigurations in the CI. As one of Datree’s maintainers, I’ll be happy to partership and assist with the work to help both out users and the entire ArgoCD community validate their resources more easily, prevent misconfigurations.

Proposal

• This should be added as a feature flag.

  argocd app template <APP_NAME>

  argocd app render <APP_NAME>

• IMHO, this should not be reflected in the web UI since its mainly used for CI/manual testing.

[crenshaw-dev]

There are a few different use cases here. We support argocd app diff --local, which I think helps with some of the use cases. I think an argocd app manifests --local would completely covert that subset of use cases.

I think the ApplicationSet case is covered here: https://github.com/argoproj/argo-cd/issues/10895

I’ll be happy to partership and assist with the work to help both out users and the entire ArgoCD community validate their resources more easily, prevent misconfigurations.

We'd welcome the help! Please let me know what you think of the adding --local support to argocd app manifests and whether that would solve your use cases. If so, I'd happily provide any guidance to help implement the feature.

[noaabarki]

Hi @crenshaw-dev! Thank you for the quick response and sorry it took me so long to reply. As you said argocd app diff --local does help with some of the use cases but unfortunately not all of them. Anyway, I'm onboard and willing to add --local support to argocd app manifests.Any advice would be greatly appreciated; this will be my first time contributing to ArgoCD, so I'm not sure where to begin 😊

[ShirMonDT]

Hasn't argocd app manifests --local been merged in #10061? A negative point on this is it required a live ArgoCD with the application as mentioned by @Cylix https://github.com/argoproj/argo-cd/issues/5525#issuecomment-1037647091

It also doesn't actually template a local "Application" manifest;

What is required here is for the CLI to receive an application manifest as an input and output the manifests that that application would apply

[jtyberg]

I totally agree with @noaabarki and @ShirMonDT that rendering local Helm charts (without involving the ArgoCD server) would be a useful feature.

We work exclusively with Helm charts, and the typical workflow when developing a chart is to test it locally before you merge and deploy.

I'm currently struggling with ArgoCD NOT rendering a Helm chart properly. helm template (v3.9 or v3.10) renders the chart just fine. When I use an umbrella chart to pull in my chart and template the same values, ArgoCD is not rendering all the manifests. Uh, what? So whatever black magic ArgoCD is doing, it's not the same as helm template with the version I'm using. So now I'm stuck in a world where Helm template rendering works just fine, but ArgoCD rendering does not.

We use a strictly declarative model, so for me to "test" ArgoCD rendering, I have to go through the PR cycle. I would much prefer that the ArgoCD CLI render my chart locally, using the same logic the server would, so that I know ArgoCD will handle my chart properly.

[jutley]

This would be a valuable feature for me as well. We don't want our CI pipeline to have direct access to ArgoCD's API, and we can fetch all Application definitions since they are stored in git. Having an extra flag like --local-application-manifest would mean that our CI would generate all manifests used by an application based on PR content.

[prein]

Is my understanding correct, that the rendered template this is meant to produce will be identical with the template rendered on the server only if no config plugins are used? If so, is there or can there be a way to render templates for review outside of the server with config plugins used?

[mmckane]

From a security perspective this would be a great addition to the project. Allowing local generation of manifest will allow you to utilize security or IAC tooling to scan the manifests for mistakes or issues and fail the deploy process before the manifests ever touch your clusters. TLDR: You shouldn't have to push an application to an argocd/cluster to be able to generate and then scan manifests, at that point you have already pushed either bad code or vulnerable configurations to the cluster.

[Justin-DynamicD]

Adding another voice, as there are lots of IaC scanning tools that simply assume you can provide a final manifest.

To add another wrinkle, I don't even see a manifest option for appsets, which should also have this support.

[reegnz]

I'm simply rendering my manifests during PR time and have the CI job commit the rendered manifests to the PR. That way we can inspect the rendered manifests before we merge it. We only use ArgoCD to sync a folder with already rendered manifests in it. This also simplifies how argocd needs to be deployed, as we don't have to ship a complicated set of tools into argocd to render manifests with (sometimes your helm values are dynamically resolved, not static files in a repo, so setting up all of that inside argocd opens up a ton of security questions). Whatever scanning tool we need to apply we can add it as a PR check and fail the check if the manifest is non-compliant.

This doesn't work of course if your rendered manifests contain sensitive info (like secrets), but that's what tools like SealedSecrets are for.

In my experience rendering the manifest can be a regular CI step, doesn't need to be performed during CD, and it's actually much more flexible (and much simpler) to render manifests into an open PR.

Some other aspects to consider:

• Network path to the production argocd instance might not be opened up to your CI, so a jenkins job, a github action, etc. can't necessarily interact with the argocd instance to speculatively render manifests. This is usually a hard security requirement. So calling out to the production argocd instance to do a speculative rendering is likely not possible in highly regulated environments.
• Doing speculative rendering from an unapproved PR inside a prod argocd instance is a potential attack vector, people could (intentionally or accidentally) open PR-s that break argocd in unexpected ways and might facilitate a container escape. Running rendering for a PR in CI with zero access to prod only runs the risk of compromising the CI environment, never prod directly.
• Shipping extra tooling into your production (eg. helm, custom scripts, and tools that those scripts are using) requires those tools to be secured, they're additional points to control and additional risk in a production environment. The fewer tools are needed to be deployed to prod the better your security posture.
• doing the work upfront speeds up the deployment, as the rendering has already been done in advance.

Anyway, I thought sharing alternative use-case would be helpful for people. I think it would also definitely help a lot of people if API access is absolutely not required to inspect the rendered manifests, although fully offline, without storing the previously rendered manifest I find it a bit more difficult to produce a diff (unless you're codifying your branching strategy somehow and extract the previous config from git too).

[krokofant]

I think this would also help the case where one keeps an ArgoCD dependency, like SealedSecrets, updated via ArgoCD itself - but you want to be able to do an initial deploy outside of ArgoCD. Being able to render the manifest of a locally defined SealedSecrets argo app would make this easier.