Managing multiple apps

[lmakarov]

If you had more than a single app to deploy into a cluster, where would you put the manifests? I've been scratching my head for a while with this and I don't see a clear approach using the current repo structure.

[kingdonb]

I think you really want to maintain them together with the app, follow the Flux components as example:

https://github.com/fluxcd/flux2/blob/fa46f05423a237a79d9f66f2bee8dec759d630be/manifests/install/kustomization.yaml

If apps are being developed to run on Kubernetes, they should be distributed in each release version with manifests to apply. Following the flux controllers example can't go wrong. If you are installing third-party apps, then it's a choice of whether to vendor manifests or refer as remote bases, similar to what Flux v2's installation kustomization above linked does mapping the constellation of Flux controllers together.

If your third party apps are all managed through Helm, you could create a base for helm installations

Could put everything in a folder for itself in apps, could put more remote app bases in the cluster/apps/base directory next to Podinfo from this example.

Could put those folders under each environment, if you maintain all those apps in staging as well as production. Maybe your cluster repo doesn't integrate staging and production, then you can do something else different.

The answers are going to depend on a lot of things. Namely, how many apps are both yours and in development, and perhaps also whether they are in a constellation together, or just neighbors. More complex examples might not just use Kustomize, but also enable some kind of elaborate (build-time) manifest generation like Jsonnet or kubecfg, as is being proposed in some new use case documentation I've written for fluxcd/flux2#1200.

How badly do you want to keep your configuration from repeating itself, and how many different environments do you need to manage within those parameters? Kustomize itself is some help here, but perhaps not the best tool for this job by itself.

The idea is that maybe some heavy lifting needed to keep your Kubernetes configs expressive should be done in a separate layer, in CI, and accordingly so also at release time.

[azkore]

I am not sure if my question is the same or just similar, but I need to install multiple copies of the same app in some of the clusters. So some clusters will have only one copy of the app, while some others two or three of them. Can I do that with the approach demonstrated in this repo?

[mtb-xt]

@kingdonb I have the same question, and I think maybe you misunderstood @lmakarov here - there's an example how to add 'podinfo' to base apps, and then to kustomize it for different clusters. What is the workflow to add another app here? For example - I want to add minecraft server. Where would we add minecraft directory and how do we add it's release, kustomizations and such?

[kingdonb]

I'm not sure there's enough information here about your repo structure for me to provide a definitive answer.

If you had more than a single app to deploy into a cluster, where would you put the manifests?

There are effectively two places you can put things in the basic Flux cluster after bootstrap according to the getting started guide: inside of the clusters/[name] directory, or outside of it.

If you are aiming for multiple deploys across different clusters, then you're going to need to add something to each cluster somehow either directly, or indirectly. One pattern is to add a new base directory in the cluster repository /apps/ and add a base app there for each app you wish to deploy. Then, in each cluster directory, add a reference to the app and customize it with things that are unique to that environment using a patch (like for example, ingress hostname, or number of replicas).

Flux is not prescriptive about repository structure, but the Repository Structure guide tries to provide a comprehensive workable guidance. If it seems complex to understand, it's because it builds on topics from earlier guides. In the first successive example after the Getting Started guide:

https://github.com/fluxcd/flux2-kustomize-helm-example

It's assumed that Production and Staging will both be running on a cluster, but flexibility is granted as to which cluster. The scenario is for multiple clusters and so, production and staging are instantiated from a base directory, so they can avoid repeating common sections.

So, in the apps directory, there are two Kustomize bases staging and production, and in the base directory, you find one app (podinfo) which is instantiated in both environments with a kustomization.yaml that refers to the app in the base, and a patch to apply that updates some values in the associated HelmRelease.

If you don't have apps that need to be repeated with minor variations for multiple environments, then a simpler configuration is possible, but that was the question I think was being asked: how to spread one release across multiple environments efficiently. It's covered by this tutorial guide. (Please read the README in that repository for a fuller understanding.)

If you needed to add minecraft then it would also go in base, and you would include it in your environments in a similar way to how podinfo is included in production from the base and how the apps are all included at once in a production cluster. If you only need one minecraft then you can skip the base, and put it directly in production directory, but even then I'd consider using a base, as you might think you need only one today but in the future, it might be useful to have the configuration separated into "bare minimum to run this app" and "config that is specific to my instance of the app" which is essentially what's provided for by this pattern.

[mtb-xt]

@kingdonb thank you so much for a detailed answer! We're getting closer! One more question, that's making me go nuts!

f you needed to add minecraft then it would also go in base, and you would include it in your environments in a similar way to how podinfo is included in production from the base and how the apps are all included at once in a production cluster.

I'm struggling with the part - how podinfo is included in production from the base - how do I add another app here? I tried doing this, and for some reason, flux ignores it completely :/

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: podinfo
resources:
  - ../base/podinfo
patchesStrategicMerge:
  - podinfo-values.yaml
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: minecraft
resources:
  - ../base/minecraft
patchesStrategicMerge:
  - minecraft-values.yaml

Is there some other way I can specify two apps, each with it's own patch? Only the first section works. If I swap them around, then minecraft works, but podinfo disappears :crying_cat_face:

[mtb-xt]

Ok, I figured it out, I was using kustomize wrong.

For posterity, the way I got it to work is

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - podinfo
  - minecraft

and then I created 2 dirs in cluser apps directory - podinfo and minecraft with their respective files and kustomizations.

@kingdonb sorry for thread-hijacking and thank you for your detailed explanation!

[darioblanco]

I stumbled upon this post as I had the same use case and I have pushed the multi-app approach to https://github.com/darioblanco/gitops

It might help somebody who wants to achieve the same.

[ametad]

This would work for not very complex cases:

apps/staging/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../base/podinfo
  - ../base/gitlab-agent
patches:
  - path: podinfo-values.yaml
    target:
      kind: HelmRelease
      name: podinfo
      namespace: podinfo
  - path: gitlab-agent-values.yaml
    target:
      kind: HelmRelease
      name: gitlab-agent
      namespace: gitlab-agent

[mischavandenburg]

This has been very useful as I'm learning Flux and adopting it for my homelab. Thank you for sharing all the links.