[Question] Multiple environments(dev, stage, ..,. prod ) example

[emirot]

Hi there,

I'm getting started with Argocd. I was able to run great examples, but I don't understand how multiple environments (dev, stage, prod) are managed.

Do I have to create an application for each environment ?

• I've seen some app.yaml like https://github.com/argoproj/argocd-example-apps/blob/master/ksonnet-guestbook/app.yaml.

[taybur43]

Please provide a doc describing details of Managing multiple env in argocd.

[davenmth]

Simple question and no answer in 10 months. The app of apps pattern works in most cases pretty good. You need to create a parameterized app template and then the values files for each environment. The biggest pain is that the names must be global galactically unique, which means you cannot deploy an app to multiple environments without modifying its name.

[yangvipguang]

any answers?

[viyorv]

It would be nice to get an answer from argocd

[danghung-dev]

I also need an answer

[ayushkr04]

do we any updates on this one? How to manage multiple environments with ArgoCD.

[kostis-codefresh]

Hello.

The reason that this question has been not answered, is that there is no standard answer. Most companies do something different and either have an ad-hoc solution or something with an external system or a custom thing over app-of-apps, appset, argo-autopilot etc.

As an example, I could provide you with a detailed answer of what we do to address this, until you realize that all my environments are different folders on a single monorepo and my solution depends on this choice. If you have chosen to have different branches per environment for your own case instead, then all my advice is useless to you.

And then let's say that you are lucky and I also follow the branch-per-environment practice in git. But then we find out that I am doing git-flow development while you are doing trunk-based development.

And then we have to answer this question for Helm, for kustomize, for plain manifests, for people with declarative setup, for people with application sets etc. The possible scenarios get quickly out of hand.

By the way this is not a problem specific to Argo CD. It is a problem with GitOps in general. See points 3 and 4 here https://codefresh.io/about-gitops/pains-gitops-1-0/

The GitOps working group is now formed (check https://opengitops.dev/) and will soon address all these questions (and more) and establish best practices and standards.

[qxmips]

anyway would be helpful to have an examples directory with various senarious

[kostis-codefresh]

@qxmips It is already on my TODO list. Stay tuned :-)

[kostis-codefresh]

@qxmips , @emirot , @taybur43

Here is the example repo https://github.com/kostis-codefresh/gitops-environment-promotion

Article with more details about the suggested structure https://codefresh.io/about-gitops/how-to-model-your-gitops-environments-and-promote-releases-between-them/

Article about what to avoid https://codefresh.io/about-gitops/branches-gitops-environments/

Any feedback welcome.

[taybur43]

Thanks a lot

On Wed, 23 Mar 2022, 6:01 pm Kostis (Codefresh), @.***> wrote:

@qxmips https://github.com/qxmips , @emirot https://github.com/emirot , @taybur43 https://github.com/taybur43

Here is the the example repo https://github.com/kostis-codefresh/gitops-environment-promotion

Article with more details about the suggested structure https://codefresh.io/about-gitops/how-to-model-your-gitops-environments-and-promote-releases-between-them/

Article about what to avoid https://codefresh.io/about-gitops/branches-gitops-environments/

Any feedback welcome.

— Reply to this email directly, view it on GitHub https://github.com/argoproj/argocd-example-apps/issues/57#issuecomment-1076295532, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKMMWAFT3HA7X6W6GTSPDATVBMBY5ANCNFSM4O3I3OOA . You are receiving this because you were mentioned.Message ID: @.***>

[ReubenTheDS]

@kostis-codefresh , very informative articles and repo, thank you. I'm in the process of implementing what you've suggested, in my employers' projects. I'd like some clarifications on the following points:

• If I understood the code & articles correctly, it seems like you suggest using 1 docker image for all environments. For example "simple-env-app" will be used across staging, qa & prod envs. Is this correct?
• is the above point in line with the concept of artifact promotion? If not, please elaborate or link an article.
• If I'm using Jenkins for CI, are the following steps for release mgmt correct?
• • testing in QA: CI tool (like GitHub Actions, GitLab runner or Jenkins) runs the command "kustomize edit set image ..." on "env/qa/version.yaml" ... testing in QA commences ...
• • for promotion to staging: the CI tool copies "env/qa/version.yaml" to "env/staging/version.yaml" - the only confusion here is do I have multiple Jenkinsfiles for each env? If I want to use just one Jenkinsfile for both the above envs & tasks, how would I go about doing that? Or have I completely misunderstood all of this (do you have an example repo)?
• Separately, let's say that the application in different regions use different CA certificates. How do you suggest this is handled? Does the Dockerfile (& therefore the image) load all the certificates & then use ENV variables? ( CA cert might be one region-specific example, there might be other settings. Does the image load all the settings & use just ENV variables for distinguishing? )

[kostis-codefresh]

Hey @ReubenTheDS

Thank you for the feedack.

1) Yes the same docker image should be promoted to all environments (but with different configuration). Actually this was a good practice already before Kubernetes/Gitops and goes all the way back to the Heroku guidelines 2) Yes it is. I didn't cover this on the blog post because I have already written about it elsewhere. Check points 5 and 8 on the Docker anti-patterns article, point 2 in the Kubernetes anti-patterns article and point 2 in the CI/CD best practices article

3) Yes these are correct. I don't use Jenkins for some time now, but if I remember correctly you can have a Jenkinsfile with parameters. So you should create a single Jenkinsfile with parameters for source and target environment. If you have a small number of environments then I guess you could also use multiple Jenkinsfiles and it should work just fine. You need to check the Jenkins 2.x documentation. I don't think this question is related to GitOps/ArgoCD

4) Very good question. I have seen both practices in the field. I personally suggest the container does NOT include the CA certificates for all possible configurations. You can fetch them during deployment. There are many ways to do this in Kubernetes such as using mounted volumes/init containers. Again this question is not related to ArgoCD.

[Jaykob]

@kostis-codefresh Thanks for these really helpful recommendations and your guide!

Can you please elaborate a bit on how the single-docker-image approach mentioned above in answer 1. is supposed to work during active development of a product? I mean we usually add features and they'll end up in a staging environment at some point while production is still running a stable version of that docker image. How are we supposed to use the same image for staging and production in my example?

[kostis-codefresh]

@Jaykob Not sure I understand what you are asking. The "staging" image should be promoted to production at some point. And then both envs are the same.

Or are you saying that you have some features that go to staging and stay there? And they never go to production?

The original best practice doesn't say that you should be running the same image at all env AT ALL TIMES. It simply says that you should create an image once and then promote it to all environments gradually (instead of building a new image for QA, a new image for staging, a new image for production and so on)

[Jaykob]

Ah ok sorry, I misunderstood you on that point. I thought you were telling us to always use the same image (version) across all environments and was seriously asking myself how that's supposed to work. I was over reading the very important word "promote" here. Again, sorry.

BTW: Do you have good reading recommendations about doing all that with ApplicationSet?

[kostis-codefresh]

There isn't a guide similar to mine yet for applicationsets (to the best of my knowledge)

But Argo CD itself is adding features to help there. See initial support here https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Progressive-Syncs/

[kostis-codefresh]

@Jaykob Since several people asked me the same thing I also wrote a guide on how to use Application Sets. You can find it here https://codefresh.io/blog/how-to-structure-your-argo-cd-repositories-using-application-sets/

[Jaykob]

@kostis-codefresh Your guide is awesome and answers so many important questions. Thank you! 🙏

[aimuz]

When I add a new deployment should I add it inside the base or should I add it somewhere.

For example, I have a microservices application with three services currently running in the production environment. At this point in time, due to business development, a new service is added.

If I add it to the base it will be deployed in all environments. However, I just want to deploy it in the test environment. I will only move to the next stage when I have completed the deployment in the test environment.

---

Single branching is great, but what should I do if I run into a situation where I need to roll back a production branch deployment? Should we care about git logging.

This is because all the changes are in a single branch, whether they are for the test environment or the qa environment. At this point, the git log should be so large that I can't tell which changes are in production!

Please feel free to correct me. Thank you.

[kostis-codefresh]

If I add it to the base it will be deployed in all environments

That doesn't happen with the setup I propose in the article. The base folder is not deployed anywhere on its own. Only overlays are in the respective environments.

Single branching is great, but what should I do if I run into a situation where I need to roll back a production branch deployment?

You run "git revert". Or even better use Argo Rollouts.

This is because all the changes are in a single branch, whether they are for the test environment or the qa environment. At this point, the git log should be so large that I can't tell which changes are in production!

Not sure what you mean by "changes". The GitHub repo has only Kubernetes manifests. So commits doesn't say anything about source code changes. Do you mean something else?

Feel free to join the argo-cd channel in the CNCF slack instance for even faster communication.

[aimuz]

@kostis-codefresh Thank you so much, I re-read your history article and it solved my problem.

When I apply to my production I have a new problem.

For example, how should I handle things like database passwords? Is it not a best practice to store it directly in the git repository?

I would like to know how you deal with this kind of problem? How do I securely use database passwords?

[kostis-codefresh]

The topic of secrets is very popular and there are already several great sources about it.

You can store the secrets in Git but in encrypted form using Sealed Secrets. Guide here https://codefresh.io/blog/handle-secrets-like-pro-using-gitops/

Or you can store them outside of Git and fetch them from there. Guide here https://codefresh.io/blog/gitops-secrets-with-argo-cd-hashicorp-vault-and-the-external-secret-operator/

There are many more options for secrets of course. https://argo-cd.readthedocs.io/en/stable/operator-manual/secret-management/

[aimuz]

Thank you very much @kostis-codefresh

You've helped me out a lot.