search

argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
17.59k stars 5.36k forks source link

Two ArgoCD instances in different namespaces mixing Application definitions #18965

Open juanmatias opened 3 months ago

juanmatias commented 3 months ago

Checklist:

Describe the bug

Given a shared K8s cluster with two ArgoCD instances:

Each instance in different namespaces: argocd-devstg and argocd-production.

Both ArgoCD instances were installed using binbash Leverage, (based on this file), in the end using Terraform, Helm provider, with this info:

  repository = "https://argoproj.github.io/argo-helm"
  chart      = "argo-cd"
  version    = "6.7.3"
argocd@argocd-devstg-application-controller-0:~$ argocd version
argocd: v2.10.4+f5d63a5
  BuildDate: 2024-03-18T08:09:23Z
  GitCommit: f5d63a5c77d2e804e51ef94bee3db441e0789d00
  GitTreeState: clean
  GoVersion: go1.21.3
  Compiler: gc
  Platform: linux/amd64
FATA[0000] Argo CD server address unspecified

There is an Application called "applications" which is deployed to both environments, for devstg:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: applications
  namespace: argocd-devstg
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd-devstg
  project: default
  source:
    helm:
      valueFiles:
      - values.yaml
    path: kubernetes/charts/applications-staging
    repoURL: git@github.com:user/webapp.git
    targetRevision: main
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

For productions is the same but changing the namespace to argocd-production and the path to kubernetes/charts/applications-production.

Note these applications deploy other apps to the same namespace in which they were created since we are following the Apps-of-Apps approach.

So, when I deploy the "applications" Applications to argocd-devstg, only the devstg ArgoCD instance picks it up.

But then, when I deploy the app to the argocd-production, and the production ArgoCD picks it up, the definitions in both apps are mixed and both ArgoCD instances start fighting trying to reach the app state of each one.

I was digging into the code and it seems to be that, when ArgoCD selects the Applications to manage, it checks the namespace in which the app lives is allowed to the instance. But it is as if after both instances pick the apps, the definitions are mixed and both ArgoCD instances start trying to change the same App.

To Reproduce

I tested this with two examples got from ArgoCD.

For this, I used from this repo https://github.com/argoproj/argocd-example-apps the examples kustomize-guestbook and guestbook.

Steps are:

  1. Deploy two different ArgoCD instances to the same cluster and different namespaces (e.g. argocd-devstg and argocd-production) using Helm with the given versions.
  2. Create two Apps named the same but pointing to different resources. Here are my templates:
resource "kubernetes_manifest" "google_microservices_dev" {
  field_manager {
    name = "argo_applications"
    force_conflicts = true
  }
  manifest = {
    "apiVersion" = "argoproj.io/v1alpha1"
    "kind"       = "Application"
    "metadata" = {
      "name"      = "microservices-dev"
      "namespace" = "argocd-devstg"
      "finalizers" = [
        "resources-finalizer.argocd.argoproj.io"
      ]
      "labels" = {
        "app" = "microservices"
        "env" = "dev"
      }
      "annotations" = {
      }
    }

    "spec" = {
      "destination" = {
        "server"    = "https://kubernetes.default.svc"
        "namespace" = "demo-microservices-dev"
      }
      "project" = "default"
      "source" = {
        "repoURL"        = "https://github.com/argoproj/argocd-example-apps.git"
        "targetRevision" = "master"
        "path"           = "kustomize-guestbook/"
      }
      "syncPolicy" = {
        "automated" = {
          "prune"    = true
          "selfHeal" = true
        }
        "syncOptions" = [
          "CreateNamespace=true",
          "Prune=true",
        ]
      }
    }
  }
}

resource "kubernetes_manifest" "google_microservices_prod" {
  field_manager {
    name = "argo_applications"
    force_conflicts = true
  }
  manifest = {
    "apiVersion" = "argoproj.io/v1alpha1"
    "kind"       = "Application"
    "metadata" = {
      "name"      = "microservices-dev"
      "namespace" = "argocd-production"
      "finalizers" = [
        "resources-finalizer.argocd.argoproj.io"
      ]
      "labels" = {
        "app" = "microservices"
        "env" = "prod"
      }
      "annotations" = {
      }
    }

    "spec" = {
      "destination" = {
        "server"    = "https://kubernetes.default.svc"
        "namespace" = "demo-microservices-prod"
      }
      "project" = "default"
      "source" = {
        "repoURL"        = "https://github.com/argoproj/argocd-example-apps.git"
        "targetRevision" = "master"
        "path"           = "guestbook/"
      }
      "syncPolicy" = {
        "automated" = {
          "prune"    = true
          "selfHeal" = true
        }
        "syncOptions" = [
          "CreateNamespace=true",
          "Prune=true",
        ]
      }
    }
  }
}
  1. Deploy one of the Apps and check the corresponding ArgoCD instance picks it up and deploys all the resources.
  2. Deploy the second App and check how, when the other ArgoCD instance picks it up, both ArgoCD start fighting to reach the app state each instance thinks is the truth.

Expected behavior

Given two different ArgoCD instances, they can manage two different Apps with the same name but with different resources being deployed to different target namespaces.

Screenshots

No screenshot. Is ArgoCD trying to create the resources it has defined and trying to delete the resources the other ArgoCD instance is trying to create. I can make a short video if it is useful.

Version

argocd@argocd-devstg-application-controller-0:~$ argocd version
argocd: v2.10.4+f5d63a5
  BuildDate: 2024-03-18T08:09:23Z
  GitCommit: f5d63a5c77d2e804e51ef94bee3db441e0789d00
  GitTreeState: clean
  GoVersion: go1.21.3
  Compiler: gc
  Platform: linux/amd64
FATA[0000] Argo CD server address unspecified

Logs

Didn´t find relevant logs.

jannfis commented 3 months ago

Have you configured two unique instance label keys for each instance, as described here?

juanmatias commented 2 months ago

@jannfis thanks, it worked! I guess this issue can be closed.

I am documenting this (and the process of deploying two or more ArgoCD instances in the same cluster) here. I am adding a note on this label here.

Please, let me know if we can collaborate with some documentation on this topic in the ArgoCD project.