argocd CLI uses wrong namespace

bfblog commented 2 years ago

The argocd CLI uses the kubernetes default namespace. For example, "argocd app list" will return an empty list, if current namespace is not set to argocd. The CLI offers "-n" or "--namespace" parameter, but when specifying it, it is detected as invalid option.

v2.3.3

nikhleshgoenka commented 2 years ago

Did you try adding applications from UI and that works and gets deployed in expected namespace?

bfblog commented 2 years ago

Yes, I deployed several applications using the UI and some YAML resources. As result, any of them works as expected. As a next step I started using the argocd CLI. "argocd app list" returns an empty list. When switching the default namespace to "argocd" the tools finds the apps. So is switched the default back to a different namespace and tried to apply the namespace to argocd CLI. Bute here the it comes to invalid parameter (see above)

I

danielhelfand commented 2 years ago

The argocd app list command doesn't support the --namespace option: https://github.com/argoproj/argo-cd/blob/c7ff388b8476a85d10e888f457e35e050658a1c8/cmd/argocd/commands/app.go#L1159

argocd app list -h           
List applications

Usage:
  argocd app list [flags]

Examples:
  # List all apps
  argocd app list

  # List apps by label, in this example we listing apps that are children of another app (aka app-of-apps)
  argocd app list -l app.kubernetes.io/instance=my-app

Flags:
  -h, --help                  help for list
  -o, --output string         Output format. One of: wide|name|json|yaml (default "wide")
  -p, --project stringArray   Filter by project name
  -r, --repo string           List apps by source repo URL
  -l, --selector string       List apps by label

None of the app commands do, which possibly means it was an intentional design choice. Listing by projects is probably the way of organizing Applications with argocd. The only commands that support namespace option for apps are related to an Application's underlying resources (i.e. patch-resource, delete-resource).

I agree it's confusing though from a kubectl user perspective. This is probably something that requires further discussion to know if this is something to move forward with, but I would reclassify this as an enhancement, not a bug.

In the meantime, you can always use kubectl to list Applications by namespace.

bfblog commented 2 years ago

I am very surprised that no one else has noticed this behavior. I have been dealing with ArgoCD for quite some time and now use it to bootstrap clusters. This works really well. The ArgoCD CLI is - as far as my understanding goes - a wrapper around kubectl and can handle the argo CRD especially well. And with kubectl, you actually always have to specify the namespace. From experience, the active namespace is never the right one. Alternatively, the CLI could look at all namespaces, mutatis mutandis an --all-namespaces.

What I am actually looking for is an option to remove an application. More precisely, the Kubernetes objects should be removed, but the ArgoCD application should remain inactive. Background: I have defined many Argo applications (also app-of-apps), and would like to realize a function "enable/disable optional features" with Argo. I was hoping to be able to customize this feature through scripts. My wish:

a super configuration for all applications of a project
via a setting, I can activate or deactivate an app
deactivate removes all resources, but the app remains without auto-sync
enable corresponds to the behavior directly after the deploy, Kubernetes objects are created

brunzefb commented 2 years ago

I am not sure if this is a related issue or not, but I will mention it. I have an argocd app

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: nca-dev-oom-api-pca-pcvg
  namespace: argocd   <-- must be argocd, otherwise the app does not load in the UI with kubectl apply

If I specify the namespace as argocd (the ns where argocd is installed), the UI immediately updates, and the app starts to sync. If I specify another namespace, this does not work. When trying to apply with the non-argocd namespace via the UI, I get Unable to create application: the namespace of the provided object does not match the namespace sent on the request

If I have the 2 similar applications, where the deployment targets two separate namespaces, I am currently forced to name my application differently, eg: pcvg-dev and pcvg-qa for the argo Application name. The argo ui allows viewing by namespace or by project, but that does not help; because there is one argocd namespace, and you can't have the same name for the same resource type -- if I deploy 2 argo apps with the same name, the last one wins.

My workaround for now is just to name the apps differently, but its annoying. I can file a different issue, if you think this is not related.

bfblog commented 2 years ago

I think it is not a related issue but a misunderstanding of Kubernetes resources:

The section "metadata" is related to Kubernetes, it is not part of argocd.
The combination of metadata.name and metadata.namespace is a unique identifier in the scope of Kubernetes. Creating several yaml with the same identifiers causes an overwriting of the existing yaml-resource. The last applied yaml wins.
ArgoCD is running in namespace ArgoCD. Setting up an Argo-Application means, to add a further yaml resource of Kind Application to argocd namespace.
ArgoCD only watches objects in its own namespace. In this way, any Application installed to a different namespace is not mentioned by ArgoCD

My recommendation is:

install all applications to argocd namespace and use unique names.
add labels to metadata and use the filter capabilities

apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-app-qa namespace: argocd <-- must be argocd, otherwise the app does not load in the UI with kubectl apply labels: app: my-app env: qa

apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-app-dev namespace: argocd <-- must be argocd, otherwise the app does not load in the UI with kubectl apply labels: app: my-app env: dev


Now, set label filter "env=qa" to see resources from the qa-environment.

Please let me know, if this commend was helpful.

brunzefb commented 2 years ago

@bfblog Yes, your answer is helpful, and I already know I can work around this with giving unique names to the argo apps.
It is just not intuitive from a K8s perspective. Think about it, I can have a deployment or pod called "myapp", and I can install it in the "dev" namespace. Now I can create the exact same deployment/pod (maybe with a different version) in the 'staging' namespace. So as a K8s devops guy, I am used to the SAME name being being in both 'dev' and 'staging' namespaces.

Now look at argocd -- ALL the apps are in one namespace the argocd one. Why this limitation? Why can't I tell argocd which namespaces to monitor for argo cd applications -- which are after all Kubernetes objects. I am making a strong argument for argo CD to pick up apps from any namespace I configure it to monitor. With this feature, all my argocd apps could be named the same (still unique names in each namespace), and I can use the UI of argocd to filter by namespace, project or label. Should I create a separate argocd feature request for this?

Now naming the the argocd apps differently does have the advantage that the title is different -- eg myapp-dev vs myapp-staging, and when looking at the UI, regardless of filtering it is very clear at what you are looking at.

Having said all of this, the argocd documentation should be clearer (maybe I missed something) that ALL argocd applications MUST live in the argocd namespace.

Best, F.

bfblog commented 2 years ago

My understanding of ArgoCD is as follows:

ArgoCD is a tool to fill one or more namespaces with resources.
Applications form the configuration for ArgoCD, it makes sense that all configurations are in one namespace. Multiple namespaces would allow conflicts. How will you resolve them in ArgoCD?
To clean up, you can simply delete other namespaces and let ArgoCD refill them. But this is not possible if the deleted namespace contains applications, too.
ArgoCD supports projects. So you can realize something like namespaces in ArgoCD. In a project, all projects must be named uniquely. With two projects you can hide different applications behind the same name.

For me, the concept of ArgoCD seems conclusive.

As for the documentation, I couldn't agree more. You first have to put together the right recipe. And that brings me back to the initial situation: the CLI does not seem well thought out to me. While all applications have to be in a namespace, the CLI searches the current namespace. And if the current namespace in kubectl is not set to ArgoCD, then the tool does not work or it does not find any applications. So I have to select the namespace argocd first via kubectl so that the ArgoCD CLI works in the next step.

Since ArgoCD is always installed into the namespace "argocd" according to the installation instructions, it would only be logical if the CLI also searches for data here.

@danielhelfand: Is this now included as a feature request?

brunzefb commented 2 years ago

@bfblog Thanks for engaging in the discussion, I am learning a lot! Your reasoning is sound, especially the part about re-creating namespaces with the argocd tool -- of course then it does not make sense to deploy argocd apps there, so my proposal to allow argocd to monitor other namespaces does not make sense. Now I did play with projects, and put two apps in separate projects -- from what I observed the applications STILL must still have unique names even though they are in different projects. This kind of makes sense, since the K8s kind is Application, and you cannot have two entities of the same kind with the same name, just as you can't have two different files with the same filepath. Looks like the projects just help with security and locking down allowed repos; I thus have no need for projects.

I agree with all you said about the docs and the CLI namespace issue -- there should at least be a warning if you either use the CLI with the wrong namespace, and another one if you do a kubectl apply -f with kind:Application and your default namespace is not the namespace where argocd is installed or have the -n [argocd-ns] flag, and/or the manifest lacks the .metadata.namespace argocd.

One should not assume that argocd is always installed in the argocd namespace. I install argocd with helm, which has the --namespace flag, allowing the chart install to target the namespace of your choosing.

One last comment about what you wrote about bootstrapping clusters with argocd - I have thought about this and have rejected the idea. In our cluster we run either Infrastructure (stuff like argo, argocd, nginx reverse proxy, cluster-autoscaler, kube2iam, external-dns etc) or application related stuff (in our case one app with around 60 microservices). The intent is to use argocd for the application only. Our cluster bootstrapping happens with a shell script. Our entire infrastructure (on aws using eks) is created with a shell script that calls terragrunt/terraform. The script sets up the vpc, subnets, security groups logging, buckets, policies, roles, RDS, the eks k8s cluster etc. Once this cloud infrastructure provisioning is done, the script uses helm to deploy the cluster-based infrastructure, which includes argocd. Reason for this approach is that that individual helm deployments require parameters that are available as terraform outputs - often vpc id's, eks cluster parameters, roles or security group ids that were created by terraform. These parameters are easily read by a terragrunt/terraform output command. If the deployment needs info about a resource that was not created by terraform (in our case the Route53 domain registration is manual), like a hosted zone, the shell script can use the aws cli to get this info. Even if you use argocd to bootstrap your cluster, something has to create the cluster beforehand. My shell script also deploys a custom argocd configmap to allow use of configManagementPlugins (https://github.com/argoproj/argo-cd/issues/2789), and sets up both the argocd-notification-cm.yaml and corresponding secret, and the argocd repo and repo-secret manifests.

The argocd itself is also a bit tricky to setup with the proper ingress on AWS, another reason for installing it via the shell script (see below) which uses helm for this. We use an ELB->nginx reverse proxy->service->argocd to make this work. Since the ELB terminates our TLS, we have to use the --insecure flag for the server, otherwise things don't work. The ELB's security group must be set to allow-list Bitbucket's (or your git cloud provider's) webhook server cidrs, otherwise the hooks calls don't reach argocd. Finally, the webhook must be setup in bitbucket (otherwise you wait an extra 3 min to sync every time, which I think is unacceptable). I set up argocd email notification for argocd and again struggled because of insufficiently detailed documentation, but now this too works. The UI changes when an application is updating are too subtle imo, and having an email record of what happened in terms of sync is great. I don't think I will be using the argocd UI much - I just want the core server functionality to be as robust as possible -- commit and push a new tag, and see the redeploy of the app right away.

function deploy_argo_cd() {
  echo "Executing deploy_argo_cd"
  local deployment_name="argo-cd"
  cd $DIRECTORY

  local ns=$(yq eval ".argocd.namespace" ../../environment.yaml)
  local ingress_class_name=$(yq eval ".argocd.ingress-class-name" ../../environment.yaml)
  local ingress_host_name=$(yq eval ".argocd.ingress-hostname" ../../environment.yaml)
  local redis_host=$(yq eval ".argocd.redis-host" ../../environment.yaml)
  local password=$(yq eval ".argocd.redis-password" ../../environment.yaml)

  check_if_helm_deployment_exists $deployment_name
  if [[ "$helm_deployment_exists" -eq 0 ]]; then
    printf "$ORANGE"
    printf "Skipping install of %s - it is already installed.\n" "$deployment_name"
    printf "$STOP"
    return
  fi
  big_title "Installing " $GREEN
  big_title "$deployment_name" $GREEN

  cd ../../../../devops-argocd/argo-cd
  helm dependency update
  helm upgrade \
    $deployment_name \
    ./ \
    --install \
    --values ./values.yaml \
    --set externalRedis.host=$redis_host \
    --set server.extraArgs="{--insecure}" \
    --set externalRedis.port='6379' \
    --set externalRedis.password=$password \
    --set server.ingress.enabled=true \
    --set server.ingress.ingressClassName=$ingress_class_name \
    --set server.ingress.hosts={"$ingress_host_name"} \
    --set notifications.extraEnv.TZ=America/Toronto \
    --namespace $ns \
    --debug

  if [[ "$?" -ne 0 ]]; then big_title "$deployment_name failed" $RED; fi

  patch_argo_configmap
}

debadutta24 commented 2 years ago

I just changed the default namespace and was able to access resources easily.

kubectl config set-context --current --namespace=argocd

ido-vcita commented 2 years ago

Any update on this? Is it expected to be resolved? Why can't we set a namespace?

I have also tried to create another context with the argocd namespace and then pass to the command like so but no success as well

argocd --kube-context argocd app list
FATA[0007] configmap "argocd-cm" not found

bfblog commented 2 years ago

I am not aware of any change here. Very few people are likely to use the CLI. I also don't understand why this namespace option is simply built in. It should not annoy anyone but make a few happy.

purajit commented 2 years ago

In my case, it's something I've wanted to use while migrating services into ArgoCD to avoid having to merge changes in each time during the import stage while I'm trying to get all the configs in sync.

So if anything it'll make the adoption process a lot nicer :)

dbackeus commented 2 years ago

Being forced to globally configure kubectl to use the argocd namespace makes argocd --core quite painful to use.

Defaulting to use argocd namespace would make sense since that's the documented namespace suggested for argocd installations, maybe one fallback attempt to default kubectl namespace would be reasonable. Then being able to explicitly override via ENV var, CLI argument or entry in ~/.config/argocd/config would make sense.

danielhelfand commented 2 years ago

I think there was some work done to allow Apps to be created in any namespace: https://github.com/argoproj/argo-cd/pull/9755

I also believe there was some work done to add this feature to the CLI. It does not look like it's enabled by default though.

gravufo commented 1 year ago

Any update on this? I have just started investigating using the argocd CLI for some automation, but hitting this issue quite fast in my case since most of my apps are not in the argocd namespace. I don't see why the -n flag should not be supported globally...Almost all commands expect the namespace to be the one containing the argocd-cm config map and/or the argocd pods. Commands that interact with resources could expose a second flag such as --resources-namespace a bit like is already done for --app-namespace in the app list command (but does not work for some reason).

I think from a user experience perspective, it should be as intuitive as possible and imo that means aligning to most kubernetes tools which follow the -n pattern. It really surprised me when I had to change my kubeconfig default context to use this tool...it's literally the first time I had to do this after having used many k8s tools over the years.

andrii-korotkov-verkada commented 1 week ago

It's supported now in cli 2.12

  -N, --app-namespace string   Only list applications in namespace

andrii-korotkov-verkada commented 1 week ago

I've checked a few app commands and they have had this arg. I'll close this, but let me know if it's missing from some commands.

Homulvas commented 1 day ago

I've checked a few app commands and they have had this arg. I'll close this, but let me know if it's missing from some commands.

It's missing from the subcommands in argocd admin notifications.

argoproj / argo-cd

argocd CLI uses wrong namespace #9123

apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-app-qa namespace: argocd <-- must be argocd, otherwise the app does not load in the UI with kubectl apply labels: app: my-app env: qa