The ConfigMap is invalid: metadata.annotations: Too long: must have at most 262144 characters

[abdennour]

If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a question in argocd slack channel.

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

+Describe the bug configmap with big size cannot be synced thru ArgoCD while it can be deployed thru helm and other alternatives

To Reproduce

1. Use jenkins helm chart or grafana helm chart or any chart accept evolved configmap(s)
2. Try to put many things inside this configmap thru helm values.
3. Try now to sync the app with ArgoCD.
4. You have to see this error in "Sync Status"

ConfigMap "ARGOCD_APP_NAME" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

Expected behavior

ArgoCD should apply the changes as long as there is no restriction from server ( etcd,... )

Screenshots

Version

argocd: v1.8.4+28aea3d
#.....
argocd-server: v1.8.4+28aea3d
#.....

Logs

ConfigMap is invalid: metadata.annotations: Too long: must have at most 262144 characters 

Duplicate of #820 . However, no way to get this one opened again while it's resolved and it's reproducable for more than 3 years

[rajivml]

I have also run into this just now and kinda stuck here

[abdennour]

@rajivml just to get things done, i created configmap with kubectl :

• if it's 1st time: use kubectl create -f ............ not kubect apply -f
• if it's not 1st time: use kubectl replace -f .. not kubect apply -f

But note that the argocd app will be remaining in outOfSync status,, which is actually embarrassing me in front of the crew :(

[servo1x]

Related: #2730

[rajivml]

@abdennour do we have to do this manually instead through argoCD because argocd uses apply for everything ?

[abdennour]

@rajivml Unfortunately , no way to do it thru ArgoCD. I just did it manually : i broke the whole the gitOps approach, and i provided ugly README.md to the team to do it if i was not there.

[abdennour]

I opened issue on the related helm chart git repo https://github.com/jenkinsci/helm-charts/issues/280 I am helping owners of this chart, to design it in a different way, so it can scale.

I would suggest to try to do the same with your charts where you are facing the same problem.

I know this issue is related initially with the executor ( ArgoCD), but let's help others with any way.

[abdennour]

any deadline for fixing this issue ? Could you , for example, patching the YAML before applying it ( removing the annotation last-applied-configuration , .. so on) ? If the resource does not work because of this error, try to run k replace instead of k apply ?

[alexmt]

PR that introduces Replace=true sync option support has been merged. The sync option can be set per resource using argocd.argoproj.io/sync-options: Replace=true annotation or for the whole application.

Feature, will be available as part of v2.0-rc1 ( hopefully by end of tomorrow )

[diranged]

@alexmt, Are we sure that the feature actually solves the issue? I upgraded to 2.0.0-rc1 and we are still seeing Argo fail here. I have tried setting the syncOptions in the application itself as a test through the Web UI and that had no effect on the sync failing:

I also tried editing the live CRD resource and adding in that annotation:

$ k get crd kafkas.kafka.strimzi.io -o json | jq .metadata.annotations | grep argocd.argoproj.io/sync-options
  "argocd.argoproj.io/sync-options": "Replace=true",

And I get the same behavior as above.. and I know of no other way in a CRD installed by a Helm chart to add in that annotation. Am I missing something?

[diranged]

Here is our current argocd version tab..

[alexmt]

Hello @diranged ,

You are right, CRD case was not fixed. The kubectl replace is dangerous to use with CRDs since kubectl might delete and recreate it. Since CRDs are often causing too long annotation issue we added support as well: now Argo CD tries to use K8S update API if Replace=true sync option is specified. This change was merged yesterday and will be cherry-picked into 2.0

[alexmt]

The closing issue as a duplicate of https://github.com/argoproj/argo-cd/issues/820

[armenr]

I have also run into this, just today...and I have no clue how to debug this or otherwise fix/approach it.

This still appears to be an issue. I'm seeing it currently.

I tried to read around, google a bit, but haven't come up with much...and I'm not sure how to fix or debug this. ANY help would be much appreciated.

In ArgoCD, the Error being displayed is:

Failed sync attempt to fa68c5c19c15882e88f303478b91b9cabbec7d39: one or more objects failed to apply, reason: CustomResourceDefinition.apiextensions.k8s.io "applicationsets.argoproj.io" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

I've adapted my approach by following the same pattern that argocd-autopilot takes with its bootstrap method...with my own slight modifications.

This is where the code lives: https://github.com/armenr/5thK8s/tree/main/dependencies/bootstrap

After installing and configuring argo-cd, this is the only file I kubectl apply -f in order to "bootstrap" all the other ArgoCD projects and apps:

https://github.com/armenr/5thK8s/blob/main/dependencies/bootstrap/autopilot-bootstrap.yaml