search

argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
17.32k stars 5.26k forks source link

Inaccurate sync error message when template value is passed as boolean instead of string #4391

Open lucinvitae opened 3 years ago

lucinvitae commented 3 years ago

Checklist:

Describe the bug

When a boolean value is used in the environment variables section of a deployment template in a Helm chart (which helm template validates successfully as can be seen below), our ArgoCD instance fails to sync the deployment (SyncFailed for deployment apps/v1/...) with the following difficult-to-read and inaccurate error message:

"" is invalid: patch: Invalid value: "map[metadata:map[annotations:map[
<snip lots and lots of annotations, see log output at bottom of this bug report>
\n]]
spec:map[template:map[spec:map[volumes:<nil>]]]]": unrecognized type: string

This error message displayed in our ArgoCD version has the following issue:

  1. It seems to suggest that there is an empty string in the values somewhere (there is not).
  2. It seems to suggest that there is a string value at fault causing the issue (incorrect, it's a boolean).
  3. The error message dumps the entire mapping of YAML values (scroll down to see example full output) instead of the particular value causing the error.

Note that I know how to fix (and have fixed) the error, but that the goal of this ticket is to improve the error handling of ArgoCD's sync to make it easier for future developers who found the same issue.

To Reproduce

To reproduce the issue, create a Helm chart setup with the following values.yaml and templates/deployment.yaml files:

values.yaml file:

boolString: "False"

templates/deployment.yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-bug-report
spec:
  template:
    spec:
      containers:
        - name: test-bug-report-container
          image: python:3.7-alpine
          env:
            - name: MY_VAR
              value: {{ .Values.boolString }}

Running helm template shows that it's a valid template:

helm template . -x ./templates/deployment.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-bug-report
spec:
  template:
    spec:
      containers:
        - name: test-bug-report-container
          image: python:3.7-alpine
          env:
            - name: MY_VAR
              value: False

Notice that helm does not care that the MY_VAR value wasn't encapsulated in double quotes. Only ArgoCD appears to care, throwing the abovementioned error during application sync: "" is invalid: patch: Invalid value: "map[metadata:map[annotations:map[ ....... \n]] spec:map[template:map[spec:map[volumes:<nil>]]]]": unrecognized type: string

The fix is to edit the following lines in the templates/deployment.yaml file to ensure string encapsulation:

              value: "{{ .Values.boolString }}"

But Argo's error message requires that the developer does quite a bit of unnecessary debugging, especially if there are many variables in the values.yaml file and a much more complicated deployment template.

Expected behavior

What would be ideal: The current error message, which seems to imply that the value at fault is a string instead of a boolean, could be improved by highlighting that the value at fault is actually a boolean and not a string. For example, it would be really helpful for debugging if the error message was simply patch failed due to the following issue: the value for key "MY_VAR" in the nested mapping was type <boolean>, but type <string> was expected. Value: "{\"name\":\"MY_VAR\",\"value\":false}"

What is perhaps more feasible: If it's not possible to drill down to the value that is causing the issue within the nested mapping, at least the following should be implemented if possible:

  1. The error message should not state that there is an empty string in the values somewhere.
  2. The error message should indicate that there is a boolean value causing the issue.

The above 2 fixes would be a significant improvement, since the error message would only be difficult to parse in that case, rather than difficult to parse and misleading.

Screenshots

N/A

Version

argocd version:

v1.6.2+3d1f37b

Logs

"" is invalid: patch: Invalid value: "map[metadata:map[annotations:map[kubectl.kubernetes.io/last-applied-configuration:{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"iam.amazonaws.com/role\":\"<redacted>\"},\"labels\":{\"app.kubernetes.io/instance\":\"gfb-docs-inference-service-dev-inference-service\",\"app.kubernetes.io/managed-by\":\"Tiller\",\"app.kubernetes.io/name\":\"gfb-docs-inference-service\",\"argocd.argoproj.io/instance\":\"gfb-docs-inference-service-dev\",\"helm.sh/chart\":\"gfb-docs-inference-service-0.1.0\"},\"name\":\"doc-scanner-inference-service\",\"namespace\":\"doc-scanner\"},\"spec\":{\"replicas\":2,\"selector\":{\"matchLabels\":{\"app.kubernetes.io/instance\":\"gfb-docs-inference-service-dev-inference-service\",\"app.kubernetes.io/name\":\"gfb-docs-inference-service\"}},\"template\":{\"metadata\":{\"annotations\":{\"iam.amazonaws.com/role\":\"<redacted>\"},\"labels\":{\"app.kubernetes.io/instance\":\"gfb-docs-inference-service-dev-inference-service\",\"app.kubernetes.io/name\":\"gfb-docs-inference-service\"}},\"spec\":{\"containers\":[{\"args\":[\"-c\",\"<redacted>\"],\"command\":[\"/bin/sh\"],\"env\":[{\"name\":\"MY_VAR\",\"value\":false},{\"name\":\"HTTP_PORT\",\"value\":\"8080\"},{\"name\":\"WORKERS\",\"value\":\"17\"},{\"name\":\"MLFLOW_TRACKING_URI\",\"value\":\"<redacted>\"},{\"name\":\"MLFLOW_EXPERIMENT_NAME\",\"value\":\"doc_classifier_v2\"},{\"name\":\"MODEL_NAME\",\"value\":\"prodigy-text-multiclass\"},{\"name\":\"MODEL_LABELS\",\"value\":\"TRF_any_page,family_history,genetic_counseling_report,insurance_card,insurance_card_info,invitae_report_coversheet,invitae_report_other_page,letter_of_medical_necessity,medical_records,pedigree,prior_auth_form\"},{\"name\":\"SUGGESTED_LABELS_CONFIDENCES\",\"value\":\"0.5,0.45,0.4,0.9,0.35,0.5,0.5,0.95,0.6,0.8,0.5\"},{\"name\":\"SUGGESTED_LABELS_BBOXES\",\"value\":\"False,False,False,False,False,False,False,False,False,False,False\"}],\"image\":\"<redacted>\",\"imagePullPolicy\":\"Always\",\"livenessProbe\":{\"failureThreshold\":20,\"httpGet\":{\"path\":\"/v1/models\",\"port\":\"http\"},\"initialDelaySeconds\":180,\"periodSeconds\":30},\"name\":\"gfb-docs-inference-service\",\"ports\":[{\"containerPort\":8080,\"name\":\"http\",\"protocol\":\"TCP\"}],\"readinessProbe\":{\"failureThreshold\":20,\"httpGet\":{\"path\":\"/v1/models\",\"port\":\"http\"},\"initialDelaySeconds\":60,\"periodSeconds\":20},\"resources\":{\"limits\":{\"cpu\":\"2\",\"memory\":\"7Gi\"},\"requests\":{\"cpu\":\"500m\",\"memory\":\"4Gi\"}}}],\"nodeSelector\":{\"group-name\":\"worker-group-1\"},\"volumes\":null}}}}\n]] spec:map[template:map[spec:map[volumes:<nil>]]]]": unrecognized type: string
jessesuen commented 3 years ago 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-bug-report
spec:
  template:
    spec:
      containers:
        - name: test-bug-report-container
          image: python:3.7-alpine
          env:
            - name: MY_VAR
              value: False

According to the YAML linter I tried, False (even if capitalized), is considered a bool value and not a string. So this would be expected behavior.

jessesuen commented 3 years ago

Is this about a better error message? Or special handling of the situation?

lucinvitae commented 3 years ago

@jessesuen better error message, since Helm is rendering the correct value and the fix is straightforward once the issue is understood. I tried documenting this in the "Expected behavior" section above.

shepely commented 3 years ago

Just encountered the same issue on our dev cluster using 3rd party helm charts and trying to add boolean labels to it. Thanks for reporting @lucinvitae. Would've really appreciate a better error message here 🙏

zack-johnson5455 commented 3 years ago

Adding my voice to this issue. Just got burned by this as well

maxvt-tonal commented 3 years ago

Same here. A better error message would certainly be helpful. Does this verbose error message come from Argo, though?

"Helm is rendering the correct value" in the comments above, to me, seems misleading: according to a similar issue in https://github.com/kubernetes/kubernetes/issues/59113, annotations are map[string]string and having unexpected booleans in the YAML causes (at least, has caused in the past) issues with other tools as well, where Argo is not involved. See also various type coercion shenanigans referenced by https://github.com/helm/helm/issues/2848. The correct value in this case is the corresponding string ("false").

ChristopherLenz commented 3 years ago

from my point of view helm and kubernetes work correctly. I had following problem

This is not working with ArgoCD. 

        env:
        {{ range $k, $v := .Values.extraEnvVars }}
          - name: {{ $v.name | quote }}
            value: {{ $v.value | quote }}
        {{- end }}

I had to change it to this in order to be accepted by ArgoCD

        env:
        {{ range $k, $v := .Values.extraEnvVars }}
          - name: "{{ $v.name }}"
            value: "{{ $v.value }}"
        {{- end }}
raweber42 commented 1 year ago

Thanks for pointing this out! Just spent >7 hours to debug this. The error message should really be changed!

crenshaw-dev commented 1 year ago

Does anyone have the full error message from a recent version of Argo CD? I'm trying to identify the line of code that's bubbling this error up. Maybe we can detect this kind of error and add more info to the error message.

mlhynfield commented 3 months ago 
one or more objects failed to apply, reason: "" is invalid: patch: Invalid value: "map[metadata:map[annotations:map[kubectl.kubernetes.io/last-applied-configuration:{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{},\"labels\":{\"app\":\"jellyfin-19.1.7\",\"app.kubernetes.io/instance\":\"jellyfin\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"jellyfin\",\"app.kubernetes.io/version\":\"10.9.2\",\"argocd.argoproj.io/instance\":\"jellyfin\",\"helm-revision\":\"1\",\"helm.sh/chart\":\"jellyfin-19.1.7\",\"release\":\"jellyfin\"},\"name\":\"jellyfin\",\"namespace\":\"jellyfin\"},\"spec\":{\"replicas\":1,\"revisionHistoryLimit\":3,\"selector\":{\"matchLabels\":{\"app.kubernetes.io/instance\":\"jellyfin\",\"app.kubernetes.io/name\":\"jellyfin\",\"pod.name\":\"main\"}},\"strategy\":{\"type\":\"Recreate\"},\"template\":{\"metadata\":{\"annotations\":{\"checksum/cnpg\":\"ffd338b55cc50a8b366097eec597eeb134dee46563555614453e28233d208d00\",\"checksum/configmaps\":\"44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a\",\"checksum/mariadb\":\"09c85576cb45b1eecd1467732b11ea8fa3363b0105c465f02a6ad64991521d52\",\"checksum/mongodb\":\"09c85576cb45b1eecd1467732b11ea8fa3363b0105c465f02a6ad64991521d52\",\"checksum/persistence\":\"0765ceacf8346b2887ac158ddb698dc4afa483aa39a182f2b7833662ff4fb1ed\",\"checksum/redis\":\"013343a028cbb3f7e08f4ba7522702dd98e52632c688641074b0b1db3df29894\",\"checksum/secrets\":\"44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a\",\"checksum/services\":\"a4a38d40bc3ed7512844016147721a1f46fd1bc7375f466225e04f9954c41d28\",\"checksum/solr\":\"29c14feeaddbf7762052db593898d274941f539cee681ddc613957587686f347\"},\"labels\":{\"app\":\"jellyfin-19.1.7\",\"app.kubernetes.io/instance\":\"jellyfin\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"jellyfin\",\"app.kubernetes.io/version\":\"10.9.2\",\"helm-revision\":\"1\",\"helm.sh/chart\":\"jellyfin-19.1.7\",\"pod.lifecycle\":\"permanent\",\"pod.name\":\"main\",\"release\":\"jellyfin\"}},\"spec\":{\"automountServiceAccountToken\":false,\"containers\":[{\"env\":[{\"name\":\"TZ\",\"value\":\"UTC\"},{\"name\":\"UMASK\",\"value\":\"002\"},{\"name\":\"UMASK_SET\",\"value\":\"002\"},{\"name\":\"NVIDIA_VISIBLE_DEVICES\",\"value\":\"void\"},{\"name\":\"PUID\",\"value\":\"0\"},{\"name\":\"USER_ID\",\"value\":\"0\"},{\"name\":\"UID\",\"value\":\"0\"},{\"name\":\"PGID\",\"value\":\"0\"},{\"name\":\"GROUP_ID\",\"value\":\"0\"},{\"name\":\"GID\",\"value\":\"0\"},{\"name\":\"S6_READ_ONLY_ROOT\",\"value\":\"1\"},{\"name\":\"JELLYFIN_PublishedServerUrl\",\"value\":\"\"}],\"image\":\"ghcr.io/jellyfin/jellyfin:10.9.2@sha256:61aa5882c73d47550a29ba6fd1e90bcf242e0a8d9351ddc185dc17aa54ac72c4\",\"imagePullPolicy\":\"IfNotPresent\",\"livenessProbe\":{\"failureThreshold\":5,\"httpGet\":{\"path\":\"/\",\"port\":8096,\"scheme\":\"HTTP\"},\"initialDelaySeconds\":12,\"periodSeconds\":15,\"successThreshold\":1,\"timeoutSeconds\":5},\"name\":\"jellyfin\",\"ports\":[{\"containerPort\":8096,\"hostPort\":null,\"name\":\"main\",\"protocol\":\"TCP\"}],\"readinessProbe\":{\"failureThreshold\":4,\"httpGet\":{\"path\":\"/\",\"port\":8096,\"scheme\":\"HTTP\"},\"initialDelaySeconds\":10,\"periodSeconds\":12,\"successThreshold\":2,\"timeoutSeconds\":5},\"resources\":{\"limits\":null,\"requests\":{\"cpu\":4,\"memory\":\"8Gi\"}},\"securityContext\":{\"allowPrivilegeEscalation\":true,\"capabilities\":{\"add\":[\"CHOWN\",\"SETUID\",\"SETGID\",\"FOWNER\",\"DAC_OVERRIDE\"],\"drop\":[\"ALL\"]},\"privileged\":true,\"readOnlyRootFilesystem\":true,\"runAsGroup\":0,\"runAsNonRoot\":false,\"runAsUser\":0,\"seccompProfile\":{\"type\":\"RuntimeDefault\"}},\"startupProbe\":{\"failureThreshold\":60,\"httpGet\":{\"path\":\"/\",\"port\":8096,\"scheme\":\"HTTP\"},\"initialDelaySeconds\":10,\"periodSeconds\":5,\"successThreshold\":1,\"timeoutSeconds\":3},\"stdin\":false,\"tty\":false,\"volumeMounts\":[{\"mountPath\":\"/cache\",\"name\":\"cache\",\"readOnly\":false},{\"mountPath\":\"/config\",\"name\":\"config\",\"readOnly\":false},{\"mountPath\":\"/dev/dma_heap\",\"name\":\"dev-dma-underscore-heap\",\"readOnly\":false},{\"mountPath\":\"/dev/dri\",\"name\":\"dev-dri\",\"readOnly\":false},{\"mountPath\":\"/dev/h265e\",\"name\":\"dev-h265e\",\"readOnly\":false},{\"mountPath\":\"/dev/hevc-service\",\"name\":\"dev-hevc-service\",\"readOnly\":false},{\"mountPath\":\"/dev/hevc_service\",\"name\":\"dev-hevc-underscore-service\",\"readOnly\":false},{\"mountPath\":\"/dev/iep\",\"name\":\"dev-iep\",\"readOnly\":false},{\"mountPath\":\"/dev/mali0\",\"name\":\"dev-mali0\",\"readOnly\":false},{\"mountPath\":\"/dev/mpp-service\",\"name\":\"dev-mpp-service\",\"readOnly\":false},{\"mountPath\":\"/dev/mpp_service\",\"name\":\"dev-mpp-underscore-service\",\"readOnly\":false},{\"mountPath\":\"/dev/rga\",\"name\":\"dev-rga\",\"readOnly\":false},{\"mountPath\":\"/dev/rkvdec\",\"name\":\"dev-rkvdec\",\"readOnly\":false},{\"mountPath\":\"/dev/rkvenc\",\"name\":\"dev-rkvenc\",\"readOnly\":false},{\"mountPath\":\"/dev/vepu\",\"name\":\"dev-vepu\",\"readOnly\":false},{\"mountPath\":\"/dev/vpu-service\",\"name\":\"dev-vpu-service\",\"readOnly\":false},{\"mountPath\":\"/dev/vpu_service\",\"name\":\"dev-vpu-underscore-service\",\"readOnly\":false},{\"mountPath\":\"/dev/shm\",\"name\":\"devshm\",\"readOnly\":false},{\"mountPath\":\"/var/lib/jellyfin\",\"name\":\"jellyfin\",\"readOnly\":false},{\"mountPath\":\"/data/media\",\"name\":\"media\",\"readOnly\":false},{\"mountPath\":\"/shared\",\"name\":\"shared\",\"readOnly\":false},{\"mountPath\":\"/var/lib/tls\",\"name\":\"tls\",\"readOnly\":false},{\"mountPath\":\"/tmp\",\"name\":\"tmp\",\"readOnly\":false},{\"mountPath\":\"/config/transcodes\",\"name\":\"transcode\",\"readOnly\":false},{\"mountPath\":\"/var/logs\",\"name\":\"varlogs\",\"readOnly\":false},{\"mountPath\":\"/var/run\",\"name\":\"varrun\",\"readOnly\":false}]}],\"dnsConfig\":{\"options\":[{\"name\":\"ndots\",\"value\":\"1\"}]},\"dnsPolicy\":\"ClusterFirst\",\"enableServiceLinks\":false,\"hostIPC\":false,\"hostNetwork\":false,\"hostPID\":false,\"hostUsers\":true,\"nodeSelector\":{\"jellyfin\":true,\"kubernetes.io/arch\":\"arm64\"},\"restartPolicy\":\"Always\",\"runtimeClassName\":null,\"securityContext\":{\"fsGroup\":0,\"fsGroupChangePolicy\":\"OnRootMismatch\",\"supplementalGroups\":[568],\"sysctls\":[]},\"serviceAccountName\":\"default\",\"shareProcessNamespace\":false,\"terminationGracePeriodSeconds\":60,\"tolerations\":[{\"effect\":\"NoSchedule\",\"key\":\"orangepi\",\"operator\":\"Exists\"}],\"topologySpreadConstraints\":[{\"labelSelector\":{\"matchLabels\":{\"app.kubernetes.io/instance\":\"jellyfin\",\"app.kubernetes.io/name\":\"jellyfin\",\"pod.name\":\"main\"}},\"maxSkew\":1,\"nodeAffinityPolicy\":\"Honor\",\"nodeTaintsPolicy\":\"Honor\",\"topologyKey\":\"kubernetes.io/hostname\",\"whenUnsatisfiable\":\"ScheduleAnyway\"}],\"volumes\":[{\"name\":\"cache\",\"persistentVolumeClaim\":{\"claimName\":\"jellyfin-cache\"}},{\"name\":\"config\",\"persistentVolumeClaim\":{\"claimName\":\"jellyfin-config\"}},{\"hostPath\":{\"path\":\"/dev/dma_heap\"},\"name\":\"dev-dma-underscore-heap\"},{\"hostPath\":{\"path\":\"/dev/dri\"},\"name\":\"dev-dri\"},{\"hostPath\":{\"path\":\"/dev/h265e\"},\"name\":\"dev-h265e\"},{\"hostPath\":{\"path\":\"/dev/hevc-service\"},\"name\":\"dev-hevc-service\"},{\"hostPath\":{\"path\":\"/dev/hevc_service\"},\"name\":\"dev-hevc-underscore-service\"},{\"hostPath\":{\"path\":\"/dev/iep\"},\"name\":\"dev-iep\"},{\"hostPath\":{\"path\":\"/dev/mali0\"},\"name\":\"dev-mali0\"},{\"hostPath\":{\"path\":\"/dev/mpp-service\"},\"name\":\"dev-mpp-service\"},{\"hostPath\":{\"path\":\"/dev/mpp_service\"},\"name\":\"dev-mpp-underscore-service\"},{\"hostPath\":{\"path\":\"/dev/rga\"},\"name\":\"dev-rga\"},{\"hostPath\":{\"path\":\"/dev/rkvdec\"},\"name\":\"dev-rkvdec\"},{\"hostPath\":{\"path\":\"/dev/rkvenc\"},\"name\":\"dev-rkvenc\"},{\"hostPath\":{\"path\":\"/dev/vepu\"},\"name\":\"dev-vepu\"},{\"hostPath\":{\"path\":\"/dev/vpu-service\"},\"name\":\"dev-vpu-service\"},{\"hostPath\":{\"path\":\"/dev/vpu_service\"},\"name\":\"dev-vpu-underscore-service\"},{\"emptyDir\":{\"medium\":\"Memory\"},\"name\":\"devshm\"},{\"name\":\"jellyfin\",\"persistentVolumeClaim\":{\"claimName\":\"jellyfin-jellyfin\"}},{\"name\":\"media\",\"nfs\":{\"path\":\"/export/media\",\"server\":\"\"}},{\"emptyDir\":{},\"name\":\"shared\"},{\"name\":\"tls\",\"secret\":{\"items\":[{\"key\":\"keystore.p12\",\"path\":\"keystore.pfx\"}],\"optional\":false,\"secretName\":\"jellyfin-tls\"}},{\"emptyDir\":{\"medium\":\"Memory\"},\"name\":\"tmp\"},{\"emptyDir\":{},\"name\":\"transcode\"},{\"emptyDir\":{\"medium\":\"Memory\"},\"name\":\"varlogs\"},{\"emptyDir\":{\"medium\":\"Memory\"},\"name\":\"varrun\"}]}}}}\n]] spec:map[template:map[metadata:map[annotations:map[checksum/services:a4a38d40bc3ed7512844016147721a1f46fd1bc7375f466225e04f9954c41d28]] spec:map[hostIPC:false hostNetwork:false hostPID:false hostUsers:true nodeSelector:map[jellyfin:true nodeType:<nil>] runtimeClassName:<nil> securityContext:map[sysctls:[]]]]]]": unrecognized type: string. Retrying attempt #3 at 8:47PM.

I think that should be the error to which this issue refers