Open lucinvitae opened 3 years ago
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-bug-report
spec:
template:
spec:
containers:
- name: test-bug-report-container
image: python:3.7-alpine
env:
- name: MY_VAR
value: False
According to the YAML linter I tried, False
(even if capitalized), is considered a bool value and not a string. So this would be expected behavior.
Is this about a better error message? Or special handling of the situation?
@jessesuen better error message, since Helm is rendering the correct value and the fix is straightforward once the issue is understood. I tried documenting this in the "Expected behavior" section above.
Just encountered the same issue on our dev cluster using 3rd party helm charts and trying to add boolean labels to it. Thanks for reporting @lucinvitae. Would've really appreciate a better error message here 🙏
Adding my voice to this issue. Just got burned by this as well
Same here. A better error message would certainly be helpful. Does this verbose error message come from Argo, though?
"Helm is rendering the correct value" in the comments above, to me, seems misleading: according to a similar issue in https://github.com/kubernetes/kubernetes/issues/59113, annotations are map[string]string
and having unexpected booleans in the YAML causes (at least, has caused in the past) issues with other tools as well, where Argo is not involved. See also various type coercion shenanigans referenced by https://github.com/helm/helm/issues/2848. The correct value in this case is the corresponding string ("false").
from my point of view helm and kubernetes work correctly. I had following problem
This is not working with ArgoCD.
env:
{{ range $k, $v := .Values.extraEnvVars }}
- name: {{ $v.name | quote }}
value: {{ $v.value | quote }}
{{- end }}
I had to change it to this in order to be accepted by ArgoCD
env:
{{ range $k, $v := .Values.extraEnvVars }}
- name: "{{ $v.name }}"
value: "{{ $v.value }}"
{{- end }}
Thanks for pointing this out! Just spent >7 hours to debug this. The error message should really be changed!
Does anyone have the full error message from a recent version of Argo CD? I'm trying to identify the line of code that's bubbling this error up. Maybe we can detect this kind of error and add more info to the error message.
one or more objects failed to apply, reason: "" is invalid: patch: Invalid value: "map[metadata:map[annotations:map[kubectl.kubernetes.io/last-applied-configuration:{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{},\"labels\":{\"app\":\"jellyfin-19.1.7\",\"app.kubernetes.io/instance\":\"jellyfin\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"jellyfin\",\"app.kubernetes.io/version\":\"10.9.2\",\"argocd.argoproj.io/instance\":\"jellyfin\",\"helm-revision\":\"1\",\"helm.sh/chart\":\"jellyfin-19.1.7\",\"release\":\"jellyfin\"},\"name\":\"jellyfin\",\"namespace\":\"jellyfin\"},\"spec\":{\"replicas\":1,\"revisionHistoryLimit\":3,\"selector\":{\"matchLabels\":{\"app.kubernetes.io/instance\":\"jellyfin\",\"app.kubernetes.io/name\":\"jellyfin\",\"pod.name\":\"main\"}},\"strategy\":{\"type\":\"Recreate\"},\"template\":{\"metadata\":{\"annotations\":{\"checksum/cnpg\":\"ffd338b55cc50a8b366097eec597eeb134dee46563555614453e28233d208d00\",\"checksum/configmaps\":\"44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a\",\"checksum/mariadb\":\"09c85576cb45b1eecd1467732b11ea8fa3363b0105c465f02a6ad64991521d52\",\"checksum/mongodb\":\"09c85576cb45b1eecd1467732b11ea8fa3363b0105c465f02a6ad64991521d52\",\"checksum/persistence\":\"0765ceacf8346b2887ac158ddb698dc4afa483aa39a182f2b7833662ff4fb1ed\",\"checksum/redis\":\"013343a028cbb3f7e08f4ba7522702dd98e52632c688641074b0b1db3df29894\",\"checksum/secrets\":\"44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a\",\"checksum/services\":\"a4a38d40bc3ed7512844016147721a1f46fd1bc7375f466225e04f9954c41d28\",\"checksum/solr\":\"29c14feeaddbf7762052db593898d274941f539cee681ddc613957587686f347\"},\"labels\":{\"app\":\"jellyfin-19.1.7\",\"app.kubernetes.io/instance\":\"jellyfin\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"jellyfin\",\"app.kubernetes.io/version\":\"10.9.2\",\"helm-revision\":\"1\",\"helm.sh/chart\":\"jellyfin-19.1.7\",\"pod.lifecycle\":\"permanent\",\"pod.name\":\"main\",\"release\":\"jellyfin\"}},\"spec\":{\"automountServiceAccountToken\":false,\"containers\":[{\"env\":[{\"name\":\"TZ\",\"value\":\"UTC\"},{\"name\":\"UMASK\",\"value\":\"002\"},{\"name\":\"UMASK_SET\",\"value\":\"002\"},{\"name\":\"NVIDIA_VISIBLE_DEVICES\",\"value\":\"void\"},{\"name\":\"PUID\",\"value\":\"0\"},{\"name\":\"USER_ID\",\"value\":\"0\"},{\"name\":\"UID\",\"value\":\"0\"},{\"name\":\"PGID\",\"value\":\"0\"},{\"name\":\"GROUP_ID\",\"value\":\"0\"},{\"name\":\"GID\",\"value\":\"0\"},{\"name\":\"S6_READ_ONLY_ROOT\",\"value\":\"1\"},{\"name\":\"JELLYFIN_PublishedServerUrl\",\"value\":\"\"}],\"image\":\"ghcr.io/jellyfin/jellyfin:10.9.2@sha256:61aa5882c73d47550a29ba6fd1e90bcf242e0a8d9351ddc185dc17aa54ac72c4\",\"imagePullPolicy\":\"IfNotPresent\",\"livenessProbe\":{\"failureThreshold\":5,\"httpGet\":{\"path\":\"/\",\"port\":8096,\"scheme\":\"HTTP\"},\"initialDelaySeconds\":12,\"periodSeconds\":15,\"successThreshold\":1,\"timeoutSeconds\":5},\"name\":\"jellyfin\",\"ports\":[{\"containerPort\":8096,\"hostPort\":null,\"name\":\"main\",\"protocol\":\"TCP\"}],\"readinessProbe\":{\"failureThreshold\":4,\"httpGet\":{\"path\":\"/\",\"port\":8096,\"scheme\":\"HTTP\"},\"initialDelaySeconds\":10,\"periodSeconds\":12,\"successThreshold\":2,\"timeoutSeconds\":5},\"resources\":{\"limits\":null,\"requests\":{\"cpu\":4,\"memory\":\"8Gi\"}},\"securityContext\":{\"allowPrivilegeEscalation\":true,\"capabilities\":{\"add\":[\"CHOWN\",\"SETUID\",\"SETGID\",\"FOWNER\",\"DAC_OVERRIDE\"],\"drop\":[\"ALL\"]},\"privileged\":true,\"readOnlyRootFilesystem\":true,\"runAsGroup\":0,\"runAsNonRoot\":false,\"runAsUser\":0,\"seccompProfile\":{\"type\":\"RuntimeDefault\"}},\"startupProbe\":{\"failureThreshold\":60,\"httpGet\":{\"path\":\"/\",\"port\":8096,\"scheme\":\"HTTP\"},\"initialDelaySeconds\":10,\"periodSeconds\":5,\"successThreshold\":1,\"timeoutSeconds\":3},\"stdin\":false,\"tty\":false,\"volumeMounts\":[{\"mountPath\":\"/cache\",\"name\":\"cache\",\"readOnly\":false},{\"mountPath\":\"/config\",\"name\":\"config\",\"readOnly\":false},{\"mountPath\":\"/dev/dma_heap\",\"name\":\"dev-dma-underscore-heap\",\"readOnly\":false},{\"mountPath\":\"/dev/dri\",\"name\":\"dev-dri\",\"readOnly\":false},{\"mountPath\":\"/dev/h265e\",\"name\":\"dev-h265e\",\"readOnly\":false},{\"mountPath\":\"/dev/hevc-service\",\"name\":\"dev-hevc-service\",\"readOnly\":false},{\"mountPath\":\"/dev/hevc_service\",\"name\":\"dev-hevc-underscore-service\",\"readOnly\":false},{\"mountPath\":\"/dev/iep\",\"name\":\"dev-iep\",\"readOnly\":false},{\"mountPath\":\"/dev/mali0\",\"name\":\"dev-mali0\",\"readOnly\":false},{\"mountPath\":\"/dev/mpp-service\",\"name\":\"dev-mpp-service\",\"readOnly\":false},{\"mountPath\":\"/dev/mpp_service\",\"name\":\"dev-mpp-underscore-service\",\"readOnly\":false},{\"mountPath\":\"/dev/rga\",\"name\":\"dev-rga\",\"readOnly\":false},{\"mountPath\":\"/dev/rkvdec\",\"name\":\"dev-rkvdec\",\"readOnly\":false},{\"mountPath\":\"/dev/rkvenc\",\"name\":\"dev-rkvenc\",\"readOnly\":false},{\"mountPath\":\"/dev/vepu\",\"name\":\"dev-vepu\",\"readOnly\":false},{\"mountPath\":\"/dev/vpu-service\",\"name\":\"dev-vpu-service\",\"readOnly\":false},{\"mountPath\":\"/dev/vpu_service\",\"name\":\"dev-vpu-underscore-service\",\"readOnly\":false},{\"mountPath\":\"/dev/shm\",\"name\":\"devshm\",\"readOnly\":false},{\"mountPath\":\"/var/lib/jellyfin\",\"name\":\"jellyfin\",\"readOnly\":false},{\"mountPath\":\"/data/media\",\"name\":\"media\",\"readOnly\":false},{\"mountPath\":\"/shared\",\"name\":\"shared\",\"readOnly\":false},{\"mountPath\":\"/var/lib/tls\",\"name\":\"tls\",\"readOnly\":false},{\"mountPath\":\"/tmp\",\"name\":\"tmp\",\"readOnly\":false},{\"mountPath\":\"/config/transcodes\",\"name\":\"transcode\",\"readOnly\":false},{\"mountPath\":\"/var/logs\",\"name\":\"varlogs\",\"readOnly\":false},{\"mountPath\":\"/var/run\",\"name\":\"varrun\",\"readOnly\":false}]}],\"dnsConfig\":{\"options\":[{\"name\":\"ndots\",\"value\":\"1\"}]},\"dnsPolicy\":\"ClusterFirst\",\"enableServiceLinks\":false,\"hostIPC\":false,\"hostNetwork\":false,\"hostPID\":false,\"hostUsers\":true,\"nodeSelector\":{\"jellyfin\":true,\"kubernetes.io/arch\":\"arm64\"},\"restartPolicy\":\"Always\",\"runtimeClassName\":null,\"securityContext\":{\"fsGroup\":0,\"fsGroupChangePolicy\":\"OnRootMismatch\",\"supplementalGroups\":[568],\"sysctls\":[]},\"serviceAccountName\":\"default\",\"shareProcessNamespace\":false,\"terminationGracePeriodSeconds\":60,\"tolerations\":[{\"effect\":\"NoSchedule\",\"key\":\"orangepi\",\"operator\":\"Exists\"}],\"topologySpreadConstraints\":[{\"labelSelector\":{\"matchLabels\":{\"app.kubernetes.io/instance\":\"jellyfin\",\"app.kubernetes.io/name\":\"jellyfin\",\"pod.name\":\"main\"}},\"maxSkew\":1,\"nodeAffinityPolicy\":\"Honor\",\"nodeTaintsPolicy\":\"Honor\",\"topologyKey\":\"kubernetes.io/hostname\",\"whenUnsatisfiable\":\"ScheduleAnyway\"}],\"volumes\":[{\"name\":\"cache\",\"persistentVolumeClaim\":{\"claimName\":\"jellyfin-cache\"}},{\"name\":\"config\",\"persistentVolumeClaim\":{\"claimName\":\"jellyfin-config\"}},{\"hostPath\":{\"path\":\"/dev/dma_heap\"},\"name\":\"dev-dma-underscore-heap\"},{\"hostPath\":{\"path\":\"/dev/dri\"},\"name\":\"dev-dri\"},{\"hostPath\":{\"path\":\"/dev/h265e\"},\"name\":\"dev-h265e\"},{\"hostPath\":{\"path\":\"/dev/hevc-service\"},\"name\":\"dev-hevc-service\"},{\"hostPath\":{\"path\":\"/dev/hevc_service\"},\"name\":\"dev-hevc-underscore-service\"},{\"hostPath\":{\"path\":\"/dev/iep\"},\"name\":\"dev-iep\"},{\"hostPath\":{\"path\":\"/dev/mali0\"},\"name\":\"dev-mali0\"},{\"hostPath\":{\"path\":\"/dev/mpp-service\"},\"name\":\"dev-mpp-service\"},{\"hostPath\":{\"path\":\"/dev/mpp_service\"},\"name\":\"dev-mpp-underscore-service\"},{\"hostPath\":{\"path\":\"/dev/rga\"},\"name\":\"dev-rga\"},{\"hostPath\":{\"path\":\"/dev/rkvdec\"},\"name\":\"dev-rkvdec\"},{\"hostPath\":{\"path\":\"/dev/rkvenc\"},\"name\":\"dev-rkvenc\"},{\"hostPath\":{\"path\":\"/dev/vepu\"},\"name\":\"dev-vepu\"},{\"hostPath\":{\"path\":\"/dev/vpu-service\"},\"name\":\"dev-vpu-service\"},{\"hostPath\":{\"path\":\"/dev/vpu_service\"},\"name\":\"dev-vpu-underscore-service\"},{\"emptyDir\":{\"medium\":\"Memory\"},\"name\":\"devshm\"},{\"name\":\"jellyfin\",\"persistentVolumeClaim\":{\"claimName\":\"jellyfin-jellyfin\"}},{\"name\":\"media\",\"nfs\":{\"path\":\"/export/media\",\"server\":\"\"}},{\"emptyDir\":{},\"name\":\"shared\"},{\"name\":\"tls\",\"secret\":{\"items\":[{\"key\":\"keystore.p12\",\"path\":\"keystore.pfx\"}],\"optional\":false,\"secretName\":\"jellyfin-tls\"}},{\"emptyDir\":{\"medium\":\"Memory\"},\"name\":\"tmp\"},{\"emptyDir\":{},\"name\":\"transcode\"},{\"emptyDir\":{\"medium\":\"Memory\"},\"name\":\"varlogs\"},{\"emptyDir\":{\"medium\":\"Memory\"},\"name\":\"varrun\"}]}}}}\n]] spec:map[template:map[metadata:map[annotations:map[checksum/services:a4a38d40bc3ed7512844016147721a1f46fd1bc7375f466225e04f9954c41d28]] spec:map[hostIPC:false hostNetwork:false hostPID:false hostUsers:true nodeSelector:map[jellyfin:true nodeType:<nil>] runtimeClassName:<nil> securityContext:map[sysctls:[]]]]]]": unrecognized type: string. Retrying attempt #3 at 8:47PM.
I think that should be the error to which this issue refers
Checklist:
argocd version
.Describe the bug
When a boolean value is used in the environment variables section of a deployment template in a Helm chart (which
helm template
validates successfully as can be seen below), our ArgoCD instance fails to sync the deployment (SyncFailed for deploymentapps/v1/...
) with the following difficult-to-read and inaccurate error message:This error message displayed in our ArgoCD version has the following issue:
Note that I know how to fix (and have fixed) the error, but that the goal of this ticket is to improve the error handling of ArgoCD's sync to make it easier for future developers who found the same issue.
To Reproduce
To reproduce the issue, create a Helm chart setup with the following
values.yaml
andtemplates/deployment.yaml
files:values.yaml
file:templates/deployment.yaml
file:Running
helm template
shows that it's a valid template:Notice that helm does not care that the
MY_VAR
value wasn't encapsulated in double quotes. Only ArgoCD appears to care, throwing the abovementioned error during application sync:"" is invalid: patch: Invalid value: "map[metadata:map[annotations:map[ ....... \n]] spec:map[template:map[spec:map[volumes:<nil>]]]]": unrecognized type: string
The fix is to edit the following lines in the
templates/deployment.yaml
file to ensure string encapsulation:But Argo's error message requires that the developer does quite a bit of unnecessary debugging, especially if there are many variables in the
values.yaml
file and a much more complicated deployment template.Expected behavior
What would be ideal: The current error message, which seems to imply that the value at fault is a string instead of a boolean, could be improved by highlighting that the value at fault is actually a boolean and not a string. For example, it would be really helpful for debugging if the error message was simply
patch failed due to the following issue: the value for key "MY_VAR" in the nested mapping was type <boolean>, but type <string> was expected. Value: "{\"name\":\"MY_VAR\",\"value\":false}"
What is perhaps more feasible: If it's not possible to drill down to the value that is causing the issue within the nested mapping, at least the following should be implemented if possible:
The above 2 fixes would be a significant improvement, since the error message would only be difficult to parse in that case, rather than difficult to parse and misleading.
Screenshots
N/A
Version
argocd version:
Logs