search

kubernetes-sigs / kustomize

Customization of kubernetes YAML configurations
Apache License 2.0
10.76k stars 2.22k forks source link

Support for integer keys #3446

Open sleepycat opened 3 years ago

sleepycat commented 3 years ago

Attempting to build Knative config with 3.9.0 on Manjaro Linux produces the error Error: json: unsupported type: map[interface {}]interface {}.

[mike@ouroboros minikube]$ kustomize version
{Version:3.9.0 GitCommit:$Format:%H$ BuildDate:2020-12-13T07:57:44Z GoOs:linux GoArch:amd64}
[mike@ouroboros minikube]$ cat kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- https://github.com/knative/serving/releases/download/v0.19.0/serving-core.yaml
[mike@ouroboros minikube]$ kustomize build .
Error: json: unsupported type: map[interface {}]interface {}

Kustomize 3.8 will successfully build.

[mike@ouroboros minikube]$ kustomize version
{Version:kustomize/v3.8.8 GitCommit:72262c5e7135045ed51b01e417a7e72f558a22b0 BuildDate:2020-12-10T18:05:35Z GoOs:linux GoArch:amd64}
[mike@ouroboros minikube]$ kustomize build .
apiVersion: v1
kind: Namespace
metadata:
  labels:
    serving.knative.dev/release: v0.19.0
  name: knative-serving
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  labels:
    knative.dev/crd-install: "true"
    serving.knative.dev/release: v0.19.0
  name: certificates.networking.internal.knative.dev
...
Shell32-Natsu commented 3 years ago

We have issues in 3.9.0. Please try 3.9.1.

stefanprodan commented 3 years ago

@Shell32-Natsu can you shed some light on this? I'm using sigs.k8s.io/kustomize/api v0.7.1 in fluxcd/kustomize-controller but this bug is still there.

I find it very hard to keep up with kustomize upstream changes, there is no changelog expect for a list of commits. For Flux users is even harder, with error outputs like json: unsupported type: map[interface {}]interface {} it's impossible to track down which object from all the manifests caused this.

echel0n commented 3 years ago

This issue is still present in API v0.7.2 as well

stefanprodan commented 3 years ago

This turns out to be a bug in kyaml, here is how to reproduce it.

Having a custom resource with an integer key:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: podinfo
spec:
  interval: 5m
  chart:
    spec:
      chart: podinfo
      sourceRef:
        kind: HelmRepository
        name: podinfo
  values:
    tcp:
      8080: "default/example-tcp-svc:9000"

Fails with:

$ kustomize build
Error: error marshaling into JSON: json: unsupported type: map[interface {}]interface {}

Works with:

kustomize build --enable_kyaml=false

Version:

kustomize version
{Version:kustomize/v3.9.2 GitCommit: BuildDate:2021-01-17T19:01:12+00:00 GoOs:darwin GoArch:amd64}

What I find really concerning is that users have no way to identify the object/resource/file when kustomize build fails. @Shell32-Natsu what is the recommend way to debug this when you have hundreds of manifests?

Shell32-Natsu commented 3 years ago

@stefanprodan Thanks for the information. I can reproduce it now. Will investigate it.

echel0n commented 3 years ago

Is there any chance you can add in the logging of the filename that kicks back the exception as well, please?

Shell32-Natsu commented 3 years ago

This issue is triggered by incompatibility between YAML marshal and unmarshal. I will create a PR to add the resources that causes this error to the error message.

More details about this issue: When kustomize is trying to get the YAML bytes from a resource, it will convert the resource into a map[string]interface{} then call yaml.Marshal. The resource is decoded to map[string]interface{} by yaml.Node.Decode. When there is an integer key, that field will be decoded as map[interface{}]interface{} instead of map[string]interface{} and that is invalid in yaml.Marshal.

Shell32-Natsu commented 3 years ago

After we delete the apimachinery codes we can get the YAML bytes from the RNode directly so this can be fixed.

Shell32-Natsu commented 3 years ago

@monopole

HansK-p commented 3 years ago

I have the same issue when using example code from https://istio.io/latest/docs/reference/config/security/peer_authentication:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: foo
spec:
  selector:
    matchLabels:
      app: finance
  mtls:
    mode: STRICT
  portLevelMtls:
    8080:
      mode: DISABLE

I'm also using similar code myself, and my only workaround right now is to stop using Kustomize for this particular deployment - or downgrade Kustomize. The argument --enable_kyaml=false doesn't seem to be a valid anymore.

nairb774 commented 3 years ago

Thank you @HansK-p for your comment as you helped identify (for me) what I need to do to work around the bug. The portLevelMtls having a key of 8080 is causing the kyaml parser to make that an int of some sort rather than the expected string. By changing it to "8080" it works again. ~Going to move this to a slightly different bug, as there is a good reproduction case, and I'm not sure this overlaps with the other issues identified in this issue.~

Edit: Just fully read https://github.com/kubernetes-sigs/kustomize/issues/3446#issuecomment-763853739 - sorry about the spam.

HansK-p commented 3 years ago

And thank you. I feel a bit stupid not testing that myself.

monopole commented 3 years ago

@Shell32-Natsu I've started #3588

this should be easier once that's done. need a few days

Shell32-Natsu commented 3 years ago

@monopole Thanks!

r0bj commented 3 years ago

I see the same error trying to kustomize tekton pipelines (https://github.com/tektoncd/pipeline):

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.21.0/release.yaml
$ kustomize build
Error: map[string]interface {}{"apiVersion":"apiextensions.k8s.io/v1", "kind":"CustomResourceDefinition", "metadata":map[string]interface {}{"labels":map[string]interface {}{"app.kubernetes.io/instance":"default", "app.kubernetes.io/part-of":"tekton-pipelines", "pipeline.tekton.dev/release":"v0.21.0", "version":"v0.21.0"}, "name":"clustertasks.tekton.dev"}, "spec":map[string]interface {}{"conversion":map[string]interface {}{"strategy":"Webhook", "webhook":map[string]interface {}{"clientConfig":map[string]interface {}{"service":map[string]interface {}{"name":"tekton-pipelines-webhook", "namespace":"tekton-pipelines"}}, "conversionReviewVersions":[]interface {}{"v1beta1"}}}, "group":"tekton.dev", "names":map[string]interface {}{"categories":[]interface {}{"tekton", "tekton-pipelines"}, "kind":"ClusterTask", "plural":"clustertasks"}, "preserveUnknownFields":false, "scope":"Cluster", "versions":[]interface {}{map[string]interface {}{"name":"v1alpha1", "schema":map[string]interface {}{"openAPIV3Schema":map[string]interface {}{"type":"object", "x-kubernetes-preserve-unknown-fields":true}}, "served":true, "storage":false, "subresources":map[string]interface {}{"status":map[string]interface {}{}}}, map[interface {}]interface {}{"name":"v1beta1", "schema":map[string]interface {}{"openAPIV3Schema":map[string]interface {}{"type":"object", "x-kubernetes-preserve-unknown-fields":true}}, "served":true, "storage":true, "subresources":map[string]interface {}{"status":map[string]interface {}{}}}}}}: json: unsupported type: map[interface {}]interface {}

It works fine with older kustomize 3.8, fails for newer versions.

bhagyesh18 commented 3 years ago

Kustomize Version : {Version:kustomize/v4.0.5 GitCommit:9e8e7a7fe99ec9fbf801463e8607928322fc5245 BuildDate:2021-03-08T20:53:03Z GoOs:linux GoArch:amd64}

I am having same issue running the following command 

kustomize build .

Error: map[string]interface {}{"apiVersion":"v1", "data":map[interface {}]interface {}
{22:"default/gitlab-test-gitlab-shell:22"}, "kind":"ConfigMap", "metadata":map[string]interface {}
{"labels":map[string]interface {}{"app":"gitlab-shell", "chart":"gitlab-shell-4.9.3", "heritage":"Helm", 
"release":"gitlab-test"}, "name":"gitlab-test-nginx-ingress-tcp", "namespace":"default"}}: 
json: unsupported type: map[interface {}]interface {}

Actual manifest file with integer key 

apiVersion: v1
kind: ConfigMap
metadata:
  name: gitlab-nginx-ingress-tcp
  namespace: default
  labels:
    app: gitlab-shell
    chart: gitlab-shell-4.9.3
    release: gitlab
    heritage: Tiller

data:
  22: "default/gitlab-gitlab-shell:22"

I have rendered manifest more than 15,00 lines, Please show the manifest area where this error is coming. Also, Should I expect the integer key part of manifest? is there anything to do with kustomize to ignore integer? Please suggest, I have been looking at this issue for a month.

Shell32-Natsu commented 3 years ago

@bhagyesh18 Sorry for the inconvenient. But unfortunately the integer key is not allowed in kustomize now but @monopole is working on deleting translation layers so this issue might be able to be fixed soon.

bhagyesh18 commented 3 years ago

was it working earlier in any previous version? Also, there are duplicate Keys issues, I can see in a new version that you guys show the actual manifest area where it has duplicate keys, that is good. Is there any way to ignore verifying/unmarshling duplicate keys using kustomize and let it be as it is?

Shell32-Natsu commented 3 years ago

@bhagyesh18 You can try the version before v4 with flag --enable_kyaml=false. For the duplicate keys, invalid YAML is invalid, we cannot generate an invalid output.

pvaneck commented 3 years ago

Ran into this issue when trying to use Kustomize with a Knative serving YAML. Seems like the error stems from the usage of !!merge <<: *version in a lot of these YAMLs including the Tekton pipelines YAML that another user mentioned above.

Simple recreation:

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- test.yaml

test.yaml

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: test.kustomize.error
spec:
  group: test.group
  versions:
    - &version
      name: v1alpha1
      served: true
      storage: false
      schema:
        openAPIV3Schema:
          type: object
    - !!merge <<: *version
      name: v1beta1

Using kustomize v4.0.5:

kustomize build .
Error: map[string]interface {}{"apiVersion":"apiextensions.k8s.io/v1", "kind":"CustomResourceDefinition", "metadata":map[string]interface {}{"name":"test.kustomize.error"}, "spec":map[string]interface {}{"group":"test.group", "versions":[]interface {}{map[string]interface {}{"name":"v1alpha1", "schema":map[string]interface {}{"openAPIV3Schema":map[string]interface {}{"type":"object"}}, "served":true, "storage":false}, map[interface {}]interface {}{"name":"v1beta1", "schema":map[string]interface {}{"openAPIV3Schema":map[string]interface {}{"type":"object"}}, "served":true, "storage":false}}}}: json: unsupported type: map[interface {}]interface {}

In older versions of kustomize (like 3.8.10), kustomize builds properly and outputs:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: test.kustomize.error
spec:
  group: test.group
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        type: object
    served: true
    storage: false
  - name: v1beta1
    schema:
      openAPIV3Schema:
        type: object
    served: true
    storage: false
Shell32-Natsu commented 3 years ago

@pvaneck Your issue should be the same as #3614 instead of this.

mathieux51 commented 3 years ago

I experienced that issue as well with v4.1.2. 

kustomize version
{Version:kustomize/v4.1.2 GitCommit:a5914abad89e0b18129eaf1acc784f9fe7d21439 BuildDate:2021-04-15T23:23:03+01:00 GoOs:darwin GoArch:amd64}

Now if I use v3.8.10, kustomize build works. https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv3.8.10

jhass commented 3 years ago

For the interested readers, I experienced similar issues from using YAML anchors and references, so not involving any numerical keys:

foo: &foo
  one: two
bar:
  <<: *foo
KnVerey commented 3 years ago

/retitle Support for integer keys

Let's keep this issue focussed on the original problem, i.e. integer key handling.

If you've arrived at this issue because of a similar error message encountered when using YAML anchors, please follow https://github.com/kubernetes-sigs/kustomize/issues/3675 instead.

perezjasonr commented 3 years ago

Kustomize Version : {Version:kustomize/v4.0.5 GitCommit:9e8e7a7fe99ec9fbf801463e8607928322fc5245 BuildDate:2021-03-08T20:53:03Z GoOs:linux GoArch:amd64}

I am having same issue running the following command

kustomize build .

Error: map[string]interface {}{"apiVersion":"v1", "data":map[interface {}]interface {}
{22:"default/gitlab-test-gitlab-shell:22"}, "kind":"ConfigMap", "metadata":map[string]interface {}
{"labels":map[string]interface {}{"app":"gitlab-shell", "chart":"gitlab-shell-4.9.3", "heritage":"Helm", 
"release":"gitlab-test"}, "name":"gitlab-test-nginx-ingress-tcp", "namespace":"default"}}: 
json: unsupported type: map[interface {}]interface {}

Actual manifest file with integer key

apiVersion: v1
kind: ConfigMap
metadata:
  name: gitlab-nginx-ingress-tcp
  namespace: default
  labels:
    app: gitlab-shell
    chart: gitlab-shell-4.9.3
    release: gitlab
    heritage: Tiller

data:
  22: "default/gitlab-gitlab-shell:22"

I have rendered manifest more than 15,00 lines, Please show the manifest area where this error is coming. Also, Should I expect the integer key part of manifest? is there anything to do with kustomize to ignore integer? Please suggest, I have been looking at this issue for a month.

wow! i just ran into this EXACT one too. Glad to know I'm not crazy. I guess I'll try to downgrade kustomize for now.

Error: error while running post render on files: error while running command /tmp/post-render/kustomize. error output:
    Error: map[string]interface {}{"apiVersion":"v1", "data":map[interface {}]interface {}{22:"gitlab/gitlab-gitlab-shell:22"}, "kind":"ConfigMap", "metadata":map[string]interface {}{"labels":map[string]interface {}{"app":"gitlab-shell", "chart":"gitlab-shell-4.12.4", "heritage":"Helm", "release":"gitlab"}, "name":"gitlab-nginx-ingress-tcp", "namespace":"gitlab"}}: json: unsupported type: map[interface {}]interface {}

@bhagyesh18 i was able to get past it for now w/ the workaround mentioned earlier, downgrade to a 3.9-ish version and disable kyaml.

eigood commented 2 years ago

I have the same issue when using example code from https://istio.io/latest/docs/reference/config/security/peer_authentication:

apiVersion: security.istio.io/v1beta1 kind: PeerAuthentication metadata: name: default namespace: foo spec: selector: matchLabels: app: finance mtls: mode: STRICT portLevelMtls: 8080: mode: DISABLE I'm also using similar code myself, and my only workaround right now is to stop using Kustomize for this particular deployment - or downgrade Kustomize. The argument --enable_kyaml=false doesn't seem to be a valid anymore.

I don't see istio changing their CRD, so is there any plan on fixing this in kustomize?

alexdyas commented 2 years ago

Bump, this still seems to be an issue in 4.4.0. Any plans for a fix?

The Kubernetes Nginx Ingress documentation example for exposing TCP ports triggers the error:

https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/

Specifically:

apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-services
  namespace: ingress-nginx
data:
  9000: "default/example-go:8080"

Thanks

fe80 commented 2 years ago

Hello same issue with 4.4.1 :)

╰─➤ kustomize version 
{Version:4.4.1 GitCommit:$Format:%H$ BuildDate:2021-12-17T23:11:01Z GoOs:linux GoArch:amd64}
midN commented 2 years ago

@natasha41575 why is this being tracked on Roadmap for 2022 and not as a hotfix for actual regression/bug?

This should be fixed ASAP, it's a regression and huge limitation for creating configmaps.

natasha41575 commented 2 years ago

@midN Unfortunately there are only two of us working on kustomize with competing priorities both within and outside of kustomize. Because of our lack of resources we are unable to immediately prioritize and fix every regression in a timely manner. The roadmap lists the ones we consider the most important and hope to get to as soon as we can.

Edited to add: anyone is welcome to contribute a PR to address this issue. We have created the roadmap so that outside contributors know which PRs and fixes would be the most helpful to us and that we have the bandwidth to review.

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

antoineozenne commented 2 years ago

/remove-lifecycle stale

dwalters commented 2 years ago

PR #4604 fixes this, could someone review it?

Akay7 commented 2 years ago

@alexdyas is there are any workaround for your case:

apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-services
  namespace: ingress-nginx
data:
  9000: "default/example-go:8080"

?

alexdyas commented 1 year ago

Hi @Akay7,

The only work around I've found at the moment is to use an older version of kustomize. We're using v3.8.10. However beware that there are various unrelated bugs in this version, so check the output.

Hope that helps.

Alex

mghantous commented 1 year ago

Quoting the integer key seemed to work for our ingress tcp configmap.

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

dwalters commented 1 year ago

/remove-lifecycle stale

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

antoineozenne commented 1 year ago

/remove-lifecycle stale

daurnimator commented 1 year ago

Quoting the integer key seemed to work for our ingress tcp configmap.

This doesn't seem to work, though the kustomize build succeeds, the configmap key gets the " in it. This then fails to apply with:

ConfigMap "tcp-services" is invalid: data["22"]: Invalid value: ""22"": a valid config key must consist of alphanumeric characters, '-', '_' or '.' (e.g. 'key.name', or 'KEY_NAME', or 'key-name', regex used for validation is '[-._a-zA-Z0-9]+'),
[... 2 more screenfuls of errors, maybe cascading from that one?]
wibed commented 7 months ago

@natasha41575 please add back to core usability bugs.

nginx-ingress-controller might have monkey patched around the issue but it still persists.

Garett-MacGowan commented 5 months ago

+1