search

hashicorp / terraform-provider-kubernetes

Terraform Kubernetes provider
https://www.terraform.io/docs/providers/kubernetes/
Mozilla Public License 2.0
1.58k stars 968 forks source link

kubernetes_manifest crashes with `panic: lists must only contain one type of element` #1359

Closed bpoland closed 1 year ago

bpoland commented 3 years ago

Terraform Version, Provider Version and Kubernetes Version

Terraform version: v1.0.4
Kubernetes provider version: v2.4.1
Kubernetes version: 1.18

Affected Resource(s)

Terraform Configuration Files

https://gist.github.com/bpoland/19084c8de0c7b8222a484e5572efe4b2

Debug Output

https://gist.github.com/bpoland/26265a364a031081cf4d4b79dd948b4f

Panic Output

(see debug output above)

Steps to Reproduce

Expected Behavior

Actual Behavior

Important Factoids

References

Community Note

kingnarmer commented 3 years ago

I am getting same error deploying kafka cluster.

Strangerxxx commented 3 years ago

same with elasticsearch cluster

zikphil commented 2 years ago

Basically any ECK resources throws that error.

komljen commented 2 years ago

The same happens with the latest version as well v2.6.0.

komljen commented 2 years ago

For this particular issue, with Argo rollouts, if I remove this part from steps:

{
  "pause" = {
    "duration" = 10
  }
},

the issue is gone.

Seems like kubernetes manifest doesn't play well when the object is DynamicPseudoType. For Argo rollouts CRD it is defined as https://github.com/argoproj/argo-rollouts/blob/release-1.1/manifests/crds/rollout-crd.yaml#L475-L482.

Setting it as string or int, fixes the issue. I guess similar happens with other resources as well.

alexsomesan commented 2 years ago

Apologies, this issue slipped through the cracks. I was able to reproduce it now.

The problem here is caused by the duration attribute being defined as int-or-string in the OpenAPI schema of the Rollouts CRD. Terraform internally uses a very strict type system to represent resources and their state. The manifest resource uses OpenAPI definitions to dynamically generate the structure of the resource for each Kubernetes kind. Because int-or-string types cannot precisely be mapped to a Terraform type (they can either be a string or an int), we approximate it with a wildcard type, the DynamicPseudoType. What happens here is that objects containing these dynamic types are used as elements in a list (the steps list) and lists in Terraform have the invariant of all attributes needing to be of the same type.

What you see in that error message is a violation of that invariant. The fix is not trivial, but I'm exploring some options.

In the mean time, I think the workaround described by @komljen is the safest one: in the Rollout CRD, the schema for the duration attribute should be modified to just be of type string. This should still allow the Rollout objects to be consumed by the rollouts controller while allowing Terraform to keep the steps list consistently typed.

I'll update here as I progress with the permanent solution.

jaypz commented 2 years ago

Running into same issue when trying to use kubernetes_manifest to apply EnvoyFilter - additionally I noticed its not possible to deploy a Deployment using kubernetes_manifest it will also error on the same thing (obviously kubernetes_deployment works fine as an alternative)

zikphil commented 2 years ago

@alexsomesan Where are we with this fix?

alexsomesan commented 2 years ago

This should now be resolved in provider version v2.8.0

namdiag commented 2 years ago

after update to 2.8.0 provider i get this error 

Stack trace from the terraform-provider-kubernetes_v2.8.0_x5 plugin:

panic: ElementKeyInt(0): can't use tftypes.Object["authentication":tftypes.Object["enable":tftypes.Bool, "keyAuth":tftypes.Object["header":tftypes.String], "type":tftypes.String], "backends":tftypes.List[tftypes.Object["resolveGranularity":tftypes.String, "serviceName":tftypes.String, "servicePort":tftypes.Number, "subset":tftypes.String, "weight":tftypes.Number]], "match":tftypes.Object["exprs":tftypes.List[tftypes.Object["op":tftypes.String, "set":tftypes.List[tftypes.String], "subject":tftypes.Object["name":tftypes.String, "scope":tftypes.String], "value":tftypes.String]], "hosts":tftypes.List[tftypes.String], "methods":tftypes.List[tftypes.String], "paths":tftypes.List[tftypes.String], "remoteAddrs":tftypes.List[tftypes.String]], "name":tftypes.String, "plugins":tftypes.Tuple[tftypes.Object["config":tftypes.DynamicPseudoType, "enable":tftypes.Bool, "name":tftypes.String], tftypes.Object["config":tftypes.DynamicPseudoType, "enable":tftypes.Bool, "name":tftypes.String]], "priority":tftypes.Number, "timeout":tftypes.Object["connect":tftypes.String, "read":tftypes.String, "send":tftypes.String], "websocket":tftypes.Bool] as tftypes.Object["authentication":tftypes.Object["enable":tftypes.Bool, "keyAuth":tftypes.Object["header":tftypes.String], "type":tftypes.String], "backends":tftypes.List[tftypes.Object["resolveGranularity":tftypes.String, "serviceName":tftypes.String, "servicePort":tftypes.Number, "subset":tftypes.String, "weight":tftypes.Number]], "match":tftypes.Object["exprs":tftypes.List[tftypes.Object["op":tftypes.String, "set":tftypes.List[tftypes.String], "subject":tftypes.Object["name":tftypes.String, "scope":tftypes.String], "value":tftypes.String]], "hosts":tftypes.List[tftypes.String], "methods":tftypes.List[tftypes.String], "paths":tftypes.List[tftypes.String], "remoteAddrs":tftypes.List[tftypes.String]], "name":tftypes.String, "plugins":tftypes.Tuple[tftypes.Object["config":tftypes.DynamicPseudoType, "enable":tftypes.Bool, "name":tftypes.String]], "priority":tftypes.Number, "timeout":tftypes.Object["connect":tftypes.String, "read":tftypes.String, "send":tftypes.String], "websocket":tftypes.Bool]

goroutine 144 [running]:
github.com/hashicorp/terraform-plugin-go/tftypes.NewValue(...)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/vendor/github.com/hashicorp/terraform-plugin-go/tftypes/value.go:277
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.morphTupleIntoType(0x2fe9888, 0xc001ce5710, 0x2960860, 0xc000f32e88, 0x2fe9888, 0xc007715920, 0xc00217b950, 0x110000000bcbc6d0, 0x203001, 0x203001, ...)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/morph.go:190 +0x1bb9
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.ValueToType(0x2fe9888, 0xc001ce5710, 0x2960860, 0xc000f32e88, 0x2fe9888, 0xc007715920, 0xc00217b950, 0x2, 0x0, 0x0, ...)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/morph.go:30 +0xa09
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.morphObjectToType(0x2fe97e8, 0xc001ce5ad0, 0x2a60e60, 0xc001cd5770, 0x2fe97e8, 0xc007715950, 0xc00217b938, 0x1100000002fe97e8, 0xc007853dd0, 0x2a60e60, ...)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/morph.go:314 +0x425
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.ValueToType(0x2fe97e8, 0xc001ce5ad0, 0x2a60e60, 0xc001cd5770, 0x2fe97e8, 0xc007715950, 0xc00217b938, 0x1, 0xc007853dd0, 0x2a60e60, ...)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/morph.go:36 +0x709
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.morphObjectToType(0x2fe97e8, 0xc001ce5ef0, 0x2a60e60, 0xc001cd4ea0, 0x2fe97e8, 0xc007853500, 0xc00217b8a8, 0x1100000000000011, 0xc006491dd0, 0x8, ...)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/morph.go:314 +0x425
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.ValueToType(0x2fe97e8, 0xc001ce5ef0, 0x2a60e60, 0xc001cd4ea0, 0x2fe97e8, 0xc007853500, 0xc00217b8a8, 0x2, 0xb, 0x0, ...)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/morph.go:36 +0x709
github.com/hashicorp/terraform-provider-kubernetes/manifest/provider.(*RawProviderServer).PlanResourceChange(0xc000c40300, 0x2fd6cf8, 0xc0005df080, 0xc000eab540, 0xc00040e548, 0xc000badd01, 0x1)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/manifest/provider/plan.go:315 +0x1cbb
github.com/hashicorp/terraform-plugin-mux.SchemaServer.PlanResourceChange(0xc00111ac30, 0xc00111ac60, 0xc0018d3fc0, 0x2, 0x2, 0xc000c40280, 0x1, 0x2fd6cf8, 0xc0005df080, 0xc000eab540, ...)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/vendor/github.com/hashicorp/terraform-plugin-mux/schema_server.go:276 +0x98
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).PlanResourceChange(0xc00002e000, 0x2fd6da0, 0xc0009c1530, 0xc0018735e0, 0x0, 0x0, 0x0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server/server.go:777 +0x7f0
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_PlanResourceChange_Handler(0x2c1d160, 0xc00002e000, 0x2fd6da0, 0xc0009c1530, 0xc000ce1c80, 0x0, 0x2fd6da0, 0xc0009c1530, 0xc0001d9b00, 0x839)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:367 +0x214
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0004caa80, 0x2fed818, 0xc000454680, 0xc001386240, 0xc00111ad20, 0x3d93a68, 0x0, 0x0, 0x0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/vendor/google.golang.org/grpc/server.go:1282 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc0004caa80, 0x2fed818, 0xc000454680, 0xc001386240, 0x0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/vendor/google.golang.org/grpc/server.go:1616 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc000526830, 0xc0004caa80, 0x2fed818, 0xc000454680, 0xc001386240)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/vendor/google.golang.org/grpc/server.go:921 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/vendor/google.golang.org/grpc/server.go:919 +0x1fd
Bluesboy commented 2 years ago

Same appears on 2.9.0

Stack trace from the terraform-provider-kubernetes_v2.9.0_x5 plugin:

panic: ElementKeyInt(0): can't use tftypes.Object["definitionType":tftypes.String, "files":tftypes.DynamicPseudoType, "internalReplication":tftypes.String, "name":tftypes.String, "replicas":tftypes.Tuple[tftypes.Object["files":tftypes.DynamicPseudoType, "httpPort":tftypes.Number, "interserverHTTPPort":tftypes.Number, "name":tftypes.String, "settings":tftypes.DynamicPseudoType, "tcpPort":tftypes.Number, "templates":tftypes.Object["clusterServiceTemplate":tftypes.String, "dataVolumeClaimTemplate":tftypes.String, "hostTemplate":tftypes.String, "logVolumeClaimTemplate":tftypes.String, "podTemplate":tftypes.String, "replicaServiceTemplate":tftypes.String, "serviceTemplate":tftypes.String, "shardServiceTemplate":tftypes.String, "volumeClaimTemplate":tftypes.String]], tftypes.Object["files":tftypes.DynamicPseudoType, "httpPort":tftypes.Number, "interserverHTTPPort":tftypes.Number, "name":tftypes.String, "settings":tftypes.DynamicPseudoType, "tcpPort":tftypes.Number, "templates":tftypes.Object["clusterServiceTemplate":tftypes.String, "dataVolumeClaimTemplate":tftypes.String, "hostTemplate":tftypes.String, "logVolumeClaimTemplate":tftypes.String, "podTemplate":tftypes.String, "replicaServiceTemplate":tftypes.String, "serviceTemplate":tftypes.String, "shardServiceTemplate":tftypes.String, "volumeClaimTemplate":tftypes.String]], tftypes.Object["files":tftypes.DynamicPseudoType, "httpPort":tftypes.Number, "interserverHTTPPort":tftypes.Number, "name":tftypes.String, "settings":tftypes.DynamicPseudoType, "tcpPort":tftypes.Number, "templates":tftypes.Object["clusterServiceTemplate":tftypes.String, "dataVolumeClaimTemplate":tftypes.String, "hostTemplate":tftypes.String, "logVolumeClaimTemplate":tftypes.String, "podTemplate":tftypes.String, "replicaServiceTemplate":tftypes.String, "serviceTemplate":tftypes.String, "shardServiceTemplate":tftypes.String, "volumeClaimTemplate":tftypes.String]]], "replicasCount":tftypes.Number, "settings":tftypes.DynamicPseudoType, "templates":tftypes.Object["clusterServiceTemplate":tftypes.String, "dataVolumeClaimTemplate":tftypes.String, "hostTemplate":tftypes.String, "logVolumeClaimTemplate":tftypes.String, "podTemplate":tftypes.String, "replicaServiceTemplate":tftypes.String, "serviceTemplate":tftypes.String, "shardServiceTemplate":tftypes.String, "volumeClaimTemplate":tftypes.String], "weight":tftypes.Number] as tftypes.Object["definitionType":tftypes.String, "files":tftypes.DynamicPseudoType, "internalReplication":tftypes.String, "name":tftypes.String, "replicas":tftypes.Tuple[tftypes.Object["files":tftypes.DynamicPseudoType, "httpPort":tftypes.Number, "interserverHTTPPort":tftypes.Number, "name":tftypes.String, "settings":tftypes.DynamicPseudoType, "tcpPort":tftypes.Number, "templates":tftypes.Object["clusterServiceTemplate":tftypes.String, "dataVolumeClaimTemplate":tftypes.String, "hostTemplate":tftypes.String, "logVolumeClaimTemplate":tftypes.String, "podTemplate":tftypes.String, "replicaServiceTemplate":tftypes.String, "serviceTemplate":tftypes.String, "shardServiceTemplate":tftypes.String, "volumeClaimTemplate":tftypes.String]]], "replicasCount":tftypes.Number, "settings":tftypes.DynamicPseudoType, "templates":tftypes.Object["clusterServiceTemplate":tftypes.String, "dataVolumeClaimTemplate":tftypes.String, "hostTemplate":tftypes.String, "logVolumeClaimTemplate":tftypes.String, "podTemplate":tftypes.String, "replicaServiceTemplate":tftypes.String, "serviceTemplate":tftypes.String, "shardServiceTemplate":tftypes.String, "volumeClaimTemplate":tftypes.String], "weight":tftypes.Number]
alexsomesan commented 2 years ago

@Bluesboy it's hard to debug your report without seeing the causing manifest. Can you please share it?

hisantoshverma commented 2 years ago

I observed same issue even on 2.10.0 !!

Error: The terraform-provider-kubernetes_v2.10.0_x5 plugin crashed!

alexsomesan commented 2 years ago

@hisantoshverma would you care to share the configuration that reproduces the crash in your case?

hisantoshverma commented 2 years ago

@alexsomesan thanks for your response :

Below is my provider.tf

terraform { required_version = ">= 0.14.11"

required_providers { aws = { source = "hashicorp/aws" version = "~> 3.56.0" } helm = { source = "hashicorp/helm" version = "2.4.1" } kubernetes = { source = "hashicorp/kubernetes" version = "2.7.1" // Already tested with 2.8.1, 2.9.1 and 2.10.0 }

reproduces steps: I faced the issue during the auth0 filter & filter policy creation of the ambassador API gateway through terraform: https://www.getambassador.io/docs/edge-stack/latest/howtos/sso/auth0/#configure-filter-and-filterpolicy

github-actions[bot] commented 1 year ago

Marking this issue as stale due to inactivity. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. This helps our maintainers find and focus on the active issues. Maintainers may also remove the stale label at their discretion. Thank you!