kubernetes_manifest secret stringData inconsistent error

[akumar-99]

Terraform Version, Provider Version and Kubernetes Version

Terraform version: v1.5.1
Kubernetes provider version: 2.21.1
Kubernetes version: 1.24

Affected Resource(s)

Terraform Configuration Files

resource "kubernetes_manifest" "eck-secret-elastic-filebeat-ingestion-role" {
  manifest = yamldecode(templatefile("../scripts/eck/eck-secret-elastic-filebeat-ingestion-role.yaml", {
    namespace = kubernetes_namespace.observability.metadata[0].name
  }))
}

kind: Secret
apiVersion: v1
metadata:
  name: elastic-filebeat-ingestion-role
  namespace: ${namespace}
stringData:
  roles.yml: |-
    elastic-filebeat-ingestion-role:
      run_as: []
      cluster: [
        "monitor",
        "all",
        "manage"
      ]
      indices:
      - names: [ "*-pod-*" ]
        privileges: [
          "create",
          "create_doc",
          "delete",
          "index",
          "write",
          "all",
          "auto_configure",
          "create_index",
          "manage"
        ]
        field_security:
          grant: [ "*" ]

Debug Output

https://gist.github.com/akumar-99/99a271e72ddf59b43ca146bd2d57d988

Panic Output

Steps to Reproduce

terraform apply -target=kubernetes_manifest.eck-secret-elastic-filebeat-ingestion-role

Expected Behavior

The kubernetes secret should have been created without any error.

Actual Behavior

The resource got created successfully but terraform gave an error.

│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to kubernetes_manifest.eck-secret-elastic-filebeat-ingestion-role, provider
│ "provider[\"registry.terraform.io/hashicorp/kubernetes\"]" produced an unexpected new value: .object.stringData: was
│ cty.MapVal(map[string]cty.Value{"roles.yml":cty.StringVal("elastic-filebeat-ingestion-role:\n  run_as: []\n  cluster: [\n    \"monitor\",\n
│ \"all\",\n    \"manage\"\n  ]\n  indices:\n  - names: [ \"*-pod-*\" ]\n    privileges: [\n      \"create\",\n      \"create_doc\",\n
│ \"delete\",\n      \"index\",\n      \"write\",\n      \"all\",\n      \"auto_configure\",\n      \"create_index\",\n      \"manage\"\n    ]\n
│ field_security:\n      grant: [ \"*\" ]")}), but now null.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵

Important Factoids

References

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[akumar-99]

This is what I am using right now which works.

kind: Secret
apiVersion: v1
metadata:
  name: elastic-filebeat-ingestion-role
  namespace: ${namespace}
data:
  roles.yml: ${base64encode(<<EOF
    elastic-filebeat-ingestion-role:
      run_as: []
      cluster: [
        "monitor",
        "all",
        "manage"
      ]
      indices:
      - names: [ "*-pod-*" ]
        privileges: [
          "create",
          "create_doc",
          "delete",
          "index",
          "write",
          "all",
          "auto_configure",
          "create_index",
          "manage"
        ]
        field_security:
          grant: [ "*" ]
EOF
)}

[arybolovlev]

Hi @akumar-99,

Thank you for sharing your solution. This seems to be expected.

Kubernetes Secret stores all secrets as a base64 encoded string. The secrets are available under the data field. The stringData field is used only to create a new secret and it accepts arbitrary strings as values. Once it is accepted, it is encoded into base64 string and stored under the data field. In this case, in a response from the Kubernetes API, you would see stringData as null(or empty value).

As it seems to work as expected and you have found a solution, I will go ahead and close this issue.

Thanks!

[akumar-99]

@arybolovlev If it is an expected behavior, wouldn't it be better if there would be a way to handle it gracefully instead of exiting out with error?

[kizzie]

Yes... having something more graceful would be nice for the exit on this one - I just spent the last couple of days trawling through the code to work out why it was doing what it was doing as well... maybe in the unmarshall process it could add the decrypted data to the stringData field again or something so that terraform handles it better?

[alexsomesan]

stringData is a convenience mechanism intended for use with clients that don't offer the ability to encode input data (such as kubectl). When applicable, it is preferred to use data in conjunction with an own text encoding mechanism (base64 or other).

Kubernetes themselves document that:

Note: The stringData field for a Secret does not work well with server-side apply.

As the kubernetes_manifest resource uses server-side-apply, this is one more reason to avoid using stringData.

We'll try to make it more obvious in our documentation.