Open mailbox171 opened 11 months ago
mailbox171
commented
11 months ago Additional comment:
All mentioned fields in the error messages start with ".object" so I guess I could try the workaround below. But, I don't know if it can be dangerous. What could be the negative consequences? Impacts?
computed_fields = ["metadata.labels", "metadata.annotations", "object"]
sheneska
commented
11 months ago Hi, @mailbox171 Could you please provide the terraform config that was used?
mailbox171
commented
11 months ago resource "kubernetes_manifest" "buildbuddy" {
manifest = yamldecode(templatefile(var.argocd_application_file, {
"app_path" = var.app_path
}))
}
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: sdp-apps
namespace: argocd
annotations:
argocd.argoproj.io/sync-options: Delete=false
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
destination:
server: https://kubernetes.default.svc
namespace: argocd
project: default
source:
path: ${app_path}
repoURL: git@github.com:Stellantis-ADX/engineering-platform.git
targetRevision: prod
syncPolicy:
automated:
selfHeal: true
# prune: true
syncOptions:
- ApplyOutOfSyncOnly=true
- PruneLast=true
- CreateNamespace=trueterraform {
backend "azurerm" {}
required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = ">=3.44"
}
helm = {
source = "hashicorp/helm"
version = ">=2.9.0"
}
kubernetes = {
source = "hashicorp/kubernetes"
version = ">= 2.19"
}
}
}
provider "azurerm" {
features {
}
}
provider "kubernetes" {
host = data.azurerm_kubernetes_cluster.aks.kube_config.0.host
client_key = base64decode(data.azurerm_kubernetes_cluster.aks.kube_config.0.client_key)
client_certificate = base64decode(data.azurerm_kubernetes_cluster.aks.kube_config.0.client_certificate)
cluster_ca_certificate = base64decode(data.azurerm_kubernetes_cluster.aks.kube_config.0.cluster_ca_certificate)
}
provider "helm" {
debug = true
kubernetes {
host = data.azurerm_kubernetes_cluster.aks.kube_config.0.host
client_key = base64decode(data.azurerm_kubernetes_cluster.aks.kube_config.0.client_key)
client_certificate = base64decode(data.azurerm_kubernetes_cluster.aks.kube_config.0.client_certificate)
cluster_ca_certificate = base64decode(data.azurerm_kubernetes_cluster.aks.kube_config.0.cluster_ca_certificate)
}
}
mailbox171
commented
10 months ago Hello @sheneska. Did you have the chance to give a look?
Maybe the argocd operator/crds are updating the fields: .object.operation.sync.resources .object.operation.sync.syncOptions .object.operation.sync.revision .object.operation.sync.revision .object.operation.initiatedBy.automated
Then, adding "object" to computed fields may solve the issue?
computed_fields = ["metadata.labels", "metadata.annotations", "object"]What I am not able to assess is if there can be unwanted (dangerous) side-effects.
Thanks a lot
jlenuffgsoi
commented
9 months ago I encounter the same issue.
I added the compute_fields as above, I still have this error :
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to kubernetes_manifest.mongo_mongodbcommunity, provider "provider[\"registry.terraform.io/hashicorp/kubernetes\"]" produced an unexpected new value:
│ .object.metadata.annotations["mongodb.com/v1.lastSuccessfulConfiguration"]: was
│ cty.StringVal("{\"members\":3,\"type\":\"ReplicaSet\",\"version\":\"4.4.27\",\"arbiters\":0,\"replicaSetHorizons\":[{\"mycompany\":\"mongodb44-rs-0.gke.prod.mycompany.fr:33241\"},{\"mycompany\":\"mongodb44-rs-1.gke.prod.mycompany.fr:33242\"},{\"mycompany\":\"mongodb44-rs-2.gke.prod.mycompany.fr:33243\"}],\"security\":{\"authentication\":{\"modes\":[\"SCRAM-SHA-1\",\"SCRAM\"],\"ignoreUnknownUsers\":true},\"tls\":{\"enabled\":true,\"optional\":true,\"certificateKeySecretRef\":{\"name\":\"tls-certificate\"},\"caCertificateSecretRef\":{\"name\":\"tls-ca-key-pair\"}}},\"users\":[{\"name\":\"mongodb44-admin-mongo\",\"db\":\"admin\",\"passwordSecretRef\":{\"name\":\"mongodb44-admin-mongo-password\",\"key\":\"\"},\"roles\":[{\"db\":\"admin\",\"name\":\"clusterAdmin\"},{\"db\":\"admin\",\"name\":\"root\"},{\"db\":\"admin\",\"name\":\"userAdminAnyDatabase\"},{\"db\":\"admin\",\"name\":\"readWriteAnyDatabase\"},{\"db\":\"admin\",\"name\":\"dbAdminAnyDatabase\"}],\"scramCredentialsSecretName\":\"mongodb44-admin-mongo\",\"additionalConnectionStringConfig\":null},{\"name\":\"tutui-toto-prod\",\"db\":\"admin\",\"passwordSecretRef\":{\"name\":\"tutui-toto-prod-password\",\"key\":\"\"},\"roles\":[{\"db\":\"tutui-toto-prod\",\"name\":\"readWrite\"}],\"scramCredentialsSecretName\":\"tutui-toto-prod\",\"additionalConnectionStringConfig\":null},{\"name\":\"app2\",\"db\":\"admin\",\"passwordSecretRef\":{\"name\":\"app2-password\",\"key\":\"\"},\"roles\":[{\"db\":\"app2\",\"name\":\"readWrite\"}],\"scramCredentialsSecretName\":\"app2\",\"additionalConnectionStringConfig\":null},{\"name\":\"app1\",\"db\":\"admin\",\"passwordSecretRef\":{\"name\":\"app1-password\",\"key\":\"\"},\"roles\":[{\"db\":\"admin\",\"name\":\"readWriteAnyDatabase\"}],\"scramCredentialsSecretName\":\"app1\",\"additionalConnectionStringConfig\":null},{\"name\":\"tutu-toto-prod\",\"db\":\"admin\",\"passwordSecretRef\":{\"name\":\"tutu-toto-prod-password\",\"key\":\"\"},\"roles\":[{\"db\":\"tutu-toto-prod\",\"name\":\"readWrite\"}],\"scramCredentialsSecretName\":\"tutu-toto-prod\",\"additionalConnectionStringConfig\":null},{\"name\":\"titi-toto-prod\",\"db\":\"admin\",\"passwordSecretRef\":{\"name\":\"titi-toto-prod-password\",\"key\":\"\"},\"roles\":[{\"db\":\"titi-toto-prod\",\"name\":\"readWrite\"}],\"scramCredentialsSecretName\":\"titi-toto-prod\",\"additionalConnectionStringConfig\":null},{\"name\":\"titi-tata-prod\",\"db\":\"admin\",\"passwordSecretRef\":{\"name\":\"titi-tata-prod-password\",\"key\":\"\"},\"roles\":[{\"db\":\"titi-tata-prod\",\"name\":\"readWrite\"}],\"scramCredentialsSecretName\":\"titi-tata-prod\",\"additionalConnectionStringConfig\":null}],\"statefulSet\":{\"spec\":{},\"metadata\":{}},\"agent\":{\"logLevel\":\"\",\"logFile\":\"\",\"maxLogFileDurationHours\":0},\"additionalMongodConfig\":{},\"prometheus\":{\"username\":\"prometheus\",\"passwordSecretRef\":{\"name\":\"prometheus-password\",\"key\":\"\"},\"tlsSecretKeyRef\":{\"name\":\"\",\"key\":\"\"}},\"additionalConnectionStringConfig\":{}}"),
│ but now
│ cty.StringVal("{\"members\":3,\"type\":\"ReplicaSet\",\"version\":\"4.4.27\",\"arbiters\":0,\"replicaSetHorizons\":[{\"mycompany\":\"mongodb44-rs-0.gke.prod.mycompany.fr:33241\"},{\"mycompany\":\"mongodb44-rs-1.gke.prod.mycompany.fr:33242\"},{\"mycompany\":\"mongodb44-rs-2.gke.prod.mycompany.fr:33243\"}],\"security\":{\"authentication\":{\"modes\":[\"SCRAM-SHA-1\",\"SCRAM\"],\"ignoreUnknownUsers\":true},\"tls\":{\"enabled\":true,\"optional\":true,\"certificateKeySecretRef\":{\"name\":\"tls-certificate\"},\"caCertificateSecretRef\":{\"name\":\"tls-ca-key-pair\"}}},\"users\":[{\"name\":\"mongodb44-admin-mongo\",\"db\":\"admin\",\"passwordSecretRef\":{\"name\":\"mongodb44-admin-mongo-password\",\"key\":\"\"},\"roles\":[{\"db\":\"admin\",\"name\":\"clusterAdmin\"},{\"db\":\"admin\",\"name\":\"root\"},{\"db\":\"admin\",\"name\":\"userAdminAnyDatabase\"},{\"db\":\"admin\",\"name\":\"readWriteAnyDatabase\"},{\"db\":\"admin\",\"name\":\"dbAdminAnyDatabase\"}],\"scramCredentialsSecretName\":\"mongodb44-admin-mongo\",\"additionalConnectionStringConfig\":null}],\"statefulSet\":{\"spec\":{},\"metadata\":{}},\"agent\":{\"logLevel\":\"\",\"logFile\":\"\",\"maxLogFileDurationHours\":0},\"additionalMongodConfig\":{},\"prometheus\":{\"username\":\"prometheus\",\"passwordSecretRef\":{\"name\":\"prometheus-password\",\"key\":\"\"},\"tlsSecretKeyRef\":{\"name\":\"\",\"key\":\"\"}},\"additionalConnectionStringConfig\":{}}").
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.Here is my TF code :
resource "kubernetes_manifest" "mongo_mongodbcommunity" {
manifest = {
apiVersion = "mongodbcommunity.mongodb.com/v1"
kind = "MongoDBCommunity"
metadata = {
name = local.name
namespace = local.namespace
}
spec = {
additionalMongodConfig = {
"storage.wiredTiger.engineConfig.journalCompressor" = "zlib"
}
members = 3
version = local.version
replicaSetHorizons = [
for i in local.rs :
{ gsoi = "${i.dns_name}:${i.port}" }
]
security = {
authentication = {
ignoreUnknownUsers = true
modes = [
"SCRAM-SHA-1",
"SCRAM",
]
}
tls = {
caCertificateSecretRef = {
name = local.secret_ca_name
}
certificateKeySecretRef = {
name = local.secret_cert_name
}
enabled = true
optional = true
}
}
prometheus = {
username = kubernetes_secret.prometheus_password.data.username
passwordSecretRef = {
name = kubernetes_secret.prometheus_password.metadata.0.name
}
}
type = "ReplicaSet"
statefulSet = {
spec = {
template = {
spec = {
nodeSelector = {
app = local.namespace
}
serviceAccountName = "${local.name}"
tolerations = [
{
effect = "NoSchedule"
key = local.namespace
operator = "Exists"
},
]
containers = [
{
name = "mongod"
resources = {
limits = {
cpu = var.cpu
memory = var.memory
}
}
}
]
affinity = {
podAntiAffinity = {
requiredDuringSchedulingIgnoredDuringExecution = [
{
labelSelector = {
matchExpressions = [
{
key = "app",
operator = "In",
values = [
"${local.name}-svc"
]
}
]
}
topologyKey = "kubernetes.io/hostname"
}
]
}
}
}
}
volumeClaimTemplates = [
{
metadata = {
name = "data-volume"
}
spec = {
storageClassName = local.storage_class
resources = {
requests = {
storage = "${local.disk_size}Gi"
}
}
}
},
]
}
}
# https://www.mongodb.com/docs/manual/reference/built-in-roles/#superuser-roles
users = concat(
[{
db = "admin"
name = "${local.name}-admin-mongo"
passwordSecretRef = {
name = "${local.name}-admin-mongo-password"
}
roles = [
{
db = "admin"
name = "clusterAdmin"
},
{
db = "admin"
name = "root"
},
{
db = "admin"
name = "userAdminAnyDatabase"
},
{
db = "admin"
name = "readWriteAnyDatabase"
},
{
db = "admin"
name = "dbAdminAnyDatabase"
},
]
scramCredentialsSecretName = "${local.name}-admin-mongo"
}],
[for key, value in var.users :
{
db = "admin"
name = key
passwordSecretRef = {
name = "${key}-password"
}
roles = [
{
db = value.db
name = value.role
},
]
scramCredentialsSecretName = key
}],
)
}
}
wait {
fields = {
"status.phase" = "Running",
}
}
# https://github.com/hashicorp/terraform-provider-kubernetes/issues/2367#issuecomment-1880571498
computed_fields = ["metadata.labels", "metadata.annotations", "object"]
depends_on = [
kubernetes_secret.admin_password,
kubernetes_secret.users_password,
]
}
kizzie
commented
9 months ago Oh I just got something similar to this as well - so its still an ongoing bug?
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to module.this.kubernetes_manifest.repos, provider "provider[\"registry.terraform.io/hashicorp/kubernetes\"]" produced an unexpected new value:
│ .object.stringData: was cty.MapVal(map[string]cty.Value{"sshPrivateKey":cty.StringVal("some_ssh_key_here\n"), "type":cty.StringVal("git"),
│ "url":cty.StringVal("git@github.com:owner/repo.git")}), but now null.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.I was trying to get it to make a secret for me from something picked up from secrets manager
apiVersion: v1
kind: Secret
metadata:
name: argocd
namespace: argo-cd-${environment}
labels:
# this tells it to put it in as a repo in the argo UI
argocd.argoproj.io/secret-type: repository
stringData:
type: git
url: git@github.com:owner/repo.git
sshPrivateKey: |
${sshPrivateKey}and
data "aws_secretsmanager_secret" "ssh_key" {
name = "..."
}
data "aws_secretsmanager_secret_version" "ssh_key" {
secret_id = data.aws_secretsmanager_secret.ssh_key.id
}
resource "kubernetes_manifest" "repos" {
manifest = yamldecode(templatefile("${path.module}/k8s/repositories.yaml", {
environment = var.environment,
sshPrivateKey = data.aws_secretsmanager_secret_version.ssh_key.secret_string
}))
}Edit: Well its not the secrets manager which is causing it - changing that to just foo instead of the data resource has the same effect.
kizzie
commented
9 months ago So, more debugging and tinkering as adding the objects to the computed value also didn't work...
Making it a plain secret without the argo labels or anything and it still failed. I just did a basic foo=bar type secret and it still failed. So its not something about odd characters that I can see. If I make a configmap then it works just fine. Everyone is happy (in fact this same bit of code is creating some istio resources, so I know the provider does work!)
it's odd that we're both having issues with argo cd related things with this. Makes me think its something in the helm chart or crd stuff as well...
Changing it to a different namespace produced the same result
apiVersion: v1
kind: Secret
metadata:
name: simple-secret
namespace: default
stringData:
foo: barProduced
module.this.kubernetes_manifest.repos: Creating...
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to module.this.kubernetes_manifest.repos, provider "provider[\"registry.terraform.io/hashicorp/kubernetes\"]" produced an unexpected new value:
│ .object.stringData: was cty.MapVal(map[string]cty.Value{"foo":cty.StringVal("bar")}), but now null.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
$ kubectl get secrets simple-secret -n default --output json | jq -r .data.foo | base64 -d
barOh. Oh I think I found my problem. Not sure if it helps the other people though but I think this is doing something. Data is handled, stringData is not handled as well.
When it reads the value back after the apply, it puts it into data, not stringData.
There's something about the stringData un-marshalling is not working when its confirming what is going on. Which definitely is a bug.
I can put:
apiVersion: v1
kind: Secret
metadata:
name: simple-secret
namespace: default
data:
foo: Zm9vCg==and it works just fine. So if I base64 encrypt all my values (and get to play with the linter trying to protect me from unsafe secrets XD) then it will work in theory.
alekhrycaiko
commented
7 months ago I'm also experiencing the same issues mentioned where we're hitting a number of similar errors like provider "provider[\"registry.terraform.io/hashicorp/kubernetes\"]" produced an unexpected new value: .object.operation.initiatedBy.automated: was null, but now cty.True.
Is there any recommended approach here for resolution? There hasn't been much movement on this issue which is concerning as this is a quite a breaking failure.
From what I can tell in issues it seems like ArgoCD Application manifests are hitting this bug frequently due to changes in the operation block that aren't consolidating here correctly.
Similar to others, adding object fields to the computed_fields list did not solve the issue.
vladborovtsov
commented
7 months ago @alekhrycaiko you can encode with base64 as suggested by @kizzie
Ive committed some time looking for other, more elegant solution, but didn't find anything better so far
alekhrycaiko
commented
6 months ago @alekhrycaiko you can encode with base64 as suggested by @kizzie
Ive committed some time looking for other, more elegant solution, but didn't find anything better so far
Unfortunately that won't work for us. We don't reference or manage Kubernetes secrets due to our use of the AWS secrets csi driver. But thanks for the suggestion 🙏
Any alternative ideas are very welcome!
It'd also be great to see this issue prioritized by the maintainers as it's extremely breaking and the suggested work around isn't viable for all users.
alexsomesan
commented
6 months ago @mailbox171 I've tried in a few ways to reproduce your issue and I'm unable to. The 'Application' object applies correctly and never shows any modifications.
Can you share a bit more details as to what might be changing it on the cluster side? Is Argo itself touching this object in any way?
alexsomesan
commented
6 months ago @kizzie Please don't use stringData with the kubernetes_manifest resource. There are actually two issues happening here:
stringData is a write-only attribute in the K8s API, which doesn't bode well with Terraform as it expects a value to be returned for it once it gets set.We recommended to use the data attribute in conjunction with the base64econde() function built into Terraform to populate secrets.
alexsomesan
commented
6 months ago @alekhrycaiko can you please share some configuration that we can use to reproduce your case. Also please see my comment above about using stringData in secrets.
alekhrycaiko
commented
6 months ago @alexsomesan my configuration usage is similar to what's mentioned earlier here https://github.com/hashicorp/terraform-provider-kubernetes/issues/2367#issuecomment-1860860619
On return, the objects field has the typings change which is what's being reported. I've tried a range of computed fields to resolve this issue but nothing has worked. From a user perspective this feels like a bug in how computed_fields functions. In the case of the ArgoCD application, the operation field is not declared in Terraform and is being dynamically updated by Argo itself upon sync.
I've tried a few more operations with computed_fields such as below but none of them worked.
computed_fields = [
"operation.sync.revision",
"operation.sync.prune",
"operation.initiatedBy.automated",
"operation.retry.limit",
"operation.sync.syncOptions",
"metadata.labels",
"metadata.annotations",
"metadata.generate_name",
"spec.ignoreDifferences"
]
corylevine
commented
6 months ago Hey @alexsomesan, I work with @alekhrycaiko. I was doing some digging into the provider source and noticed the RemoveServerSideFields function. I added the removal of the operation field to it and that actually seemed to fix the problem.
What I believe is happening is:
operation field, as there are currently no operations running on the Argo app, so they all get nulled out.We haven't experienced this issue until we started using the Secrets Store CSI Driver and added a SecretProviderClass to this Argo app's helm chart. As our secrets are sourced from AWS Secrets Manager, these checks against it seem to cause Argo to take slightly longer to perform its sync operation, which causes the operation field to have a different value during apply time.
Without the fix, we can durably reproduce this issue by adding or removing a secret to the SecretProviderClass, causing it to reconcile on apply, leading to Argo's reporting of an ongoing operation.
I hope this makes sense.
EDIT: I dropped a PR with the change in question.
alexsomesan
commented
6 months ago I'm trying to come up with a configuration that reproduces the issue, but so far haven't been successful.
If anyone is able to share a snippet that reproduces consistently, it would really help us move towards a solution and would be much appreciated.
corylevine
commented
6 months ago I'm trying to come up with a configuration that reproduces the issue, but so far haven't been successful.
If anyone is able to share a snippet that reproduces consistently, it would really help us move towards a solution and would be much appreciated.
Ours was nothing out of the ordinary, albeit pretty massive. We did not run into this issue until we added the SecretProviderClass with a large amount of secrets being pulled from AWS Secrets Manager. It would be rather difficult to generate a version of it that we would post here that would actually work, but I'd be happy to hop on a call and show you our setup and reproduce the issue.
jlenuffgsoi
commented
6 months ago I'm trying to come up with a configuration that reproduces the issue, but so far haven't been successful.
If anyone is able to share a snippet that reproduces consistently, it would really help us move towards a solution and would be much appreciated.
Can you have a look to my code ? => https://github.com/hashicorp/terraform-provider-kubernetes/issues/2367#issuecomment-1966051491
Because of this, I had to use kustomize instead.
Terraform Version, Provider Version and Kubernetes Version
Affected Resource(s)
argocd Application resource
Terraform Configuration Files
Debug Output
Panic Output
Steps to Reproduce
we run PLAN and then APPLY, this is out main.tf
Expected Behavior
What should have happened?
We did not change the resource manifest. APPLY should have returned 0
Actual Behavior
PLAN
APPLY
The apply seems to have being done. But the output is the following
Important Factoids
References
Community Note