Use kubernetes provider to create k8s secrets from tf outputs

[brianrexrode]

As a consumer of IaC, I'd like the ability to create k8s secrets from terraform outputs (specifically bigbang-dependencies) in my eks cluster. I would later leverage these k8s secrets to set variables for my zarf deployment of Bigbang.

For example, create a namespace called iac and create a k8s secret called keycloak-connect-info that contains the RDS info

host        = module.rds_postgres_keycloak[0].db_instance_address
database    = module.rds_postgres_keycloak[0].db_instance_name
username    = module.rds_postgres_keycloak[0].db_instance_username
password    = var.keycloak_db_password

as well as a secret for loki with values from

  name            = var.loki_s3_bucket_name
  region          = var.region
  secret_string   = jsonencode({
      bucket      = module.loki_s3_bucket.s3_bucket

AC:

• [ ] add an example which shows how you could do this

[zack-is-cool]

Have we considered using SSM parameter store as an alternative? It may be easier to maintain and update secrets.

[brianrexrode]

I was testing using AWS Secrets Manager to store the secrets but that doesn't work for on-prem (non-AWS) deployments. I think we are looking for a portable solution and storing secrets in-cluster made the most sense...so far. Keeping in mind the need is to support zarf deployments of Bigbang.

[RothAndrew]

How are we thinking we want to do this? A new module? Do we think there's enough there to warrant maintaining one? It's possible that the solution here could be just 2 terraform resources -- A kubernetes_namespace and one or more kubernetes_secret resources that could be added to the root module.

[RothAndrew]

Do we want this to happen in the IaC layer, as the cluster is being stood up? Or later in the "platform layer" as part of DUBBD?

[RothAndrew]

Is having a secret in a different/new namespace able to meet the need? I don't believe I've ever done that before. The secret has always been in the same namespace as the stuff that needs to use it.

[brianrexrode]

I was thinking of just creating another module in the bigbang-dependencies.tf since that gets run after the eks module. Or a "secrets" module under modules directory that gets referenced from the bigbang-dependencies.tf

If we try to do this in the platform layer as part of DUBBD I'd be concerned about how zarf would access the existing tf-state in order to retrieve to tf outputs to set as ZARF_VARs.

The idea was that by creating a new namespace (prior to BB deployment) and creating the "known" secrets in that namespace, then with zarf we could run some kubectl commands to retrieve the values from the k8s secrets and then set them as ZARF_VARs for the BB deployment.

[ntwkninja]

plan to add an example of how you may want to do this before deciding to add a supported module.