Difficult to automatically provision the plugin (via TF) #116

Open ambis opened 1 month ago

I created a Terraform module for this plugin.

It is very difficult to fully configure this via TF, since the token must always be provided:

1) You cannot add roles until secret engine is fully configured (gives an error that backend is not configured when trying to) 2) You cannot lifecycle ignore_changes the token, since it is within a json field

I could read the PAT from our other secret store, but eventyally after a year that initial token would expire (gitlab max token ttl), and it would then also replace the token that the plugin itself has rotated (right?).

I suggest that you could post an empty token field value, in case the plugin already has configured/rotated plugin in use.

Here is my simple module (below is the usage part):

variable "path" {
  description = "Path where to mount"
  type = string
}

variable "gitlab_base_url" {
  description = "GitLab base URL, eg. https://gitlab.com"
  type = string
}

variable "roles" {
  type = map(object({
    token_type = string
    path = string
    scopes = list(string)
    access_level = optional(string, "guest")
    gitlab_revokes_token = optional(bool, false)
    ttl = optional(string, "1h")
  }))
}

resource "vault_mount" "mount" {
  path        = var.path
  type        = "gitlab"
  description = "Secret backend which generates Personal Access Tokens in GitLab."
}

resource "vault_generic_endpoint" "mount_config" {
  path      = "${vault_mount.mount.path}/config"
  disable_delete = true # Will delete if engine is unmounted

  write_fields = [
    "base_url",
    "auto_rotate_token",
    "auto_rotate_before",
  ]

  data_json = jsonencode({

    token = "glpat-XXX" # <<<----- HERE lies the problem, this always requires a valid value to be provided

    base_url = var.gitlab_base_url
    auto_rotate_token = true
    auto_rotate_before = "48h"
  })

  depends_on = [
    vault_mount.mount
  ]
}

resource "vault_generic_endpoint" "project_roles" {
  for_each = var.roles

  path = "${vault_mount.mount.path}/roles/${each.value.token_type}--${replace(each.value.path, "/", "-")}--${each.key}"

  write_fields = [
    "access_level",
    "gitlab_revokes_token",
    "name",
    "path",
    "role_name",
    "scopes",
    "token_type",
    "ttl",
  ]

  data_json = jsonencode({
    name = each.key
    path = each.value.path
    scopes = each.value.scopes
    access_level = each.value.access_level
    token_type = each.value.token_type
    gitlab_revokes_token = each.value.gitlab_revokes_token
    ttl = each.value.ttl
  })

  depends_on = [
    vault_generic_endpoint.mount_config
  ]
}

And when using it:

module "gitlab_token" {
  source = "./modules/secret_gitlab_token"

  path = "gitlab"
  gitlab_base_url = "https://gitlab.com"

  roles = {
    test-role = {
      token_type = "project"
      path = "my/project"
      scopes = [
        "read_registry"
      ]
      access_level = "guest"
      gitlab_revokes_token = false
    }
  }
}

The reason it requires a valid access token is, so it can check that the token is valid when you initially configure the config endpoint.

You can just ignore the whole data_json. Won't that work?

I'm 100% fine if the first apply requires the token. This I can give via ENV when running the module initial apply.

If I ignore data_json, that would prevent me (or anyone else) from changeing the other options. I don't like it, because then there would be a need to have big "NOTE! Changeing these values won't actually change anything!" Like auto_rotate_before.

The best solution would be to require the token key present at initial install/config, but then one could gitlab/config without providing the token entirely, if the module already has a valid token.

I can check if patch will work on the endpoint, but then I don't know if vault_generic_endpoint supports patch operations. Checked their documentation and PATCH is not supported in their terraform provider. https://registry.terraform.io/providers/hashicorp/vault/latest/docs/resources/generic_endpoint#path

Initially it wasn't required, it was on a periodic check, but that just made it more complicated. And I had some issues with the periodic not running when you need it.

The token is also required at the beginning to check when it expires as well, so it can be rotated, and also retrieve the configured scopes.

I'll check if patch is supported, but then you have to figure out how to apply them with terraform or a manual vault patch

Example:

vault patch gitlab/config type=saas

The best solution would be to require the token key present at initial install/config, but then one could gitlab/config without providing the token entirely, if the module already has a valid token.

I don't agree with this, for one I want to be able to change the token when ever I want. There are cases when autorotation is disabled, so I want to be able to change the token whenever I want. If I start adding if conditions, how can I differentiate between what is a valid reason and what is not.

118

With this, you should be able to patch all the properties separately as needed.

Can you give it a try @ambis

Released under v0.5.0

I will! Monday morning at the latest! Thank you!

Thanks for the updates. I updated the plugin to v0.6.0.

Unfortunately the problem persists. I initialized the plugin with a valid token, and everything was applied and started all proper. Then, another apply without token field (also tested with empty token value), and I got this:

module.gitlab_token.vault_generic_endpoint.mount_config: Modifying... [id=gitlab/config/default]
╷
│ Error: error writing to Vault: Error making API request.
│
│ URL: PUT https://my-vault/v1/gitlab/config/default
│ Code: 500. Errors:
│
│ * 1 error occurred:
│   * 1 error occurred:
│   * token: required field

I realized I should do a PATCH to the gitlab/config/default endpoint and I'm now trying to figure out how to accomplish this with vault_generic_endpoint (currently does not look to be possible).

You could try with https://developer.hashicorp.com/terraform/language/resources/provisioners/local-exec

Either with the vault command or curl

I have now tried to use this with local-exec.

resource "terraform_data" "mount_config" {
  for_each = local.config

  provisioner "local-exec" {
    command = "vault patch gitlab/config/default ${each.key}=\"$${VALUE}\""

    environment = {
      VALUE = each.value # Hide token value from any output
    }
  }

  depends_on = [
    vault_mount.mount
  ]
}

While this does run the PATCH requests, since terraform does not know what is actually happening here, there is no state management what so ever.

If I first pass a token to it, and then not in the following apply, it'll try to remove the exec from state

  # module.gitlab_token.terraform_data.mount_config["token"] will be destroyed
  # (because key ["token"] is not in for_each map)
  - resource "terraform_data" "mount_config" {
      - id = "7c4ad16e-ca54-210e-95b6-ba556fb4c086" -> null
    }

If only one could just POST any number of config keys to gitlab/config/default once it has been initially setup satisfactorily with a valid token.

Looking at the plugins that supply credentials in the official internal ones https://github.com/hashicorp/vault/tree/main/builtin/logical

None of them allow you to do what you want, partially allow you to submit the data on the creation of the config.

Let me check some stuff and see if there is a simpler way to do this with terraform.

This seems to work for me. If you change any of the other values, this will change them. Of course, the GitLab token is excluded, as it's managed by Vault.

If you don't want to manage the token with Vault, then gitlab_auto_rotate_token should be set to false, and then you always pass the value and terraform will make sure that it update the values as needed.

variable "gitlab_base_url" {
  description = "GitLab base URL, eg. https://gitlab.com"
  type        = string
}

variable "gitlab_token" {
  description = "GitLab Token"
  type        = string
  sensitive   = true
}

variable "gitlab_type" {
  description = "GitLab Type can be saas, self-managed or dedicated"
  type        = string
  default     = "self-managed"
}

variable "gitlab_auto_rotate_token" {
  type    = bool
  default = true
}

variable "gitlab_auto_rotate_before" {
  type    = string
  default = "48h"
}

locals {
  vault_config_default_data = {
    token              = var.gitlab_token
    base_url           = var.gitlab_base_url
    auto_rotate_token  = var.gitlab_auto_rotate_token
    auto_rotate_before = var.gitlab_auto_rotate_before
    type               = var.gitlab_type
  }

  vault_config_default_patch_data = {
    for k, v in local.vault_config_default_data : k => v if k != "token"
  }
}

resource "vault_generic_endpoint" "mount_default_config" {
  path                 = "gitlab/config/default"
  disable_delete       = true
  ignore_absent_fields = true

  write_fields = [
    "base_url",
    "auto_rotate_token",
    "auto_rotate_before",
    "type",
    "scopes",
  ]

  data_json = jsonencode(local.vault_config_default_data)

  lifecycle {
    ignore_changes = [
      data_json
    ]
  }
}

resource "null_resource" "mount_default_config_patch" {
  for_each = local.vault_config_default_patch_data
  triggers = { (each.key) = each.value }

  provisioner "local-exec" {
    command     = <<EOT
      vault patch gitlab/config/default ${each.key}=${each.value} >/dev/null
    EOT
    interpreter = ["bash", "-c"]
  }

  depends_on = [
    vault_generic_endpoint.mount_default_config,
  ]
}

ilijamt / vault-plugin-secrets-gitlab

Difficult to automatically provision the plugin (via TF) #116

118