flux_bootstrap_git parameters when using GitLab/GitHub #368

Closed cdenneen closed 1 year ago

Currently there are parameters for the flux_bootstrap_git resource that aren't documented of where to apply them when following GitLab/GitHub examples.

Also additional boostrap parameters that seem to be missing:

--read-write-key --private

Where do you apply those in the GitLab/GitHub examples along with other parameters like --author-name, --author-email?

I found --components-extra can be added to:

# Flux
data "flux_install" "main" {
  target_path      = var.target_path
  components_extra = var.components_extra
}

but thought that if the bootstrap is to follow the CLI bootstrap the components should be parameter there and pass through to flux_install.

the --read-write-key is handled by:

resource "gitlab_deploy_key" "main" {
  title    = var.cluster_name
  project  = gitlab_project.main.id
  key      = tls_private_key.main.public_key_openssh
  can_push = true # equivalent of --read-write-key

  depends_on = [gitlab_project.main]
}

I'm using the same module to create multiple EKS clusters

resource "gitlab_project" "main" {
  name                   = var.repository_name
  namespace_id           = data.gitlab_group.owner.id
  visibility_level       = var.repository_visibility
  initialize_with_readme = true
  default_branch         = var.branch
}

Flux is set to use clustername as part of path:

# Flux
data "flux_install" "main" {
  target_path      = "clusters/${var.cluster_name}"
  components_extra = var.components_extra
  toleration_keys  = ["CriticalAddonsOnly"]
}

data "flux_sync" "main" {
  target_path = "clusters/${var.cluster_name}"
  url         = "ssh://git@gitlab.domain.com/${var.gitlab_project_path}/${var.repository_name}.git"
  branch      = var.branch
}

Issue is the repo is already created and causes:

╷
│ Error: POST https://gitlab.domain.com/api/v4/projects: 400 {message: {path: [has already been taken]}}
│
│   with gitlab_project.main,
│   on flux.tf line 128, in resource "gitlab_project" "main":
│  128: resource "gitlab_project" "main" {
│
╵

so therefore the gitlab_project needs to be converted to a data source and means the project needs to be pre-created (unless someone can figure out best way to check data source, if doesn't exist then create resource otherwise used the data source). (This also required the depends_on for gitlab_project to be removed.

Hoping this helps someone because the example included is flawed because anyone using this would theoretically be using it for more than 1 cluster so therefore couldn't use the project resource.

the gitlab example ends up leaving resources that want to update every TF run. Really need to get gitlab working with the bootstrap command rather than the kubectl_manifest current method:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following
symbols:
  ~ update in-place

Terraform will perform the following actions:

  # kubectl_manifest.install["apps/v1/deployment/flux-system/helm-controller"] will be updated in-place
  ~ resource "kubectl_manifest" "install" {
        id                      = "/apis/apps/v1/namespaces/flux-system/deployments/helm-controller"
        name                    = "helm-controller"
      ~ yaml_incluster          = (sensitive value)
        # (14 unchanged attributes hidden)
    }

  # kubectl_manifest.install["apps/v1/deployment/flux-system/kustomize-controller"] will be updated in-place
  ~ resource "kubectl_manifest" "install" {
        id                      = "/apis/apps/v1/namespaces/flux-system/deployments/kustomize-controller"
        name                    = "kustomize-controller"
      ~ yaml_incluster          = (sensitive value)
        # (14 unchanged attributes hidden)
    }

  # kubectl_manifest.install["apps/v1/deployment/flux-system/source-controller"] will be updated in-place
  ~ resource "kubectl_manifest" "install" {
        id                      = "/apis/apps/v1/namespaces/flux-system/deployments/source-controller"
        name                    = "source-controller"
      ~ yaml_incluster          = (sensitive value)
        # (14 unchanged attributes hidden)
    }

Plan: 0 to add, 3 to change, 0 to destroy.

Hey @cdenneen sorry for not answering this issue earlier.

Could you try out the new resource instead? It may solve all of the issues you are currently seeing. We are currently working on promoting people to use the new resource. https://registry.terraform.io/providers/fluxcd/flux/latest/docs/resources/bootstrap_git

Yeah I was asking for the gitlab example to be updated to this resource.

On Wed, Mar 1, 2023 at 2:41 PM Philip Laine @.***> wrote:

Hey @cdenneen https://github.com/cdenneen sorry for not answering this issue earlier.

Could you try out the new resource instead? It may solve all of the issues you are currently seeing. We are currently working on promoting people to use the new resource.

https://registry.terraform.io/providers/fluxcd/flux/latest/docs/resources/bootstrap_git

— Reply to this email directly, view it on GitHub https://github.com/fluxcd/terraform-provider-flux/issues/368#issuecomment-1450748272, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFPZYITNOSYUDWYHYLLB6DWZ6Q6ZANCNFSM6AAAAAAU746GCI . You are receiving this because you were mentioned.Message ID: @.***>

I have not had time to rewrite the guides yet. It is in my todo list to get this done for GitHub and GitLab, but I do not know exactly when I will get it done.

@phillebaba one thing I'm running into issue is with the tolerations wanting to be lowercase but in my case that keeps the deployment Pending:

╷
│ Error: Invalid Attribute Value Match
│
│   with flux_bootstrap_git.this,
│   on flux.tf line 27, in resource "flux_bootstrap_git" "this":
│   27:   toleration_keys        = ["CriticalAddonsOnly"]
│
│ Attribute toleration_keys[Value("CriticalAddonsOnly")] a lowercase RFC 1123 label must consist of lower case alphanumeric characters
│ or '-', and must start and end with an alphanumeric character, got: CriticalAddonsOnly

Example GitLab:

provider "flux" {
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
  token                  = data.aws_eks_cluster_auth.default.token
}

provider "kubectl" {
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
  token                  = data.aws_eks_cluster_auth.default.token
  load_config_file       = false
}

provider "gitlab" {
  base_url = "https://gitlab.example.com/api/v4/"
  token    = var.gitlab_token
}

resource "flux_bootstrap_git" "this" {
  url                    = "ssh://git@gitlab.example.com/${var.gitlab_project_path}/${var.repository_name}.git"
  author_email           = "flux@gitlab.example.com"
  author_name            = "Flux"
  path                   = "clusters/${var.cluster_name}"
  branch                 = "main"
  namespace              = "flux-system"
  components_extra       = var.components_extra
  toleration_keys        = ["CriticalAddonsOnly"]
  ssh                    = {
    username    = "git"
    private_key = tls_private_key.main.private_key_pem
  }
  kustomization_override = file("${path.module}/kustomization-override.yaml")

  depends_on             = [kubernetes_namespace.flux_system]
}

resource "tls_private_key" "main" {
  algorithm   = "ECDSA"
  ecdsa_curve = "P256"
}

resource "kubernetes_namespace" "flux_system" {
  metadata {
    name = "flux-system"
  }

  lifecycle {
    ignore_changes = [
      metadata[0].labels,
    ]
  }
}

data "gitlab_project" "main" {
  id = "${var.gitlab_project_path}/${var.repository_name}"
}

resource "gitlab_deploy_key" "main" {
  title    = var.cluster_name
  project  = data.gitlab_project.main.id
  key      = tls_private_key.main.public_key_openssh
  can_push = true # <-- this is equivalent of the --read-write-key parameter on the command line.
}

kustomization-override.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
labels:
  - pairs:
      toolkit.fluxcd.io/tenant: sre-team
patches:
  - patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --concurrent=20
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --requeue-dependency=5s
    target:
      kind: Deployment
      name: "(kustomize-controller|helm-controller|source-controller)"

This looks good, however I do not think that you need to create the namespace separately. That was more of an issue with the old data source. Otherwise this all looks good.

If you check the PR #390 you'll see I tested against your newest reconfigure client so then the example would need to be modified accordingly. Once I get the git client configured properly with your new setup and the toleration issue fixed I can work on removing the namespace creation.

provider "flux" {
  git = {
    url = "ssh://git@gitlab.example.com/${var.gitlab_project_path}/${var.repository_name}.git"
    ssh = {
      username    = "git"
      private_key = tls_private_key.main.private_key_pem
    }
  }
}

provider "kubectl" {
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
  token                  = data.aws_eks_cluster_auth.default.token
  load_config_file       = false
}

provider "gitlab" {
  base_url = "https://gitlab.example.com/api/v4/"
  token    = var.gitlab_token
}

resource "flux_bootstrap_git" "this" {
  author_email           = "flux@gitlab.example.com"
  author_name            = "Flux"
  path                   = "clusters/${var.cluster_name}"
  branch                 = "main"
  namespace              = "flux-system"
  components_extra       = var.components_extra
  toleration_keys        = ["CriticalAddonsOnly"]
  kustomization_override = file("${path.module}/kustomization-override.yaml")

  depends_on             = [kubernetes_namespace.flux_system]
}

resource "tls_private_key" "main" {
  algorithm   = "ECDSA"
  ecdsa_curve = "P256"
}

data "gitlab_project" "main" {
  id = "${var.gitlab_project_path}/${var.repository_name}"
}

resource "gitlab_deploy_key" "main" {
  title    = var.cluster_name
  project  = data.gitlab_project.main.id
  key      = tls_private_key.main.public_key_openssh
  can_push = true # <-- this is equivalent of the --read-write-key parameter on the command line.
}

fluxcd / terraform-provider-flux

flux_bootstrap_git parameters when using GitLab/GitHub #368