Implement drift detection and correction for cluster state

stefanprodan commented 1 month ago

This PR implements Flux readiness checks and drift detection for the cluster state. The provider applies changes to the Flux components and GitRepository/Kustomization manifests on the cluster, thus enabling changes to Git URL and branch to be actuated. It also adds a check for verifying the kubeconfig during planning phase.

Description

Changes:

Detect is Flux is running in the cluster and if it is ready during planning phase.
Restore Flux on a cluster if readiness checks fails.
Apply changes in configuration directly on the cluster on updates.
Disallow changes to the bootstrap path field in the same way the CLI does it (breaking change).

Motivation and Context

Being able to update Flux in the cluster by detecting drift in the cluster state during planning and apply.

Fix: #500 Fix: #656 Fix: #653 Fix: #564 Fix: #176 Fix: #499

How has this been tested?

[x] Have you added an acceptance test for the functionality being added?
[x] Have you run the acceptance tests on this branch?

Manual testing for:

change Git branch
change Git URL
cluster will Flux controllers deleted
cluster with Flux GitRep object delete
recreated cluster without Flux outside of TF

Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[x] Breaking change (fix or feature that would cause existing functionality to change)

Documentation

[x] I have updated the documentation (if required) with make docs

Checklist:

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I've read the CONTRIBUTION guide
[x] I have signed-off my commits with git commit -s

stefanprodan commented 1 month ago

@swade1987 it would be great if you could test this PR on your clusters, especially the kubeconfig validation during planning.

swade1987 commented 1 month ago

@stefanprodan I'll take this for a spin next week and keep you posted.

swade1987 commented 4 weeks ago

@swade1987 it would be great if you could test this PR on your clusters, especially the kubeconfig validation during planning.

Looking good @stefanprodan

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: Get "https://xxxx/api/v1/namespaces/flux-system": dial tcp: lookup XXXX: no such host
│
│   with kubernetes_namespace.flux_system,
│   on main.tf line 67, in resource "kubernetes_namespace" "flux_system":
│   67: resource "kubernetes_namespace" "flux_system" {
│
╵

swade1987 commented 4 weeks ago

Changing the git branch works as well (as long as the branch exists in GitHub) (see below)

provider "flux" {
  kubernetes = {
    host                   = kind_cluster.this.endpoint
    client_certificate     = kind_cluster.this.client_certificate
    client_key             = kind_cluster.this.client_key
    cluster_ca_certificate = kind_cluster.this.cluster_ca_certificate
  }
  git = {
    url = "ssh://git@github.com/${var.github_org}/${var.github_repository}.git"
    branch = "test-branch"
    ssh = {
      username    = "git"
      private_key = tls_private_key.flux.private_key_pem
    }
  }
}

Ran terraform apply then ...

flux export source git flux-system | grep branch
    branch: test-branch

swade1987 commented 4 weeks ago

LGTM @stefanprodan, I ran a number of tests locally and things look good.

fluxcd / terraform-provider-flux