Support for Flux v2 AKS cluster extension

[au2001]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

AKS managed clusters have a preview feature to install Flux v2. This is done with the Microsoft.KubernetesConfiguration/extensions/microsoft.flux cluster extension which is automatically installed when creating a Microsoft.KubernetesConfiguration/fluxConfigurations resource. The latter is the one which would be nice to be able to control with this Terraform provider.

New or Affected Resource(s)

• New Resource azurerm_kubernetes_configuration_flux

Potential Terraform Configuration

resource "azurerm_kubernetes_configuration_flux" "flux_install" {
  name                  = "gitops-demo"
  cluster_name          = "flux-demo-aks"
  cluster_resource_name = "managedClusters"
  cluster_rp            = "Microsoft.ContainerService"
  resource_group_name   = "flux-demo-rg"
  subscription_id       = "00000000-0000-0000-0000-000000000000"

  configuration_protected_settings = {}

  git_repository = {
    https_ca_file  = "./certs/github.pem"
    https_user     = "git"
    local_auth_ref = "flux-system"

    repository_ref = {
      branch = "main"
      # or
      commit = "abcd012"
      # or
      semver = ">=1.0.0 <2.0.0"
      # or
      tag    = "v1.0.0"
    }

    ssh_known_hosts = base64encode(join("\n", [
      "github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=",
      "gitlab.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY="
    ]))

    sync_interval_in_seconds = 600
    timeout_in_seconds       = 120
    url                      = "https://github.com/fluxcd/flux2-kustomize-helm-example"
  }

  kustomizations = {
    apps = {
      depends_on = [
        {
          kustomization_name = "infra"
        }
      ]

      force                     = false
      path                      = "./apps/staging"
      prune                     = true
      retry_interval_in_seconds = null
      sync_interval_in_seconds  = 600
      timeout_in_seconds        = 120
      validation                = "server"
    }

    infra = {
      depends_on                = []
      force                     = false
      path                      = "./infrastructure"
      prune                     = true
      retry_interval_in_seconds = null
      sync_interval_in_seconds  = 600
      timeout_in_seconds        = 120
      validation                = "client"
    }
  }

  namespace   = "flux-system"
  scope       = "cluster"
  source_kind = "GitRepository"
  suspend     = false
}

References

• https://docs.microsoft.com/en-us/cli/azure/k8s-configuration/flux?view=azure-cli-latest
• https://docs.microsoft.com/en-us/azure/azure-arc/kubernetes/conceptual-gitops-flux2
• https://docs.microsoft.com/en-us/azure/azure-arc/kubernetes/tutorial-use-gitops-flux2
• https://docs.microsoft.com/en-us/rest/api/kubernetesconfiguration/2021-11-01-preview/flux-configurations
• https://pkg.go.dev/github.com/Azure/azure-sdk-for-go@v61.5.0+incompatible/services/preview/kubernetesconfiguration/mgmt/2020-07-01-preview/kubernetesconfiguration

[avbanks]

Is there any update on this? I see it's possible with Azure Bicep and Pulumi it would be a tremendous help to have this available for Terraform. It would help facilitate the bootstrapping of clusters with applications.

[vainikkaj]

GitOps AKS extension is now GA (announcement)

[mustafazeya]

any update on this?

[IvanJosipovic]

Also looking for this!

[techdecline]

I got tired of waiting for support within the resource and decided to go with a local_exec provisioner for the time being. I'll share my solution, if anyone is interested.

provisioner "local-exec" {
    interpreter = ["pwsh", "-Command"]
    command     = <<-EOT
    . az login --service-principal -u $Env:ARM_CLIENT_ID -p $Env:ARM_CLIENT_SECRET --tenant $Env:ARM_TENANT_ID 
    . az account set --subscription $env:ARM_SUBSCRIPTION_ID

    $maxWaitSeconds = 1800
    $counter = 0
    $interval = 30

    . az provider register --namespace Microsoft.Kubernetes
    . az provider register --namespace Microsoft.ContainerService
    . az provider register --namespace Microsoft.KubernetesConfiguration
    start-sleep -seconds 15

    while ((. az provider show -n Microsoft.KubernetesConfiguration -o json | convertfrom-json).registrationState -ne "Registered" -and ($counter -lt $maxWaitSeconds) ) {
        Write-Host "Registering Kubernetes Configuration Provider for $counter seconds."
        start-sleep -seconds $interval
        $counter = $counter + $interval
    }

    Write-Host "Successfully installed Kubernetes Configuration Provider for subscription. Installing CLI Extensions..."
    . az extension add -n k8s-configuration
    . az extension add -n k8s-extension

    Start-Sleep -Seconds 120
    . az k8s-configuration flux create -g ${var.resource_group_name} --cluster-name ${azurerm_kubernetes_cluster.aks.name} --cluster-type managedClusters --name flux-main --scope cluster --namespace default --kind git --url ${var.flux_url} --branch ${var.flux_branch} --https-user git --https-key ${var.flux_pat} --kustomization name=main path=${var.flux_path} prune=true
EOT
}

[tiwood]

@techdecline, ill suggest to use the azapi provider instead of using local-exec.

I did it like this:

resource "azapi_resource" "flux_addon" {
  type       = "Microsoft.KubernetesConfiguration/extensions@2022-03-01"
  name       = "flux"
  parent_id  = azurerm_kubernetes_cluster.main.id
  locks      = [azurerm_kubernetes_cluster.main.id]
  body = jsonencode({
    properties = {
      extensionType           = "microsoft.flux"
      autoUpgradeMinorVersion = true
    }
  })

  timeouts {
    create = "20m"
  }
}

resource "azapi_resource" "flux_config" {
  type      = "Microsoft.KubernetesConfiguration/fluxConfigurations@2022-03-01"
  name      = "cluster-config"
  parent_id = azurerm_kubernetes_cluster.main.id

  depends_on = [
    azapi_resource.flux_addon
  ]

  body = jsonencode({
    properties = {
      scope      = "cluster"
      namespace  = "flux-system"
      sourceKind = "GitRepository"
      suspend    = false
      gitRepository = {
        url                   = local.gitops_repo_url
        timeoutInSeconds      = 120
        syncIntervalInSeconds = 120
        repositoryRef = {
          branch = var.gitOpsSettings.branchName
        }
        httpsUser = var.gitOpsSettings.httpsUser
      }
      configurationProtectedSettings = {
        httpsKey = sensitive(base64encode(var.gitOpsSettings.httpsPassword))
      }
      kustomizations = {
        shared = {
          path                   = "cluster-config/shared"
          timeoutInSeconds       = 600
          syncIntervalInSeconds  = 60
          retryIntervalInSeconds = 60
          prune                  = false
          force                  = false
        }
        applications = {
          path                   = "cluster-config/applications"
          dependsOn              = ["shared"]
          timeoutInSeconds       = 600
          syncIntervalInSeconds  = 60
          retryIntervalInSeconds = 60
          prune                  = false
          force                  = false
        }
      }
    }
  })

[techdecline]

Hey @tiwood, this is much cleaner, thanks for sharing. Any experience how this behaves when deploying and destroying the module multiple times?

[tiwood]

@techdecline , no issues with a blank cluster development and destruction.

I've seen some instances, where the destruction of the flux_config ran into an timeout, which was either due to destroyed firewall rules and/or deployments not cleaning up (deployments made by flux).

But I've seen the same with the native AKS resources if deployments, stateful sets, etc. are still present during destruction.

FWIW, I've captured the API requests made by the CLI if you enable GitOps, that is how I got to properties/defintion above.

[tidux]

Will this create any Python dependencies for installing the Azure CLI extensions?

[jeliasson]

@techdecline, ill suggest to use the azapi provider instead of using local-exec.

I tried this, along with a plain azurerm_kubernetes_cluster_extension resource, but I'm getting the following error;

ExtensionOperationFailed: The received access token is not valid: at least one of the claims 'puid' or 'altsecid' or 'oid' should be present. If you are accessing as application please make sure service principal is properly created in the tenant.

Full error

│ Error: creating/updating "Resource: (ResourceId \"/subscriptions/e121f1ae-b316-1b3d-bc3b-c05782ba71b8/resourceGroups/rg-example-kclsr1-aks-resources-dev-euw/providers/Microsoft.ContainerService/managedClusters/aks-example-kclsr1-dev-euw/providers/Microsoft.KubernetesConfiguration/extensions/flux\" / Api Version \"2022-11-01\")": GET https://management.azure.com/subscriptions/e121f1ae-b316-1b3d-bc3b-c05782ba71b8/resourceGroups/rg-example-kclsr1-aks-resources-dev-euw/providers/Microsoft.ContainerService/ManagedClusters/aks-example-kclsr1-dev-euw/providers/Microsoft.KubernetesConfiguration/extensions/flux/operations/8d772ea2-b10b-4a3e-9478-3cb8b45b3b45
│ --------------------------------------------------------------------------------
│ RESPONSE 200: 200 OK
│ ERROR CODE: ExtensionOperationFailed
│ --------------------------------------------------------------------------------
│ {
│   "id": "/subscriptions/e121f1ae-b316-1b3d-bc3b-c05782ba71b8/resourceGroups/rg-example-kclsr1-aks-resources-dev-euw/providers/Microsoft.ContainerService/ManagedClusters/aks-example-kclsr1-dev-euw/providers/Microsoft.KubernetesConfiguration/extensions/flux/operations/8d772ea2-b10b-4a3e-9478-3cb8b45b3b45",
│   "name": "8d772ea2-b10b-4a3e-9478-3cb8b45b3b45",
│   "status": "Failed",
│   "error": {
│     "code": "ExtensionOperationFailed",
│     "message": "The extension operation failed with the following error:  Request failed to https://management.azure.com/subscriptions/e121f1ae-b316-1b3d-bc3b-c05782ba71b8/resourceGroups/rg-example-kclsr1-aks-resources-dev-euw/providers/Microsoft.ContainerService/managedclusters/aks-example-kclsr1-dev-euw/extensionaddons/flux?api-version=2021-03-01. Error code: Unauthorized. Reason: Unauthorized.{\"error\":{\"code\":\"InvalidAuthenticationToken\",\"message\":\"The received access token is not valid: at least one of the claims 'puid' or 'altsecid' or 'oid' should be present. If you are accessing as application please make sure service principal is properly created in the tenant.\"}}.",
│     "additionalInfo": []
│   }
│ }
│ --------------------------------------------------------------------------------
│ 
│ 
│   with module.kclsr1[0].azapi_resource.flux_addon,
│   on .terraform/modules/kclsr1/flux.tf line 16, in resource "azapi_resource" "flux_addon":
│   16: resource "azapi_resource" "flux_addon" {
│ 

The application has sufficient permissions to create the underlying AKS cluster and the subscription in question has Microsoft.ContainerService, Microsoft.Kubernetes and Microsoft.KubernetesConfiguration registered as providers.

Any idea to what may cause this error?

Thanks!

Edit Seems to be fixed in version 3.65.0

[tiwood]

With the release of azurerm_kubernetes_cluster_extension we should no longer need the azapi provider, or am I missing something?

[horihel]

@tiwood i played with it a bit and it looks like azurerm_kubernetes_cluster_extension can install the flux cluster extension fine but not the fluxconfiguration. Additionally, if you add a fluxconfiguration with some other way (az cli), terraform can no longer destroy the extension.

[SebSa]

@tiwood i played with it a bit and it looks like azurerm_kubernetes_cluster_extension can install the flux cluster extension fine but not the fluxconfiguration. Additionally, if you add a fluxconfiguration with some other way (az cli), terraform can no longer destroy the extension.

Dealing with this now, it's a real pain

[stephybun]

Closed by #21579

[userfailures]

I have the same issue, I'm able to install flux but not add the flux configuration. I have to do the configuration with flux bootstrap command.

Also, this is not happening to me:

This is done with the Microsoft.KubernetesConfiguration/extensions/microsoft.flux cluster extension which is automatically installed when creating a Microsoft.KubernetesConfiguration/fluxConfigurations resource.

if I use Microsoft.KubernetesConfiguration/fluxConfigurations never it installs the extension, I'm in an infinite loop.

[vanhoutenbos]

Closed by #21579

Maybe I am missing something but it doesn’t seem like ‘prune’ was included in the PR even though it is in the azure documentation. I don’t see it on the TF documentation and when I try to add it by assuming where it should be I get errors

man I missing something?

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.