🚀 Add support for labels and annotations for pods created by AgentPool agentDeployment (To support Microsoft Workload Identity OIDC runs)

LaurentLesle commented 4 weeks ago

Description

Azurerm provider support OIDC (Workload Identities) which is a password-less and Microsoft recommended approach to run Terraform jobs. When using TFE or HPC self hosted agents on an AKS with OIDC enabled I want to run terraform jobs in this setup.

Currently it is possible to set the serviceAccountName but the azure workload identity admission controller requires as well the labels and (optional) annotations to the added into the template spec.

Therefore I am proposing the add in the AgentPool schema 2 new attributes to support labels and annotations.

Potential YAML Configuration

# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: MPL-2.0

apiVersion: app.terraform.io/v1alpha2
kind: AgentPool
metadata:
  name: eslz-platform
spec:
  organization: tfeoidc
  token:
    secretKeyRef:
      name: terraformrc
      key: token
  name: eslz-platform
  agentTokens:
    - name: eslz-platform
  agentDeployment:
    replicas: 3
    spec:
      containers:
        - name: tfc-agent
          image: "hashicorp/tfc-agent:1.15.4"
      serviceAccountName: "workload-identity-sa"
    labels:
      azure.workload.identity/use: "true"
    annotations:
      azure.workload.identity/service-account-token-expiration: 3600
  autoscaling:
    minReplicas: 0
    maxReplicas: 10
    cooldownPeriod:
      scaleUpSeconds: 30
      scaleDownSeconds: 30

References

https://azure.github.io/azure-workload-identity/docs/topics/service-account-labels-and-annotations.html

Additional context on how OIDC is used (https://techcommunity.microsoft.com/t5/fasttrack-for-azure/use-azure-ad-workload-identity-for-kubernetes-with-a-user/ba-p/3654928)

Expected state of the pod created by the agentPool (Service Assount and Labels (optionally annotations) set. Here a manual pod crafted. Note the env AZURE_* added by the admission controller based on the label and service account:

k describe pod tfe-agent
Name:             tfe-agent
Namespace:        default
Priority:         0
Service Account:  workload-identity-sa
Node:             tfe-oidc-worker2/172.18.0.5
Start Time:       Thu, 15 Aug 2024 02:41:46 +0000
Labels:           azure.workload.identity/use=true
Annotations:      <none>
Status:           Running
IP:               10.244.1.18
IPs:
  IP:  10.244.1.18
Containers:
  tfe-agent:
    Container ID:  containerd://09579a3d288fa37b308201d77f0f858f74340421db1fd0cf755ca91767ddf02f
    Image:         hashicorp/tfc-agent:1.15.4
    Image ID:      docker.io/hashicorp/tfc-agent@sha256:c8b4cfc17ea7d310d9dd9228bd1111e0f5435b9eddf786524d3b40e1aa5a2448
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      export TFC_AGENT_TOKEN=$(cat /token-volume/token) && ./bin/tfc-agent
    State:          Running
      Started:      Thu, 15 Aug 2024 02:41:50 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      TFC_AGENT_AUTO_UPDATE:       patch
      AZURE_CLIENT_ID:             cfa2a192-06f6-444b-aead-643428f71b4e
      AZURE_TENANT_ID:             00000000000-ad6c-4ee8-93cd-f7619e8a4c9d
      AZURE_FEDERATED_TOKEN_FILE:  /var/run/secrets/azure/tokens/azure-identity-token
      AZURE_AUTHORITY_HOST:        https://login.microsoftonline.com/
    Mounts:
      /token-volume from token-volume (rw)
      /var/run/secrets/azure/tokens from azure-identity-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2wsnx (ro)

We can see only the agent pool name is added. The ask of this feature is to inject (and merge) also the spec.agentDeployment.labels and spec.agentDeployment.annotations:

https://github.com/hashicorp/hcp-terraform-operator/blob/main/controllers/agentpool_controller_deployment.go

https://github.com/hashicorp/hcp-terraform-operator/blob/6127ef00412aad4852198581e0181b204fa3eb58/api/v1alpha2/agentpool_types.go#L93

Community Note

Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.

sheneska commented 4 weeks ago

Hi @LaurentLesle, thank you for opening this issue - and for explaining the issue so well. We are currently working on this and it will be included in one of the upcoming releases!

LaurentLesle commented 4 weeks ago

Oh that's a great news and will unblock lot of our customers! Thanks

BBBmau commented 4 weeks ago

contributors should take notes of how you presented your issue. Really good stuff! @LaurentLesle

hashicorp / hcp-terraform-operator