Reconciliation and secret-rotation.max-age

[oitTim]

We have installed Azurerator, and to date love it, however we noticed that the reconciliation does not seem to honor the secret-rotation.max-age automatically.

If we delete the Azure secrets associated with the AzureAdApplication, and wait an hour, nothing syncs or happens, however if we delete the pod and spawn another one, it reconciles and syncs.

If we add the annotation azure.nais.io/rotate: 'true' to the azureAdApplication, it will synchronize as well.

Also, restarting the pod every hour, will honor the max-age and create new secrets.

Any insight or help is appreciated.

This is the deployment we used:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Chart.Name }}
  labels:
    {{ include "labels" . | nindent 4 }}
spec:
  selector:
    matchLabels:
      app: {{ .Chart.Name | quote }}

  template:
    metadata:
      labels:
        {{ include "labels" . | nindent 8 }}

    spec:
      serviceAccountName: {{ .Chart.Name }}
      volumes:
        - name: azurerator-config
          secret:
            secretName: adp-azurerator-3rdparty-app

      containers:
        - name: {{ .Chart.Name | quote }}
          image: ghcr.io/nais/azurerator:latest
          volumeMounts:
            - name: azurerator-config
              mountPath: /azurerator.yaml
              subPath: azurerator.yaml

This is our azurerator.yaml:

azure:
  auth:
    client-id: "***REDACTED***"
    client-secret: "***REDACTED***"
  tenant:
    id: "***REDACTED***"
    name: "***REDACTED***"
  permissiongrant-resource-id: "***REDACTED***"
cluster-name: ***REDACTED***
secret-rotation:
  max-age: 1h
controller:
  context-timeout: 5m
  max-concurrent-reconciles: 15

And finally, our AzureAdApplication:

apiVersion: nais.io/v1
kind: AzureAdApplication
metadata:
  name: azurerator-test-app
  namespace: test-system
  labels:
    team: no-i-in-team
spec:
  secretName: azurerator-test-app

This is the log output on startup of pod:

{"level":"info","logger":"config","msg":"azure.auth.client-id: ***REDACTED***","time":"2024-08-29T11:23:26.665667288Z"}
2024-08-29T11:23:26.666108110Z {"level":"info","logger":"config","msg":"azure.auth.client-secret: ***REDACTED***","time":"2024-08-29T11:23:26.66578822Z"}
2024-08-29T11:23:26.666120280Z {"level":"info","logger":"config","msg":"azure.auth.google.enabled: false","time":"2024-08-29T11:23:26.665798398Z"}
2024-08-29T11:23:26.666130317Z {"level":"info","logger":"config","msg":"azure.auth.google.project-id: ","time":"2024-08-29T11:23:26.665803923Z"}
{"level":"info","logger":"config","msg":"azure.delay.between-modifications: 5s","time":"2024-08-29T11:23:26.665809684Z"}
2024-08-29T11:23:26.666148849Z {"level":"info","logger":"config","msg":"azure.features.app-role-assignment-required.enabled: false","time":"2024-08-29T11:23:26.665814371Z"}
2024-08-29T11:23:26.666157998Z {"level":"info","logger":"config","msg":"azure.features.claims-mapping-policies.enabled: false","time":"2024-08-29T11:23:26.665829804Z"}
2024-08-29T11:23:26.666166986Z {"level":"info","logger":"config","msg":"azure.features.claims-mapping-policies.id: ","time":"2024-08-29T11:23:26.66583443Z"}
2024-08-29T11:23:26.666261140Z {"level":"info","logger":"config","msg":"azure.features.cleanup-orphans.enabled: false","time":"2024-08-29T11:23:26.665841312Z"}
{"level":"info","logger":"config","msg":"azure.features.custom-security-attributes.enabled: false","time":"2024-08-29T11:23:26.665859328Z"}
2024-08-29T11:23:26.666288287Z {"level":"info","logger":"config","msg":"azure.features.group-membership-claim.default: ApplicationGroup","time":"2024-08-29T11:23:26.665865147Z"}
2024-08-29T11:23:26.666291787Z {"level":"info","logger":"config","msg":"azure.features.groups-assignment.all-users-group-id: []","time":"2024-08-29T11:23:26.665871237Z"}
2024-08-29T11:23:26.666295022Z {"level":"info","logger":"config","msg":"azure.features.groups-assignment.enabled: false","time":"2024-08-29T11:23:26.66587946Z"}
2024-08-29T11:23:26.666298391Z {"level":"info","logger":"config","msg":"azure.pagination.max-pages: 1000","time":"2024-08-29T11:23:26.665885945Z"}
2024-08-29T11:23:26.666301662Z {"level":"info","logger":"config","msg":"azure.permissiongrant-resource-id: ***REDACTED***","time":"2024-08-29T11:23:26.665891189Z"}
2024-08-29T11:23:26.666305073Z {"level":"info","logger":"config","msg":"azure.tenant.id: ***REDACTED***","time":"2024-08-29T11:23:26.665901099Z"}
2024-08-29T11:23:26.666308335Z {"level":"info","logger":"config","msg":"azure.tenant.name: ***REDACTED***","time":"2024-08-29T11:23:26.665905844Z"}
2024-08-29T11:23:26.666311646Z {"level":"info","logger":"config","msg":"cluster-name: ***REDACTED***","time":"2024-08-29T11:23:26.665910289Z"}
2024-08-29T11:23:26.666314864Z {"level":"info","logger":"config","msg":"controller.context-timeout: 5m","time":"2024-08-29T11:23:26.66591455Z"}
{"level":"info","logger":"config","msg":"controller.max-concurrent-reconciles: 15","time":"2024-08-29T11:23:26.665919803Z"}
2024-08-29T11:23:26.666331648Z {"level":"info","logger":"config","msg":"kafka.brokers: [localhost:9092]","time":"2024-08-29T11:23:26.665925544Z"}
{"level":"info","logger":"config","msg":"kafka.enabled: false","time":"2024-08-29T11:23:26.665934752Z"}
2024-08-29T11:23:26.666338376Z {"level":"info","logger":"config","msg":"kafka.max-processing-time: 10s","time":"2024-08-29T11:23:26.665943042Z"}
2024-08-29T11:23:26.666341520Z {"level":"info","logger":"config","msg":"kafka.retry-interval: 5s","time":"2024-08-29T11:23:26.665947497Z"}
2024-08-29T11:23:26.666345323Z {"level":"info","logger":"config","msg":"kafka.tls.ca-path: ","time":"2024-08-29T11:23:26.665951659Z"}
{"level":"info","logger":"config","msg":"kafka.tls.certificate-path: ","time":"2024-08-29T11:23:26.665958116Z"}
2024-08-29T11:23:26.666351728Z {"level":"info","logger":"config","msg":"kafka.tls.enabled: false","time":"2024-08-29T11:23:26.665963071Z"}
2024-08-29T11:23:26.666354853Z {"level":"info","logger":"config","msg":"kafka.tls.private-key-path: ","time":"2024-08-29T11:23:26.665967444Z"}
2024-08-29T11:23:26.666357984Z {"level":"info","logger":"config","msg":"kafka.topic: azurerator-events","time":"2024-08-29T11:23:26.665972105Z"}
2024-08-29T11:23:26.666361245Z {"level":"info","logger":"config","msg":"leader-election.enabled: false","time":"2024-08-29T11:23:26.665976549Z"}
2024-08-29T11:23:26.666364353Z {"level":"info","logger":"config","msg":"leader-election.namespace: ","time":"2024-08-29T11:23:26.665981124Z"}
2024-08-29T11:23:26.666367483Z {"level":"info","logger":"config","msg":"metrics-address: :8080","time":"2024-08-29T11:23:26.665985374Z"}
2024-08-29T11:23:26.666370612Z {"level":"info","logger":"config","msg":"secret-rotation.cleanup: true","time":"2024-08-29T11:23:26.665989735Z"}
2024-08-29T11:23:26.666373837Z {"level":"info","logger":"config","msg":"secret-rotation.max-age: 1h","time":"2024-08-29T11:23:26.665994356Z"}
2024-08-29T11:23:26.666376997Z {"level":"info","logger":"config","msg":"validations.tenant.required: false","time":"2024-08-29T11:23:26.665999322Z"}
{"level":"info","timestamp":"2024-08-29T11:23:27.277212342Z","logger":"setup","caller":"azurerator/main.go:143","msg":"starting metrics refresh goroutine"}
{"level":"info","timestamp":"2024-08-29T11:23:27.277317681Z","logger":"setup","caller":"azurerator/main.go:147","msg":"starting manager"}
2024-08-29T11:23:27.278795019Z {"level":"info","timestamp":"2024-08-29T11:23:27.278597229Z","logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"}
2024-08-29T11:23:27.280166290Z {"level":"info","timestamp":"2024-08-29T11:23:27.279983764Z","logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":8080","secure":false}
2024-08-29T11:23:27.396320854Z {"level":"info","timestamp":"2024-08-29T11:23:27.396181611Z","caller":"controller/controller.go:175","msg":"Starting EventSource","controller":"azureadapplication","controllerGroup":"nais.io","controllerKind":"AzureAdApplication","source":"kind source: *nais_io_v1.AzureAdApplication"}
2024-08-29T11:23:27.396341703Z {"level":"info","timestamp":"2024-08-29T11:23:27.396255203Z","caller":"controller/controller.go:183","msg":"Starting Controller","controller":"azureadapplication","controllerGroup":"nais.io","controllerKind":"AzureAdApplication"}
{"level":"info","timestamp":"2024-08-29T11:23:27.501027003Z","caller":"controller/controller.go:217","msg":"Starting workers","controller":"azureadapplication","controllerGroup":"nais.io","controllerKind":"AzureAdApplication","worker count":15}

[tronghn]

Hello! 👋

This is likely because of max-age being set to 1 hour. Azurerator relies on the controller-runtime caching mechanisms for reconciliation. The default SyncPeriod for these caches appears to be ~10 hours, so objects aren't reconciled until then unless they're modified.

As you've discovered, using the azure.nais.io/rotate: 'true' annotation will force an immediate one-time rotation for the given AzureAdApplication. Restarting Azurerator will reconcile all objects as the cache is no longer populated.

We haven't really considered the use case with such low max-age values before, but we could look into it if you want

[oitTim]

Thanks for the response. I was just setting a low max-age to test the software. I will change it to 1d to see how it works :D

[oitTim]

I updated the secret-rotation.max-age: 24h and we are experiencing the same issue? It's just been a tad over 48 hours .. so perhaps I just need to patient :P

{"level":"info","logger":"config","msg":"secret-rotation.max-age: 24h","time":"2024-09-04T17:50:15.380904613Z"}
{"level":"info","logger":"config","msg":"validations.tenant.required: false","time":"2024-09-04T17:50:15.380909261Z"}
{"level":"info","timestamp":"2024-09-04T17:50:15.804381225Z","logger":"setup","caller":"azurerator/main.go:143","msg":"starting metrics refresh goroutine"}
{"level":"info","timestamp":"2024-09-04T17:50:15.804554871Z","logger":"setup","caller":"azurerator/main.go:147","msg":"starting manager"}
{"level":"info","timestamp":"2024-09-04T17:50:15.805144924Z","logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"}
{"level":"info","timestamp":"2024-09-04T17:50:15.805807351Z","logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":8080","secure":false}
{"level":"info","timestamp":"2024-09-04T17:50:15.92348925Z","caller":"controller/controller.go:175","msg":"Starting EventSource","controller":"azureadapplication","controllerGroup":"nais.io","controllerKind":"AzureAdApplication","source":"kind source: *nais_io_v1.AzureAdApplication"}
{"level":"info","timestamp":"2024-09-04T17:50:15.923572536Z","caller":"controller/controller.go:183","msg":"Starting Controller","controller":"azureadapplication","controllerGroup":"nais.io","controllerKind":"AzureAdApplication"}
{"level":"info","timestamp":"2024-09-04T17:50:16.029670653Z","caller":"controller/controller.go:217","msg":"Starting workers","controller":"azureadapplication","controllerGroup":"nais.io","controllerKind":"AzureAdApplication","worker count":15}
{"ClientID":"41cbddce-d8f3-463f-93ab-fa5bddcf19aa","CorrelationID":"aaf06575-ec3e-4d4f-b08a-386b5d33ecfb","ObjectID":"8cd928f4-ade1-4e81-b047-d1232f45415f","ServicePrincipalID":"243ae3ea-1978-4658-93c1-9702d05f6b37","application_name":"adp-azurerator-test-app","application_namespace":"ais-system","level":"debug","msg":"updated status fields with values from Azure","time":"2024-09-04T17:50:16.578183668Z"}
{"CorrelationID":"aaf06575-ec3e-4d4f-b08a-386b5d33ecfb","application_name":"adp-azurerator-test-app","application_namespace":"ais-system","level":"debug","msg":"resource is addressed to tenant 'catmail.ohio.edu', processing...","time":"2024-09-04T17:50:16.578232327Z"}
{"level":"info","msg":"metrics with namespace labels initialized","time":"2024-09-04T17:50:16.824499724Z"}
{"CorrelationID":"aaf06575-ec3e-4d4f-b08a-386b5d33ecfb","application_name":"adp-azurerator-test-app","application_namespace":"ais-system","level":"debug","msg":"existing credentials are valid and in sync with Azure","time":"2024-09-04T17:50:17.485806828Z"}

[tronghn]

Hmm, might have something to do with the event filtering mechanism. I'll have a closer look into it