Use IRSA for `external-dns`

Story

-As a cluster admin, I want external-dns to use IAM Roles for Service Accounts (IRSA) instead of Kiam in order to improve the security and stability of clusters given that Kiam is now deprecated.

Background

Current state:

Kiam is deprecated and upstream recommends IAM Roles for Service Accounts (IRSA) as the replacement;
The AWS v17.1.0 release introduced IAM Roles for Service Accounts (IRSA) support;
Giant Swarm is going to remove Kiam from the list of default applications;
external-dns is still using Kiam in order to access Route53 to configure DNS;

Changes

[ ] configure external-dns to use IRSA instead of Kiam;
[ ] keep the option of switching back to Kiam in case we encounter issues during cluster upgrades;

I spent some time thinking what we need to do in order to make it work.

Main tasks:

[x] PR https://github.com/giantswarm/aws-pod-identity-webhook/pull/26 to allow aws-pod-identity-webhook injecting IAM role credentials into kube-system
[x] PR https://github.com/giantswarm/external-dns-app/pull/168 for external-dns-app
[x] PR https://github.com/giantswarm/cluster-operator/pull/1528 for cluster-operator to add cluster-values, unfortunately we need the AWS account ID to make the IRSA annotation work
[x] PR https://github.com/giantswarm/aws-operator/pull/3337 for aws-operator to add additional trust-relationship for CLUSTER_ID-Route53Manager-Role

cluster-operator:

We need to add the values AWS account ID (this value might also added to cluster-apps-operator for CAPI) and a IRSA flag in cluster-values. Cluster-operator already checks if the annotation for IRSA is set, so it can enable the IRSA flag in cluster-values.

AWS Account ID is needed because we need to use the full ARN arn:aws:iam::ACCOUNT_ID:role/CLUSTER_ID-Route53Manager-Role. Unlike kiam we cannot skip the first part and only use the role name 😞. I tested this and it's not possible so far.

The values can be picked up from external-dns and maybe other apps.

aws-operator:

Because we add the IAM role for Route53 we need to modify it slightly to add an additional trust relationship to allow external-dns assuming this role. See https://github.com/giantswarm/aws-operator/blob/master/service/controller/resource/tccpn/template/template_main_iam_policies.go#L125-L135

giantswarm / roadmap