search

loft-sh / vcluster

vCluster - Create fully functional virtual Kubernetes clusters - Each vcluster runs inside a namespace of the underlying k8s cluster. It's cheaper than creating separate full-blown clusters and it offers better multi-tenancy and isolation than regular namespaces.
https://www.vcluster.com
Apache License 2.0
6.27k stars 398 forks source link

AWS EKS using external-secrets from within vcluster not able to authenticate to AWS via OIDC. #1275

Open willjw1 opened 11 months ago

willjw1 commented 11 months ago

What happened?

external-secrets from within vcluster not able to authenticate to AWS via OIDC.

Seen in log for external-secrets pod:

"error":"could not validate provider: WebIdentityErr: failed to retrieve credentials\ncaused by: InvalidIdentityToken: No OpenIDConnect provider found in your account for https://kubernetes.default.svc.cluster.local

I find it odd that it's referring to the cluster as https://kubernetes.default.svc.cluster.local, as I would think it should be referring to is as will.default.svc.cluster.local

I have an AWS EKS cluster where I've created an OIDC provider for it. I have an AWS IAM Role which has a trusted entity of the Host cluster's sync'd SA I have an SA within my vcluster which has the annotation set to the Role in IAM.

vcluster's SA:

external-secrets   external-secrets

host cluster's sync'd SA:

vcluster           external-secrets-x-external-secrets-x-will

IAM Role which has trustedEntity statement:

{
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam:[ACCOUNTID]oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/[PROVIDER_ID]"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-east-1.amazonaws.com/id/[PROVIDER_ID]:sub": "system:serviceaccount:vcluster:secrets-manager-x-external-secrets-x-will"
                }
            }
        }

What did you expect to happen?

I was expecting the external-secrets pod to come up without issue. Then to go on and create my ClusterSecretStore and then define my ExternalSecret but it cannot do that as it doesn't have the correct credentials.

How can we reproduce it (as minimally and precisely as possible)?

create aws eks cluster with associated OIDC provider Use helm to install vcluster

helm upgrade --install will vcluster --values vcluster.yaml --repo https://charts.loft.sh --namespace vcluster --repository-config=''

I'm adding the following to a pretty standard vcluster.yaml:

sync:
  nodes:
    enabled: true
    syncAllNodes: true
  serviceaccounts:
    enabled: true

Anything else we need to know?

I named the vcluster and the host cluster the same name, that probably isn't a best practice.

I put two statements in my trust policy one for the host SA and one for the vcluster SA, didn't think that should matter.

Host cluster Kubernetes version

```console $ kubectl version Client Version: v1.28.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.27.4-eks-2d98532 ```

Host cluster Kubernetes distribution

``` Kubernetes version Info 1.27 ```

vlcuster version

```console $ vcluster --version vcluster version 0.15.7 ```

Vcluster Kubernetes distribution(k3s(default)), k8s, k0s)

``` vcluster: image: rancher/k3s:v1.28.2-k3s1 ```

OS and Arch

``` Kubernetes version 1.27 latest eks node version. ```
willjw1 commented 11 months ago

So went over this closely with a colleague, He points out that between different tests of external-secrets in vcluster it isn't cleaning up the local ClusterSecretStore and the External secret. Only once I've configured the ClusterSecretStore did I start to see the error messages in the log about the WebIdentityErr.

vumdao commented 11 months ago

The pod external-secrets-7c9c68c4dc-5tms8-x-external-secrets-x-dev8 of ESO (external-secrets operator) from the host view tried to use the ServiceAccount inside the vcluster core-service-parameterstore-ess and of course the vcluster does not have OIDC resource 

{"level":"error","ts":1696576259.4663882,"msg":"Reconciler error","controller":"secretstore","controllerGroup":"external-secrets.io","controllerKind":"SecretStore","SecretStore":{"name":"core-service-parameterstore-ess","namespace":"default"},"namespace":"default","name":"core-service-parameterstore-ess","reconcileID":"5ba826bc-25e5-4027-9335-945b014950dd","error":"could not validate provider: WebIdentityErr: failed to retrieve credentials\ncaused by: InvalidIdentityToken: No OpenIDConnect provider found in your account for https://kubernetes.default.svc.cluster.local\n\tstatus code: 400, request id: 2cb1c956-e194-42aa-8e2d-8ae1853f8076","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.1/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.1/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.1/pkg/internal/controller/controller.go:226"}

When I updated the external-secret to reference to SA which was synced to host core-service-secret-store-x-default-x-dev8, the error not found.


{"level":"error","ts":1696577723.0725968,"msg":"Reconciler error","controller":"secretstore","controllerGroup":"external-secrets.io","controllerKind":"SecretStore","SecretStore":{"name":"core-service-parameterstore-ess","namespace":"default"},"namespace":"default","name":"core-service-parameterstore-ess","reconcileID":"cfbcb8c2-b084-401c-a4e2-70113e4f19d9","error":"could not get provider client: unable to create session: ServiceAccount \"core-service-secret-store-x-default-x-dev8\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.1/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.1/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.1/pkg/internal/controller/controller.go:226"}
willjw1 commented 11 months ago

ok, so I think if I understand what you are saying is do not use the Service Account that you create within the vcluster. I instead should be using the name of the one which was sync'd back to the host machine.

So if the service account is created for me in namespace external-secrets and is called external-secrets then in the host cluster I would see: namespace: vcluster serviceaccount: external-secrets-x-external-secrets-x-will

so I would now have:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: global-secret-store
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-x-external-secrets-x-will
            namespace: vcluster

In my ClusterSecretStore configuration

this resulted in: "error":"could not get provider client: unable to create session: ServiceAccount \"external-secrets-x-external-secrets-x-will\" not found"

willjw1 commented 11 months ago

that does make sense at least, as vcluster should have no idea about a namespace called vcluster and the ServiceAccounts under it as I'm not syncing those to the vcluster.

joaocc commented 10 months ago

@willjw1, did you manage to solve this? I just noticed this after answering on another ticket https://github.com/loft-sh/vcluster/issues/1310#issuecomment-1800185219. Thanks

willjw1 commented 10 months ago

no I've still not gotten this working. I've moved on to trying to setup my own irsa provider by spoofing s3 as suggested here: https://www.youtube.com/watch?app=desktop&v=8YNXTI8E13s. but I cannot seem to get the vcluster bootstrapped correctly to take the additional parameters it would need.

frank-bee commented 9 months ago

I have vcluater running on EKS and ext. Secrets work. I did not sync service Accounts, at least I did not set the helm value explicitly to true. IRSA role annotation I have set in the SA in the Host cluster

joaocc commented 9 months ago

IRSA inside vcluster seems to work as intended (we tried with other softwaare), but in the case of external-secrets, it doesn't seem to work. We also gave up (details here), and switched using a IAM user and setting IAM key and secret in the cluster secret store. We would still love to get IAM to work for this use case, as it would simplify management and increase security. Thx

iasikou commented 8 months ago

We have exactly the same situation.. We can use and attach the ServiceAccount to a POD within vcluster and access ASM secrets but there is no luck using the same service account as authentication for external-secrets SecreStore/ClusterSecretStore, no matter what we tried... That means that IRSA works fine with vcluster but for some reason that does not work with external-secrets operator.

Unfortunately IAM access/secret keys for authentication is not an option for us at the moment, we have to make that working with IRSA.

Did anyone manage to make external-secrets working with IRSA within vcluster?

frank-bee commented 8 months ago

For me irsa works with the secrets controller. I think with k8s 1.25 or so, can look it up tomorrow.... You could also use the new EKS pod identities.

Ioannis Asikou @.***> schrieb am Mo., 8. Jan. 2024, 21:49:

We have exactly the same situation.. We can use and attach the ServiceAccount to a POD within vcluster and access ASM secrets but there is no luck using the same service account as authentication for external-secrets SecreStore/ClusterSecretStore, no matter what we tried... That means that IRSA works fine with vcluster but for some reason that does not work with external-secrets operator.

Unfortunately IAM access/secret keys for authentication is not an option for us at the moment, we have to make that working with IRSA.

Did anyone manage to make external-secrets working with IRSA within vcluster?

— Reply to this email directly, view it on GitHub https://github.com/loft-sh/vcluster/issues/1275#issuecomment-1881798993, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACDG747UCA4RFBPOF3LX3LYNRLW7AVCNFSM6AAAAAA5RQZWJ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBRG44TQOJZGM . You are receiving this because you commented.Message ID: @.***>

iasikou commented 8 months ago

Managed to make that working as well, regularly I was using a dedicated service account with IRSA for SecretsManager access and used that service account in the cluster store as

  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: my-sa
            namespace: default

but this was not working in vcluster (not sure why), what worked instead was to use the external-secrets service account that is created when the ESO is installed, this way you do not have to use auth.jwt.serviceAccountRef at all, and this makes the trick.

  1. Create IAM role with trust policy for external-secrets service account of the host cluster

    {
"Version": "2012-10-17",
"Statement": [
    {
        "Effect": "Allow",
        "Principal": {
            "Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/xxxx"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
            "StringEquals": {
                "oidc.eks.us-east-1.amazonaws.com/id/xxxx:sub": "system:serviceaccount:my-vcluster:external-secrets-x-external-secrets-x-my-vcluster",
                "oidc.eks.us-east-1.amazonaws.com/id/xxxx:aud": "sts.amazonaws.com"
            }
        }
    }
]
}

  2. Install ESO using this Helm values

    serviceAccount:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/my-role
name: external-secrets

  3. Create ClusterSecretStore (same for SecretStore)

    apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
name: secret-store
spec:
provider:
aws:
  region: us-east-1
  service: SecretsManager