search

kubernetes-sigs / cluster-api-provider-nested

Cluster API Provider for Nested Clusters
Apache License 2.0
299 stars 65 forks source link

✨ Projected ServiceAccount Support #282

Closed christopherhein closed 5 months ago

christopherhein commented 2 years ago

User Story

As a user I would like to use ServiceAccount Projection from a VirtualCluster so that I can expose my cluster as an Identity provider and issue certs to tools like vault.

Detailed Description

Upstream Kubernetes added support for ServiceAccount projection but it requires that the kubelet make requests on behalf of a workload to grant it tokens with specific audiences and expirations this is difficult to do with shared kubelets in VC. It would be nice if we could figure a way to support this.

Docs: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection

Anything else you would like to add:

You can make these with a TokenRequest definition and using curl against the token subresource for ServiceAccounts like so:

curl -X "POST" {API}/api/v1/namespaces/{namespace}/serviceaccounts/{serviceaccount-name}/token -H 'Authorization: Bearer {token}' -H 'Content-Type: application/json; charset=utf-8' -d $'{}'

the Body of this usually accepts this object - https://github.com/kubernetes/api/blob/v0.20.1/authentication/v1/types.go#L131

For example:

{
  "apiVersion": "authentication.k8s.io/v1",
  "kind": "TokenRequest",
  "metadata": {
    "name": "{name}"
  },
  "spec": {
    "audiences": [
      "vault"
    ],
    "expirationSeconds": 7200,
    "boundObjectRef": {
      "kind": "Pod",
      "apiVersion": "v1",
      "name": "{name}",
      "uid": "{uid}"
    }
  }
}

Ideas

My initial thought process is adding a step in the pod syncer, where it currently checks for ServiceAccount tokens being created before creating the pod that it would call out to a "projected serviceaccount syncer" to make this request against the tenant control plane then stores the contents in a secret only in the super cluster then we mutate the pod spec to change from projected to a secret source and mount like a normal secret. We'd then need something that was checking validity of these tokens, the "projected serviceaccount syncer" for example, and when it got close to expiring we redo the call and update the token causing the secret mount to update as well.

Alternatively, we could modify the Kubelet to be "tenant aware" but I imagine this would become a massive effort.

/kind feature

Fei-Guo commented 2 years ago

I agree. From modeling perspective, the syncer can be viewed as the "Kubelet" for all tenants since it honors the Pod lifecycle events for all tenants and provision the actual workload. It can "fork" the kubelet behavior to support SA token auto refresh.

During Pod creation, syncer can request a valid token, add the projection spec in a special annotation, and mutate the project volume to secret mount. Syncer can implement a patroller to scan all super pods with SA project enabled by checking the annotation and refreshing their secrets on behalf the pods. It is a nontrivial change but should work.

wondywang commented 2 years ago

Hi @christopherhein , do you have any plan to support this feature? Just want to make sure this issue is still on track. :smile:

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 1 year ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes-sigs/cluster-api-provider-nested/issues/282#issuecomment-1399189267): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
christopherhein commented 1 year ago

/reopen

k8s-ci-robot commented 1 year ago

@christopherhein: Reopened this issue.

In response to [this](https://github.com/kubernetes-sigs/cluster-api-provider-nested/issues/282#issuecomment-1399409602): >/reopen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 1 year ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes-sigs/cluster-api-provider-nested/issues/282#issuecomment-1437906539): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
christopherhein commented 1 year ago

/reopen

k8s-ci-robot commented 1 year ago

@christopherhein: Reopened this issue.

In response to [this](https://github.com/kubernetes-sigs/cluster-api-provider-nested/issues/282#issuecomment-1498255086): >/reopen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
christopherhein commented 1 year ago

Hi @christopherhein , do you have any plan to support this feature? Just want to make sure this issue is still on track. 😄

Hey @wondywang I have not been able to prioritize this feature on our end, is this something you are doing? It would be a great addition.

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 1 year ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes-sigs/cluster-api-provider-nested/issues/282#issuecomment-1536901327): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
christopherhein commented 1 year ago

/reopen

k8s-ci-robot commented 1 year ago

@christopherhein: Reopened this issue.

In response to [this](https://github.com/kubernetes-sigs/cluster-api-provider-nested/issues/282#issuecomment-1537049440): >/reopen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 1 year ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes-sigs/cluster-api-provider-nested/issues/282#issuecomment-1576084639): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
christopherhein commented 1 year ago

/reopen /remove-lifecycle rotten

k8s-ci-robot commented 1 year ago

@christopherhein: Reopened this issue.

In response to [this](https://github.com/kubernetes-sigs/cluster-api-provider-nested/issues/282#issuecomment-1579522160): >/reopen >/remove-lifecycle rotten Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
wondywang commented 1 year ago

Hey @wondywang I have not been able to prioritize this feature on our end, is this something you are doing? It would be a great addition.

Hi @christopherhein. Sorry for taking so long to reply to you. I have developed a feature based on Kubernetes 1.22 to be compatible with the kube-root-ca feature-gate. And kubernetes 1.22 has not completely discarded the ServiceAccount token, which will not be discarded until 1.24.

At the same time, I have developed a feature to be compatible with Kubernetes 1.24, which the ServiceAccount token no longer exists. Currently, this feature is still being verified (due to the current internal cluster, the 1.24 version cluster is lacking).

cc @Fei-Guo

k8s-triage-robot commented 7 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 6 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 5 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 5 months ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes-sigs/cluster-api-provider-nested/issues/282#issuecomment-2017280320): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.