search

kubernetes / kubernetes

Production-Grade Container Scheduling and Management
https://kubernetes.io
Apache License 2.0
110.23k stars 39.44k forks source link

Allow admin to set the default bandwidth limit per port (network interface) for a container in a pod. #123501

Closed kundan2707 closed 2 months ago

kundan2707 commented 7 months ago

What would you like to be added?

CPU/Memory like bandwidth in ResourceQuota/LimitRange

Improve k8s to handle ingress-bandwidth and egress-bandwidth with ResourceQuota and LimitRange

Allow to set bandwidth settings in ResourceQuota as follows.

apiVersion: v1 kind: ResourceQuota metadata: name: pod-and-network-quota namespace: foo-ns spec: hard: pods: 4 ingress-bandwidth: "1G" <=== egress-bandwidth: "1G" <===

Allow to set bandwidth settings in LimitRange as follows.

apiVersion: v1 kind: LimitRange metadata: name: cpu-and-network-limitrange spec: limits:

Expected behaviors

Reject pod creation when (*):
    ResourceQuota has ingress/egress-bandwidth parmaeters.
    User hadn't set kubernetes.io/ingress-bandwidth and kubernetes.io/egress-bandwidth to their pod.
    LimitRange has no default values for ingress/egress-bandwidth.
Set default value to kubernetes.io/ingress-bandwidth and kubernetes.io/egress-bandwidth of a pod when:
    LimitRange has some default values for ingress/egress-bandwidth.
Reject pod creation when:
    The value of kubernetes.io/ingress-bandwidth or kubernetes.io/egress-bandwidth of a pod is larger/lower than the max/min value of LimitRange.
Reject pod creation when:
    The total kubernetes.io/ingress-bandwidth values of pods running on the namespace exceeds the ResourceQuota's value.

(*) It's best to reject to pod creation request when pod has no bandwidth annoatation even though ResourceQuota has bandwidth setting, since pod which has no bandwidth annotation can exceeds the quota. Feature to allow admin to set bandwith limit per port in a container. Pod level bandwidth limit should be supported by ResourceQuota and LimitRange

Why is this needed?

Users can set kubernetes.io/ingress-bandwidth and kubernetes.io/egress-bandwidth to their pods. But there is no way through which admit can force it to users. Such feature is very necessary in PaaS usage.

k8s-ci-robot commented 7 months ago

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
kundan2707 commented 7 months ago

/sig network

kundan2707 commented 7 months ago

@kubernetes/sig-network-feature-requests

thockin commented 7 months ago

xref some open issues:

At the moment, all bandwidth control is implementation-centric. Some don't support it at all, or support only one direction. Until we support it as a semi-standard (and TBD what that means, really) we can't really have standard quota.

I think the QoS issue is closest to how we really want to define the API, and that MUST solve for quota, IMO.

For bookkeeping I am going to close this, since those KEPs cover.

kundan2707 commented 6 months ago

@thockin

I am redefining complete use case here and reopening it to keep in track for possible solution. /reopen

k8s-ci-robot commented 6 months ago

@kundan2707: Reopened this issue.

In response to [this](https://github.com/kubernetes/kubernetes/issues/123501#issuecomment-2007193761): >@thockin >- https://github.com/kubernetes/enhancements/pull/2808: issue has been discussed partially and agreed on some API for its solutions . >- https://github.com/kubernetes/enhancements/issues/3008 : Currently it has been discussed, changed for cpu and memory only no bandwidth scenario discussed. > >I am redefining complete use case here and reopening it to keep in track for possible solution. >/reopen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
thockin commented 6 months ago

There isn't really a standard way to enforce bandwidth requests. We have those annotations, but they are totally optional and can't be relied upon.

Even if those WERE standardized and universally implemented, we're not scheduling against those, so they are limits without requests. We have no design for if/how to requests (guaranteed access) for bandwidth.

Without that, what does quota mean?

To be really clear - I am not against finding a way to model and manage network resources. I just don't know what it really looks like - I have more questions than answers. If we want to have an issue open for this larger goal, we can repurpose this issue and dump all the questions here.

thockin commented 4 months ago

@kundan2707 - I think we should close this. The real work is happening elsewhere, and unless you are proposing a DIFFERENT path, this is redundant

uablrek commented 2 months ago

@kundan2707 - I think we should close this. The real work is happening elsewhere, and unless you are proposing a DIFFERENT path, this is redundant

Bandwidth limitation is a very difficult topic. Since people have an "intuitive" idea what it is (which is often different), it is often mistaken for a simple thing. It's not!

I agree with Tim that this should be investigated/discussed at one place.

/close

k8s-ci-robot commented 2 months ago

@uablrek: Closing this issue.

In response to [this](https://github.com/kubernetes/kubernetes/issues/123501#issuecomment-2267448630): >> @kundan2707 - I think we should close this. The real work is happening elsewhere, and unless you are proposing a DIFFERENT path, this is redundant > >Bandwidth limitation is a _very_ difficult topic. Since people have an "intuitive" idea what it is (which is often different), it is often mistaken for a simple thing. It's not! > >I agree with Tim that this should be investigated/discussed at one place. > >/close > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.