search

kubernetes / enhancements

Enhancements tracking repo for Kubernetes
Apache License 2.0
3.43k stars 1.48k forks source link

Local Ephemeral Storage Capacity Isolation #361

Closed jingxu97 closed 1 year ago

jingxu97 commented 7 years ago

Feature Description

idvoretskyi commented 7 years ago

@jingxu97 @kubernetes/sig-storage-feature-requests any updates for 1.8? Is this feature still on track for the release?

jingxu97 commented 7 years ago

This feature is on track for 1.8. Details is here #43607

idvoretskyi commented 7 years ago

@jingxu97 please, update the features tracking board with the relevant data.

saad-ali commented 6 years ago

We intend to move local ephemeral storage to beta in 1.10.

Bradamant3 commented 6 years ago

@jingxu97 it looks as though docs need updating for 1.10. Can you please submit a docs PR as soon as possible (it's now officially late), and update the 1.10 feature tracking spreadsheet? Thanks!

jingxu97 commented 6 years ago

Hi Jennifer,

I submitted the PR https://github.com/kubernetes/website/pull/7614, but I could not edit the spreadsheet. Could you please help me check it? Thanks!

Best, Jing

On Fri, Mar 2, 2018 at 12:47 PM, Jennifer Rondeau notifications@github.com wrote:

@jingxu97 https://github.com/jingxu97 it looks as though docs need updating for 1.10. Can you please submit a docs PR as soon as possible (it's now officially late), and update the 1.10 feature tracking spreadsheet https://docs.google.com/spreadsheets/d/17bZrKTk8dOx5nomLrD1-93uBfajK5JS-v1o-nCLJmzE/edit#gid=0? Thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/features/issues/361#issuecomment-370047811, or mute the thread https://github.com/notifications/unsubscribe-auth/ASSNxQeScFENfuXNtUh9ctkN6reg_PWzks5taa_hgaJpZM4OkMYE .

--

Bradamant3 commented 6 years ago

Hi @jingxu97 -- Thanks for the docs PR. The spreadsheet is updated. Please note that you need to rebase your docs PR against the 1.10 docs branch -- we branch docs differently from the code repos. Thanks again! Jennifer

warmchang commented 6 years ago

Hi @jingxu97 @saad-ali , the local ephemeral storage management only applies for the root partition in release-1.9 (alpha). Does it suppot the runtime partition in release-1.10 (beta)?

jingxu97 commented 6 years ago

@warmchang, for beta version, it will be the same as alpha which only applies for the root partition. We currently don't plan to support other runtime partition due to the complexity. Could you please let me know what user case you need it for different partitions? Thanks!

warmchang commented 6 years ago

@jingxu97 I checked the original proposal local-storage-overview, it include the "Runtime Patition" description.

One scenario: The K8S deploy on IaaS (OpenStack or VMware) platform, base on considerations such as disk capacity, the nodes VMs would mount cloud disk as the "Docker Root Dir" instead of using the VMs' system root partitions. And then, how to manage the ephemeral storage for the containers running on the nodes? Thanks!

dashpole commented 6 years ago

@warmchang the runtime partition still has the same support it has had in the past. The kubelet will monitor the runtime partition, and perform evictions if space runs low based on the highest consumers of the runtime partition.

In your example, I'm not sure why using a cloud disk requires you to split the kubelet's and the runtime's partitions.

warmchang commented 6 years ago

@dashpole Before this Local Ephemeral Storage features, because the container writable layer unlimited write temporary files (such as logs) lead to full disk, resulting in the operating system hang, in order to prevent this behavior, we mount a Separate partition for Docker Root Dir.

We try the feature by this scenario, and found that it can not limit the capacity of container.

From a technical point of view, what is the difference between the capacity limits of the runtime partition and the root partition? Thanks!

dashpole commented 6 years ago

We try the feature by this scenario, and found that it can not limit the capacity of container.

The behavior you describe should work regardless of this feature. Make sure you have --root-dir set correctly. Docker reports its root directory to the kubelet, so as long as your images are stored on the same partition that contains /var/lib/docker (or whatever your docker root dir is), this should work correctly.

warmchang commented 6 years ago

@dashpole Very useful skill!

After verification (ping @zhangxiaoyu-zidif ), the expected effect can be achieved. 👏👏

[root@k8s-master-controller:/]$ kubectl get rs
NAME                             DESIRED   CURRENT   READY     AGE
busybox-apps-v1beta1-7f8dd8d89   1         1         1         21m
[root@k8s-master-controller:/]$ kubectl get pod --show-all
NAME                                   READY     STATUS    RESTARTS   AGE
busybox-apps-v1beta1-7f8dd8d89-kh6xc   1/1       Running   0          19m
busybox-apps-v1beta1-7f8dd8d89-mg7ls   0/1       Evicted   0          21m
[root@k8s-master-controller:/]$ kubectl describe pod busybox-apps-v1beta1-7f8dd8d89-mg7ls
Name:           busybox-apps-v1beta1-7f8dd8d89-mg7ls
Namespace:      default
Node:           172.160.134.17/
Start Time:     Mon, 23 Apr 2018 09:27:02 +0800
Labels:         app=busybox-apps-v1beta1
                pod-template-hash=394884845
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"busybox-apps-v1beta1-7f8dd8d89","uid":"6c817aea-4695-11e8-9103-f...
Status:         Failed
Reason:         Evicted
Message:        The node was low on resource: ephemeral-storage.
IP:
Created By:     ReplicaSet/busybox-apps-v1beta1-7f8dd8d89
Controlled By:  ReplicaSet/busybox-apps-v1beta1-7f8dd8d89
Containers:
  busybox:
    Image:  busybox
    Port:   <none>
    Command:
      sleep
      3600
    Limits:
      ephemeral-storage:  50Mi
    Requests:
      ephemeral-storage:  50Mi
    Environment:          <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7tchh (ro)
Volumes:
  default-token-7tchh:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-7tchh
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     <none>
Events:
  Type     Reason                 Age   From                     Message
  ----     ------                 ----  ----                     -------
  Normal   Scheduled              22m   default-scheduler        Successfully assigned busybox-apps-v1beta1-7f8dd8d89-mg7ls to 172.160.134.17
  Normal   SuccessfulMountVolume  22m   kubelet, 172.160.134.17  MountVolume.SetUp succeeded for volume "default-token-7tchh"
  Normal   Pulled                 22m   kubelet, 172.160.134.17  Container image "busybox" already present on machine
  Normal   Created                22m   kubelet, 172.160.134.17  Created container
  Normal   Started                22m   kubelet, 172.160.134.17  Started container
  Warning  Evicted                19m   kubelet, 172.160.134.17  pod ephemeral local storage usage exceeds the total limit of containers {{52428800 0} {<nil>} 50Mi BinarySI}
  Normal   Killing                19m   kubelet, 172.160.134.17  Killing container with id docker://busybox:Need to kill Pod
[root@k8s-master-controller:/]$
zhangxiaoyu-zidif commented 6 years ago

that's great for us. thanks for your help =) @dashpole

justaugustus commented 6 years ago

@jingxu97 @vishh Any plans for this in 1.11?

If so, can you please ensure the feature is up-to-date with the appropriate:

cc @idvoretskyi

justaugustus commented 6 years ago

This feature current has no milestone, so we'd like to check in and see if there are any plans for this in Kubernetes 1.12.

If so, please ensure that this issue is up-to-date with ALL of the following information:

Set the following:

Once this feature is appropriately updated, please explicitly ping @justaugustus, @kacole2, @robertsandoval, @rajendar38 to note that it is ready to be included in the Features Tracking Spreadsheet for Kubernetes 1.12.

Please note that Features Freeze is tomorrow, July 31st, after which any incomplete Feature issues will require an Exception request to be accepted into the milestone.

In addition, please be aware of the following relevant deadlines:

Please make sure all PRs for features have relevant release notes included as well.

Happy shipping!

P.S. This was sent via automation

kacole2 commented 6 years ago

Hi This enhancement has been tracked before, so we'd like to check in and see if there are any plans for this to graduate stages in Kubernetes 1.13. This release is targeted to be more ‘stable’ and will have an aggressive timeline. Please only include this enhancement if there is a high level of confidence it will meet the following deadlines:

Please take a moment to update the milestones on your original post for future tracking and ping @kacole2 if it needs to be included in the 1.13 Enhancements Tracking Sheet

Thanks!

PatrickLang commented 5 years ago

I mentioned this in the meeting today, but I wanted to add it here too. I think that apps must have a way to identify what quotas exist, and adapt to them. That would be a blocker from enabling enforcement later because if an app keeps exceeding quota and is killed repeatedly - that's bad. At the very least can we get this visible in a downward API if existing filesystem mechanisms won't work to discover the quota?

From a Kubernetes API standpoint, we need to be careful not to require an OS or cloud provider specific implementation. On Windows filesystems, quotas are not available so we would probably use a loopback volume to implement this. That preserves the Windows API behavior where if an app queries for the free space, it gets the space within that loopback volume.

claurence commented 5 years ago

@jingxu97 @vishh Hello - I’m the enhancement’s lead for 1.14 and I’m checking in on this issue to see what work (if any) is being planned for the 1.14 release. Enhancements freeze is Jan 29th and I want to remind that all enhancements must have a KEP

kacole2 commented 5 years ago

Hello @jingxu97 @vishh , I'm the Enhancement Lead for 1.15. Is this feature going to be graduating alpha/beta/stable stages in 1.15? Please let me know so it can be tracked properly and added to the spreadsheet. This will also require a KEP for inclusion

Once coding begins, please list all relevant k/k PRs in this issue so they can be tracked properly.

arunbpt7 commented 5 years ago

We try the feature by this scenario, and found that it can not limit the capacity of container.

The behavior you describe should work regardless of this feature. Make sure you have --root-dir set correctly. Docker reports its root directory to the kubelet, so as long as your images are stored on the same partition that contains /var/lib/docker (or whatever your docker root dir is), this should work correctly.

Hello ,

I would like to check if there any way to restrict Pods usage on ephemeral storage /var/lib/docker) regardless of mounted on node root fs or separate file system of /var/lib/docker .

Because the pods run time writable layers or logs growing up and that will fill out /var/lib/docker file system . This behavior is getting fill up the file system and stop other pods to run.

It would be great if we restrict pods use a limited amount of ephemeral storage on cluster wide . eg: set 20G quota for PODS that mean each pods can use only 20GB on ephemeral storage , if need more space should use the PV. Is there any possibility to do that

dashpole commented 5 years ago

Yes, you can do this with ephemeral storage. See the documentation. Make sure you have eviction enabled for both the "imagefs" and the "nodefs" (documentation).

arunbpt7 commented 5 years ago

Yes, you can do this with ephemeral storage. See the documentation. Make sure you have eviction enabled for both the "imagefs" and the "nodefs" (documentation).

Thanks for the update , have defined ephemeral-storage request and limit in resources (spec.hard.requests.ephemeral-storage , spec.hard.limits.ephemeral-storage) on the deployment and verified that evictionHard: is enabled for "imagefs and "nodefs" on the node . but when when deploying the pod and it is not restricting the pod to use the defined ephemeral storage . when creating large file inside the container it is still able to create files more that the ephemeral-storage request and limit.

evictionHard: imagefs.available: 15% memory.available: 100Mi nodefs.available: 10% nodefs.inodesFree: 5%

containers:

dashpole commented 5 years ago

Sounds like a bug. Feel free to open a separate issue and cc me, as this is for feature tracking.

arunbpt7 commented 5 years ago

Sounds like a bug. Feel free to open a separate issue and cc me, as this is for feature tracking.

Thank you , have opened a new issue ( local ephemeral Storage limitation for pods in the cluster #1094)

kacole2 commented 5 years ago

Hi @arunbpt7 @jingxu97 @vishh , I'm the 1.16 Enhancement Lead/Shadow. Is this feature going to be graduating alpha/beta/stable stages in 1.16? Please let me know so it can be added to the 1.16 Tracking Spreadsheet. If not's graduating, I will remove it from the milestone and change the tracked label.

Once coding begins or if it already has, please list all relevant k/k PRs in this issue so they can be tracked properly.

As a reminder, every enhancement requires a KEP in an implementable state with Graduation Criteria explaining each alpha/beta/stable stages requirements.

Milestone dates are Enhancement Freeze 7/30 and Code Freeze 8/29.

Thank you.

jeremyrickard commented 5 years ago

Hey there @arunbpt7 @jingxu97 @vishh -- 1.17 Enhancements shadow here 👋 . I wanted to check in and see if you think this Enhancement will be graduating to alpha/beta/stable in 1.17?

The current release schedule is:

If you do, I'll add it to the 1.17 tracking sheet (https://bit.ly/k8s117-enhancements). Once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

We'll also need to convert the design proposal into a KEP. To be accepted in the release, all enhancements MUST have a KEP, the KEP MUST be merged, in an implementable state, and have both graduation criteria/test plan.

Thanks!

palnabarun commented 4 years ago

Hey there @arunbpt7 @jingxu97 @vishh -- 1.18 Enhancements shadow here. I wanted to check in and see if you think this Enhancement will be graduating to alpha/beta/stable in 1.18 or having a major change in its current level?

The current release schedule is:

To be included in the release,

  1. The KEP PR must be merged
  2. The KEP must be in an implementable state
  3. The KEP must have test plans and graduation criteria.

If you would like to include this enhancement, once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

We'll be tracking enhancements here: http://bit.ly/k8s-1-18-enhancements

Thanks! :)

palnabarun commented 4 years ago

@arunbpt7 @jingxu97 @vishh Just a friendly reminder, we are just 7 days away from the Enhancement Freeze (Tuesday, January 28th).

palnabarun commented 4 years ago

@arunbpt7 @jingxu97 @vishh Just a friendly reminder, we are just 2 days away from the Enhancement Freeze (3 PM Pacific Time, Tuesday, January 28th).

palnabarun commented 4 years ago

Unfortunately, the deadline for the 1.18 Enhancement freeze has passed. For now, this is being removed from the milestone. If there is a need to get this in, please file an enhancement exception.

tedyu commented 4 years ago

@jingxu97 Can you review this ?

jingxu97 commented 4 years ago

currently we don't have any pending work on this feature. Before we can go to GA, just need to make sure all functions are working as expected for both linux and windows. From e2e tests, I think it works.

@tedyu , is there something particular I need to review?

tedyu commented 4 years ago

@jingxu97 Nothing in particular. There is linked issue, such as #78865 but I think they can be addressed on their own.

msedzins commented 4 years ago

Hey there @jingxu97 -- 1.19 Enhancements shadow here. I wanted to check in and see if you think this Enhancement will be graduating in 1.19?

In order to have this part of the release:

  1. The KEP PR must be merged in an implementable state
  2. The KEP must have test plans
  3. The KEP must have graduation criteria.

The current release schedule is:

If you do, I'll add it to the 1.19 tracking sheet (http://bit.ly/k8s-1-19-enhancements). Once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

Thanks!

msedzins commented 4 years ago

Hi there @jingxu97 ,

Kind reminder about my question above.

Regards, Mirek

msedzins commented 4 years ago

Hi there @jingxu97 ,

Kind reminder about my question above.

Regards, Mirek

msedzins commented 4 years ago

Hi there @vishh ,

Kind reminder about my question above.

Regards, Mirek

msedzins commented 4 years ago

Hi there @vishh ,

Kind reminder about my question above.

Regards, Mirek

msedzins commented 4 years ago

Hey @vishh , Enhancement shadow for the v1.19 release cycle here. Just following up on my earlier update to inform you of the upcoming Enhancement Freeze scheduled on Tuesday, May 19.

Regards, Mirek

palnabarun commented 4 years ago

@vishh -- Unfortunately the deadline for the 1.19 Enhancement freeze has passed. For now, this is being removed from the milestone and 1.19 tracking sheet. If there is a need to get this in, please file an enhancement exception.

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

Bessonov commented 4 years ago

/remove-lifecycle stale

kikisdeliveryservice commented 4 years ago

Hi @jingxu97 @vishh

Enhancements Lead here. Any plans to graduate this in 1.20?

Thanks, Kirsten

kikisdeliveryservice commented 4 years ago

Hi @jingxu97 @vishh

Following up: 1.20 Enhancements Freeze is October 6th. Could you let us know if you are planning to GA in 1.20? To be included in the milestone: The KEP must be merged in an implementable state The KEP must have test plans The KEP must have graduation criteria

This enhancement is quite old and not in the current format, see: https://github.com/kubernetes/enhancements/tree/master/keps/NNNN-kep-template

Also if you could update the description as requested: https://github.com/kubernetes/enhancements/issues/361#issuecomment-409037521 that would be great.

Thanks Kirsten

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

immanuelfodor commented 3 years ago

/remove-lifecycle stale

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

immanuelfodor commented 3 years ago

/remove-lifecycle stale