Unclear and contradictory information about filesystem layout on kubelet nodes

[rptaylor]

Hello,

According to [1] https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/#eviction-signals

kubelet supports only two filesystem partitions.

    The nodefs filesystem that kubelet uses for volumes, daemon logs, etc.
    The imagefs filesystem that container runtimes uses for storing images and container writable layers.

imagefs is optional. kubelet auto-discovers these filesystems using cAdvisor. kubelet does not care about any other filesystems. Any other types of configurations are not currently supported by the kubelet. For example, it is not OK to store volumes and logs in a dedicated filesystem.

What exactly is meant by "it is not OK to store volumes and logs in a dedicated filesystem" ? How or why is it not okay? This same paragraph says it IS okay to store volumes and logs in a "nodefs" filesystem, so it seems to be a contradiction. Does it depend on whether "nodefs" is dedicated? How can we know what "nodefs" is? Is it where /var/lib/kubernetes is located, or ... ?

There is some related information here: [2] https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#configurations-for-local-ephemeral-storage Issues with the description of the single filesystem layout:

• It describes putting all ephemeral local data into one filesystem, and dedicating the filesystem to Kubernetes. But after that, it explains that the default locations for those files are /var/lib/kubernetes and /var/log . The only way for those two directories to be in the same filesystem is if they are in the root filesystem or a /var filesystem . But /var/log is not dedicated for Kubernetes, all other system applications log there too. Moreover, if everything is stored in a / or /var filesystem, it is even less dedicated. No matter what, it seems there is no way for this filesystem to be dedicated to Kubernetes as the documentation describes.
• This paragraph implies it should be a dedicated FS (though as I said, it can not be) , while the first documentation page [1] says it can not be a dedicated FS.

Regarding the two filesystem layout:

• It describes a second filesystem, "the directory where you tell the kubelet to place container image layers and writeable layers". What directory is that? Where is it located by default? How can you change the kubelet configuration to use a different location? AFAICT this is actually about the container engine storage, e.g. /var/lib/docker , which is not directly related to the kubelet, is it?

It takes a bit of connecting the dots to figure out (at least this is the tentative inference/hunch I get from reading the documentation but I could definitely be wrong), but a simpler and more direct way to write these requirements could instead be:

• The number 1 most important point is that the logs and ephemeral volume locations (/var/log and /var/lib/kubelet) need to be on the same FS for ephemeral storage limits to be enforced
• Optionally you can have a separate FS for the container engine
• Optionally you can have other filesystems unrelated to k8s if you want

[tengqm]

@rptaylor Thanks for the indepth analysis. It is really helpful. Would you like propose a PR to get this issue closed?

[sftim]

/sig node /kind bug /language en /priority important-longterm

Also relevant, a bit: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#local-ephemeral-storage

[rptaylor]

Well I could propose some text if we have some information confirming and clarifying what the situation is, but I don't have that information needed to know for sure what the documentation should say. I have just speculated and guessed my way to a tentative understanding of the situation based on ~ 30 minutes of googling. It would be good to have some authoritative confirmation of what the situation really is and what the intended meaning was and if it has since changed, perhaps from whoever wrote or is responsible for the original documentation. The official documentation says "it is not OK to store volumes and logs in a dedicated filesystem" with italics for emphasis so it must have been very important.

[sftim]

/triage accepted

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[tengqm]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community. /close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to [this](https://github.com/kubernetes/website/issues/23915#issuecomment-858216716): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[sftim]

/reopen /remove-lifecycle rotten

[k8s-ci-robot]

@sftim: Reopened this issue.

In response to [this](https://github.com/kubernetes/website/issues/23915#issuecomment-979135683): >/reopen >/remove-lifecycle rotten Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[pkit]

Subscribed. I suppose somebody did know what was there and how it should have be configured, but the knowledge was lost...

[sftim]

A related thing we don't describe well: the difference between:

• the node's root filesystem which is always / and its same-filesystem children
• the kubelet's “root filesystem”, which defaults to whatever filesystem contains /var/lib/kubelet

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[rptaylor]

The page https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/#eviction-signals seems to be gone; now it redirects to https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/#eviction-signals The information about supported filesystem layouts for the node is changed, now it says

The kubelet supports the following filesystem partitions:

    nodefs: The node's main filesystem, used for local disk volumes, emptyDir, log storage, and more. For example, nodefs contains /var/lib/kubelet/.
    imagefs: An optional filesystem that container runtimes use to store container images and container writable layers.
Kubelet auto-discovers these filesystems and ignores other filesystems. Kubelet does not support other configurations.

It is still not totally clear what nodefs and imagefs would actually be on a node. Reading this page, I think an admin would still not have enough information to e.g. decide how/where to attach additional storage volumes and configure filesystems on cluster nodes.

[vaibhav2107]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[rptaylor]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[rptaylor]

/remove-lifecycle stale

[kannon92]

/cc

[kannon92]

@sftim Since you helped review the container runtime filesystem, should we consider updating these docs as part of this issue?

[kannon92]

In this case,

nodefs corresponds to the filesystem that /var/lib/kubelet (by default) lives on. Generally, I think we need to suggest that the root filesystem (ie /) should be the same filesystem that /var lives on.

And we do support cases where /var/lib/containers or /var/lib/containerd is located on a separate partition. Now, these locations are also configureable so it gets a bit confusing. These are the default locations for cri-o and containerd.

There is also future work to allow for more configurations for Kubelet for logs and the kubelet root location.

[sftim]

@sftim Since you helped review the container runtime filesystem, should we consider updating these docs as part of this issue?

I'm very happy for someone else to comment here.

[RonPortnoy]

Need clarification to