Closed nnlkcncff closed 1 month ago
This issue is currently awaiting triage.
SIG Docs takes a lead on issue triage for this website, but any Kubernetes member can accept issues by applying the triage/accepted
label.
The triage/accepted
label can be added by org members by writing /triage accepted
in a comment.
Page related to issue: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ /language en
Sorry, I think I've misunderstood about writable layers. I think the documentation implies a single writable layer, but for each container. Because of this, "writable layers" is plural. But the directory for these layers is still in question.
There are running containers, which the container runtime cares about, and then there are container images.
Plausibly there's a CRI implementation where the kubelet and CRI listening socket are on server B, and then each Pod is on a separate hardware server (C,D,E,F) with a special OS, and the container image data is copied onto those other servers during Pod launch.
Container images have layers; they just do. Running containers might use OverlayFS, but they could also use another technology so long as the app in the container sees the same files and directories.
We can document that better, but IMO https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ isn't where we need to make the improvement.
Storage of container images is the responsibility of CRI, not kubelet.
CRI is an interface between the kubelet and the container runtime. It's defined by the Kubernetes project. The kubelet tells the container runtime where to store the pulled image data (and the runtime does the actual fetch).
Earlier at the all points I wrote CRI, but I meant container runtime (CR), sorry for confusing.
I understand there could be different implementation, e.g.:
Also I understand that OverlayFS is not the only option for storage driver.
I also realize that I don't understand enough :)
In the case of containerd, I did some research based on running container configurations and image manifests and saw the following:
k8s.io
CR namespace, their root filesystems are placed under /run/containerd/io.containerd.runtime.v2.task/k8s.io/.../rootfs
default
CR namespace, /run/containerd/io.containerd.runtime.v2.task/default/.../rootfs
/var/lib/containerd/io.containerd.content.v1.content/blobs
CR seems to be the one that controls image placement and layers.
Most data under the /var/lib/kubelet/pods/
looks like volumes (which could writable or not), there are no writable layers or images.
Because all of that, the phrase The kubelet also uses this kind of storage to hold node-level container logs, container images, and the writable layers of running containers.
doesn't look quite accurate to me.
You can close this issue at any time if you think it's the right thing to do, because this time I opened an issue without being completely sure about it. Maybe I should have discussed this here first.
/sig node Let's triage this.
(BTW, and only as far as I know: if you don't configure imagefs, the kubelet allows the container runtime to put the image wherever it likes)
/var/lib/containerd/io.containerd.content.v1.content/blobs
is part of what Kubernetes considers:
The node level bit is obvious, but maybe the “ephemeral” isn't clear enough. A related problem is that the whole concept is hard to explain and is much more of a node concept than a storage concept or a resource management thing.
I'd welcome a new page covering kubelet and what goes in local node filesystems; the new page could be part of https://kubernetes.io/docs/reference/node/
/retitle Confusing and possibly inaccurate docs about node-level ephemeral storage
According to what I understand I think that /var/lib/containerd/io.containerd.content.v1.content/blobs
shouldn't be considered as a node level (if you don't mount this directory directly via hostPath) nor an ephemeral, because for Linux container layer blobs are just tar.gzip archives, and no blobs are directly mounted.
During pull operation all kinds of blobs (index, manifest, config and layers) are downloaded to the blobs
directory and every layer blob is extracted to snapshots
directory.
While container starts:
rootfs
directory (for OverlayFS this is the lowerdir)rootfs
directory under the snapthots
directory (for OverlayFS this is usually the workdir and the upperdir)All volume mount operations happen on the writable layer or somewhere at sandbox level (due to volume propagation policies, tmpfs mount specifics and maybe something else).
Directory paths in the case of containerd, OverlayFS, and Kubernetes:
/var/lib/containerd/io.containerd.content.v1.content/blobs/...
/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/.../...
/run/containerd/io.containerd.runtime.v2.task/k8s.io/.../rootfs
It's the way I see it.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
/sig storage /remove-lifecycle rotten
We should triage this.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/reopen
/remove-lifecycle rotten
Please send feedback to sig-contributor-experience at kubernetes/community.
/close not-planned
@k8s-triage-robot: Closing this issue, marking it as "Not Planned".
The information about container images and writable layers doesn't seem quite correct.
Storage of container images is the responsibility of CRI, not kubelet. For new empty cluster it is easy to check by comparing CRI and kubelet catalog sizes, e.g. (with containerd as CRI):
You will see that the CRI directory will be larger than the kubelet directory, mainly because of the container images.
Kubelet places only some files in some directories, especially in the
/var/lib/kubelet/pods
directory, and some of them are then mounted by CRI (including ephemeral storage volumes, backed by emptyDir).But there is only one writable layer — the upper one, and it seems to be the responsibility of the storage driver (OverlayFS2 by default), so this layer located in CRI directory tree, for Kubernetes with containerd as CRI it's
/run/containerd/io.containerd.runtime.v2.task/k8s.io/.../rootfs
. For example:Isn't that right?