Open philipsparrow opened 4 years ago
From what I can read it seems as though /lib
should be symlinked to either /lib64
or /lib32
and the same for /usr/lib
to /usr/lib64
or /usr/lib32
From what I can read it seems as though
/lib
should be symlinked to either/lib64
or/lib32
and the same for/usr/lib
to/usr/lib64
or/usr/lib32
That seems fair. If it needs to be done the same way as you did yours then I'll need to look into why the path was /usr/src/image/
Hi everyone,
I actually have the same difficulties trying to deploy fluentd on a single-node k3s cluster.
I deployed Rancher and k3s in an air-gaped environnment, imported my k3c cluster in Rancher I pull images through a private repository I had no problem deploying personnal apps or things like Wordpress, Fluentd-aggregator, Longhorn
But trying to enable logging from Rancher UI does not work
kubectl get pods -n cattle-logging
NAME READY STATUS RESTARTS AGE
rancher-logging-log-aggregator-linux-5gkjc 1/1 Running 1 `123m`
rancher-logging-fluentd-linux-5f52b 1/2 CreateContainerError 0 19m
kubectl describe pod rancher-logging-fluentd-linux
Warning Failed 6m57s kubelet, k3s-node.mydomain.com Error: failed to generate container "8c4f16a28392d86f548f6a344abbac6b4edf2c70769cb1f34931be90575dde80" spec: failed to mkdir "/usr/lib64": mkdir /usr/lib64: read-only file system
Also, even when I ssh this node I cant manually create the directory /usr/lib64
mkdir: cannot create directory ‘/usr/lib64’: Read-only file system
Trying to change authorisations on /usr/ directory does not work, when I type sudo chmod -R 777 ./usr/
chmod: changing permissions of './usr/src/linux-headers-5.0.0-43-generic/tools/objtool/subcmd-config.o': Read-only file system
Same goes for every file in the /usr/ directory.
So going through differents forums I was able to find some informations but I don't really know if this part is relevant :
k3s-node[/]$ fdisk -l
Disk /dev/loop1: 56.55 MiB, 59281408 bytes, 115784 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/loop2: 223.3 MiB, 234135552 bytes, 457296 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/sda: 20 GiB, 21474836480 bytes, 41943040 sectors
Disk model: Virtual disk
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 860D0536-5838-48BB-ADEC-6B50975FC623
Device Start End Sectors Size Type
/dev/sda1 2048 98303 96256 47M EFI System
/dev/sda2 98304 41943006 41844703 20G Linux filesystem
But when trying to launch a filesystem check, /dev/sda1 return an error:
k3s-node[/home/rancher]$ fsck.ext4 /dev/sda1
e2fsck 1.45.5 (07-Jan-2020)
ext2fs_open2: Bad magic number in super-block
fsck.ext4: Superblock invalid, trying backup blocks...
fsck.ext4: Bad magic number in super-block while trying to open /dev/sda1
The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem. If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>
or
e2fsck -b 32768 <device>
/dev/sda1 contains a vfat file system labelled 'K3OS_GRUB'
No special comportements for the other partitions:
k3s-node[/home/rancher]$ fsck.ext4 /dev/sda
e2fsck 1.45.5 (07-Jan-2020)
/dev/sda is in use.
e2fsck: Cannot continue, aborting.
Restoring superblocks does not correct the bug.
Thanks for you help, I will keep you tuned with my developments
Version k3OS : v0.10.2
Version K3s : v1.17.6+k3s1
Version Rancher : v2.4.4
Folks, /usr
is a read-only mount of a squashfs appended to the /k3os/system/k3os/current/k3os
executable. It would be nice for such use-cases if it was a lowerdir
of an overlayfs but that isn't currently the case. However, the rootfs is writable so setting up such might be an adventure to your liking?
As for /usr/lib(32|64)
I believe this is a glibc-ism whereas k3OS userspace is based on Alpine and therefore musl-c. My concern here is that the runtimes we are attempting to install rely on glibc which will absolutely break.
All of this said, I would like rancher/rancher syslog to work =) So, I will look into what can be done.
Version (k3OS / kernel) v0.10.1 5.0.0-43-generic
Architecture x86_64
Describe the bug Enabling Syslog logging (in Rancher) deploys pods that fail to start due to K3OS readonly filesystems.
rancher-logging-fluentd-linux
pods get the error:Error: failed to generate container "<containerID>" spec: failed to mkdir "/usr/lib64": mkdir /usr/lib64: read-only file system
Because there is a hostPath mount of/usr/lib64
which doesn't exist in K3OS so I think Kubernetes tries to create the directoryTo Reproduce Add a K3OS cluster to Rancher server, enable Syslog logging, on the cluster (any configuration will suffice). The DaemonSet pods fail to start because Kubernetes cannot
mkdir /usr/lib64
on a read-only filesystem.Expected behavior
kubectl get pods -n=cattle-logging
should show pods atReady
status.Actual behavior
Additional context I think a related issue has recently been resolved by https://github.com/rancher/k3os/pull/447 This related issue is for
rancher-logging-log-aggregator-linux
pods getting the error:Error: failed to generate container "<containerID>" spec: failed to mkdir "/usr/libexec/kubernetes/kubelet-plugins/volume/exec": mkdir /usr/libexec/kubernetes: read-only file system
I'm guessing that a similar fix is required here to create an empty/usr/lib64
directory or to symlink/usr/lib
to/usr/lib64
.